Python 如何在 NLTK 中标记字符串句子？
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/15057945/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
How do I tokenize a string sentence in NLTK?
提问by diegoaguilar
I am using nltk, so I want to create my own custom texts just like the default ones on nltk.books. However, I've just got up to the method like
我正在使用 nltk，所以我想创建我自己的自定义文本，就像 nltk.books 上的默认文本一样。但是，我刚刚开始使用类似的方法
my_text = ['This', 'is', 'my', 'text']
I'd like to discover any way to input my "text" as:
我想发现以任何方式输入我的“文本”：
my_text = "This is my text, this is a nice way to input text."
Which method, python's or from nltk allows me to do this. And more important, how can I dismiss punctuation symbols?
哪种方法，python 的或来自 nltk 的方法允许我这样做。更重要的是，我怎样才能忽略标点符号？
采纳答案by Pavel Anossov
This is actually on the main page of nltk.org:
这实际上是在nltk.org 的主页上：
>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']
回答by alvas
As @PavelAnossov answered, the canonical answer, use the word_tokenizefunction in nltk:
正如@PavelAnossov 回答的那样，规范的答案是使用word_tokenizenltk 中的函数：
from nltk import word_tokenize
sent = "This is my text, this is a nice way to input text."
word_tokenize(sent)
If your sentence is truly simple enough:
如果你的句子真的足够简单：
Using the string.punctuationset, remove punctuation then split using the whitespace delimiter:
使用string.punctuation集合，删除标点符号，然后使用空格分隔符拆分：
import string
x = "This is my text, this is a nice way to input text."
y = "".join([i for i in x if not in string.punctuation]).split(" ")
print y
