How do I tokenize a string sentence in NLTK?

问题

I am using nltk, so I want to create my own custom texts just like the default ones on nltk.books. However, I\'ve just got up to the method like

my_text = [\'This\', \'is\', \'my\', \'text\']

I\'d like to discover any way to input my \"text\" as:

my_text = \"This is my text, this is a nice way to input text.\"

Which method, python\'s or from nltk allows me to do this. And more important, how can I dismiss punctuation symbols?

回答1:

This is actually on the main page of nltk.org:

>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']

回答2:

As @PavelAnossov answered, the canonical answer, use the word_tokenize function in nltk:

from nltk import word_tokenize
sent = "This is my text, this is a nice way to input text."
word_tokenize(sent)

---

If your sentence is truly simple enough:

Using the string.punctuation set, remove punctuation then split using the whitespace delimiter:

import string
x = "This is my text, this is a nice way to input text."
y = "".join([i for i in x if not in string.punctuation]).split(" ")
print y

来源：https://stackoverflow.com/questions/15057945/how-do-i-tokenize-a-string-sentence-in-nltk