I am learning Natural Language Processing using NLTK.
I came across the code using PunktSentenceTokenizer whose actual use I cannot understand in the given code
def process_content(corpus):
tokenized = PunktSentenceTokenizer().tokenize(corpus)
try:
for sent in tokenized:
words = nltk.word_tokenize(sent)
tagged = nltk.pos_tag(words)
print(tagged)
except Exception as e:
print(str(e))
process_content(train_text)
Without even training it on other text data it works the same as it is pre-trained.