Use of PunktSentenceTokenizer in NLTK

后端 未结 4 1866
不思量自难忘°
不思量自难忘° 2020-12-07 16:55

I am learning Natural Language Processing using NLTK. I came across the code using PunktSentenceTokenizer whose actual use I cannot understand in the given code

4条回答
  •  情歌与酒
    2020-12-07 17:17

    def process_content(corpus):
    
        tokenized = PunktSentenceTokenizer().tokenize(corpus)
    
        try:
            for sent in tokenized:
                words = nltk.word_tokenize(sent)
                tagged = nltk.pos_tag(words)
                print(tagged)
        except Exception as e:
            print(str(e))
    
    process_content(train_text)
    

    Without even training it on other text data it works the same as it is pre-trained.

提交回复
热议问题