问题
I am working on a python NLTK tagging program. My input file is Hindi text containing several lines. On tokenizing the text and using pos_tag the output I get is with NN tag only. but with English sentence as input it does proper tagging. Kindly Help. Version - Python 3.4.1, from NLTK 3.0 documentation
Kindly help! here is what I tried.
word_to_be_tagged = u"ताजो स्वास आनी चकचकीत दांत तुमचें व्यक्तीमत्व परजळायतात."
from nltk.corpus import indian
train_data = indian.tagged_sents('hindi.pos')[:300]
test_data = indian.tagged_sents('hindi.pos')[301:]
print(word_to_be_tagged)
print (train_data)
and the output I get is different.
ताजो स्वास आनी चकचकीत दांत तुमचें व्यक्तीमत्व परजळायतात.
[[('पूर्ण', 'JJ'), ('प्रतिबंध', 'NN'), ('हटाओ', 'VFM'), (':', 'SYM'), ('इराक', 'NNP')], [('संयुक्त', 'NNC'), ('राष्ट्र', 'NN'), ('।', 'SYM')], ...]
回答1:
The problem is that you should use hindi POS Tagger:
from nltk.corpus import indian
from nltk.tag import tnt
train_data = indian.tagged_sents('hindi.pos')
tnt_pos_tagger = tnt.TnT()
tnt_pos_tagger.train(train_data) #Training the tnt Part of speech tagger with hindi data
print tnt_pos_tagger.tag(nltk.word_tokenize(word_to_be_tagged))
The problem is that a Part Of Speech tagger is accurate in a specific domain (mostly combination of language and topic). In English, most of the words the tagger haven't seen yet are Nouns (NN), it tags you data with NN only.
If you train it with the same domain you want it to tag after (Hindi), it should be OK.
See this for more explanations.
来源：https://stackoverflow.com/questions/30555426/unicode-tagging-in-python-nltk