NLTK accuracy: “ValueError: too many values to unpack”

Question

I\'m trying to do some sentiment analysis of a new movie from Twitter using the NLTK toolkit. I\'ve followed the NLTK \'movie_reviews\' example and I\'ve built my own Catego

Answer 1

The error message

File "/usr/lib/python2.7/dist-packages/nltk/classify/util.py", line 87, in accuracy
  results = classifier.classify_many([fs for (fs,l) in gold])
ValueError: too many values to unpack

arises because items in gold can not be unpacked into a 2-tuple, (fs,l):

[fs for (fs,l) in gold]  # <-- The ValueError is raised here

It is the same error you would get if gold equals [(1,2,3)], since the 3-tuple (1,2,3) can not be unpacked into a 2-tuple (fs,l):

In [74]: [fs for (fs,l) in [(1,2)]]
Out[74]: [1]
In [73]: [fs for (fs,l) in [(1,2,3)]]
ValueError: too many values to unpack

gold might be buried inside the implementation of nltk.classify.util.accuracy, but this hints that your inputs, classifier or testfeats are of the wrong "shape".

There is no problem with classifer, since calling accuracy(classifier, trainfeats) works:

In [61]: print 'accuracy:', nltk.classify.util.accuracy(classifier, trainfeats)
accuracy: 0.9675

The problem must be in testfeats.

---

Compare trainfeats with testfeats. trainfeats[0] is a 2-tuple containing a dict and a classification:

In [63]: trainfeats[0]
Out[63]: 
({u'!': True,
  u'"': True,
  u'&': True,
  ...
  u'years': True,
  u'you': True,
  u'your': True},
 'neg')           # <---  Notice the classification, 'neg'

but testfeats[0] is just a dict, word_feats(tweets.words(fileids=[f])):

testfeats = [(word_feats(tweets.words(fileids=[f]))) for f in tweetsids]

So to fix this you would need to define testfeats to look more like trainfeats -- each dict returned by word_feats must be paired with a classification.