How do I fix this unicode/cPickle error in Python?

Question

ids = cPickle.loads(gem.value)

loads() argument 1 must be string, not unicode

Answer 1

The result of cPickle.dumps() is a str object, not a unicode object. You need to find the step in your code where you are decoding the pickled str object, and omit that step.

DON'T try to convert your unicode object to a str object. Two wrongs don't make a right. Example (Python 2.6):

>>> import cPickle
>>> ps = cPickle.dumps([1,2,3], -1)
>>> ps
'\x80\x02]q\x01(K\x01K\x02K\x03e.'
>>> ups = ps.decode('latin1')
>>> str(ups)
Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'ascii' codec can't encode character u'\x80' in position 0: ordinal not in range(128)
>>>

You may well be using the default (and inefficient) Protocol 0 which produces "human readable" output:

>>> ps = cPickle.dumps([1,2,3])
>>> ps
'(lp1\nI1\naI2\naI3\na.'
>>>

which is presumably ASCII (but not documented to be so) so the str(gem.value) kludge may well """work""":

>>> ps == str(unicode(ps))
True
>>>