Trying to store Utf-8 data in datastore getting UnicodeEncodeError

Question

Trying to store utf-8 into datastore and getting error :

Traceback (most recent call last):
  File "/sinfo/google_appengine/google/appengine/ext/webapp/__init__.py", line 511, in __call__
    handler.get(*groups)
  File "/sinfo/siteinfo/siteinfo.py", line 1911, in get
    seoEntity.put()
  File "/sinfo/google_appengine/google/appengine/ext/db/__init__.py", line 833, in put
    return datastore.Put(self._entity, rpc=rpc)
  File "/sinfo/google_appengine/google/appengine/api/datastore.py", line 275, in Put
    req.entity_list().extend([e._ToPb() for e in entities])
  File "/sinfo/google_appengine/google/appengine/api/datastore.py", line 680, in _ToPb
    properties = datastore_types.ToPropertyPb(name, values)
  File "/sinfo/google_appengine/google/appengine/api/datastore_types.py", line 1499, in ToPropertyPb
    pbvalue = pack_prop(name, v, pb.mutable_value())
  File "/sinfo/google_appengine/google/appengine/api/datastore_types.py", line 1322, in PackString
    pbvalue.set_stringvalue(unicode(value).encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

How do i solve this? The data is already utf-8 encoded and when I enter it into the datastore it uses the ascii codec and fails?

Answer 1

I use following helper in my projects

def force_utf8(string):
    if type(string) == str:
        return string
    return string.encode('utf-8')

Use it to escape all your unicode data before passing to GAE. Also you can find useful the following snippet:

def force_unicode(string):
    if type(string) == unicode:
        return string
    return string.decode('utf-8')