couchbase python sdk ascii exception

问题

First of all, this is the exception

Traceback (most recent call last):
      File "C:\Python27\lib\site-packages\scrapy\middleware.py", line 62, in _process_chain
        return process_chain(self.methods[methodname], obj, *args)
      File "C:\Python27\lib\site-packages\scrapy\utils\defer.py", line 65, in process_chain
        d.callback(input)
      File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 383, in callback
        self._startRunCallbacks(result)
      File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 491, in _startRunCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "C:\Python27\lib\site-packages\twisted\internet\defer.py", line 578, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "D:\ScrapyProjects\General_Spider_code_version_4\General_Spider_code_version_4\pipelines.py", line 14, in process_item
        connection.set(fileName, dict(item)) #write the item to the couchbase database
      File "C:\Python27\lib\site-packages\couchbase-1.2.5-py2.7-win-amd64.egg\couchbase\connection.py", line 331, in set
        persist_to, replicate_to)
      File "C:\Python27\lib\site-packages\couchbase-1.2.5-py2.7-win-amd64.egg\couchbase\_bootstrap.py", line 99, in _json_encode_wrapper
        return json.dumps(*args, ensure_ascii=False, separators=(',', ':'))
      File "C:\Python27\lib\json\__init__.py", line 250, in dumps
        sort_keys=sort_keys, **kw).encode(obj)
      File "C:\Python27\lib\json\encoder.py", line 210, in encode
        return ''.join(chunks)
    couchbase.exceptions.ValueFormatError: <Couldn't encode value, inner_cause='ascii' codec can't decode byte 0xe2 in position 5: ordinal not in range(128), C Source=(src\convert.c,131), OBJ={'bathrooms': 1.0, 'furnished': 'No', 'ad_title': 'Large Studio For Rent in IMPZ just 45K/4chqs(KK)', 'agent_fees': -1, 'size': 550.0, 'category': 'Apartment', 'company_rera_number': '12913', 'agent_company': 'AL ANAS REAL ESTATE BROKER', 'ded_licence_number': '700590', 'source': 'dubizzleproperty', 'location': 'UAE \xe2\x80\xaa>\xe2\x80\xaa Dubai \xe2\x80\xaa>\xe2\x80\xaa IMPZ International Media Production Zone ; 3.1 km from Meadows Town Centre \xc2\xa0', 'image_links': [u'http://87421a79fde09fda7e57-79445249ccb41a60f7b99c8ef6df8604.r12.cf3.rackcdn.com/4_async/2015/2/18/73ff34e2a38c7b104401c9e5c54b03628971053f/main.jpeg', u'http://87421a79fde09fda7e57-79445249ccb41a60f7b99c8ef6df8604.r12.cf3.rackcdn.com/4_async/2015/2/18/24ec831f6b4afb47fecc1c3e0991cf3090c90c24/main.jpeg', u'http://87421a79fde09fda7e57-79445249ccb41a60f7b99c8ef6df8604.r12.cf3.rackcdn.com/4_async/2015/2/18/77fee11394090aaea2d668cfe2754b92d6e36264/main.jpeg', u'http://87421a79fde09fda7e57-79445249ccb41a60f7b99c8ef6df8604.r12.cf3.rackcdn.com/4_async/2015/2/18/5d4113319ccbabcdd65b0ffe7302da59b374b5fe/main.jpeg', u'http://87421a79fde09fda7e57-79445249ccb41a60f7b99c8ef6df8604.r12.cf3.rackcdn.com/4_async/2015/2/18/8070689f309759d5860e97aa35d3f0eac425dc1d/main.jpeg', u'http://87421a79fde09fda7e57-79445249ccb41a60f7b99c8ef6df8604.r12.cf3.rackcdn.com/4_async/2015/2/18/8e86702847e69d485d147629dd2e48e1ad831e63/main.jpeg'], 'latitude': -1, 'description': 'Central A/C & Heating , Balcony , Shared Pool , Built in Wardrobes , Walk-in Closet , Shared Gym , Security , Built in Kitchen Appliances', 'bedrooms': 'Studio', 'rent_is_paid': 'Quarterly', 'action': 'Rent', 'link': 'http://dubai.dubizzle.com/property-for-rent/residential/apartmentflat/2015/2/18/large-studio-for-rent-in-impz-just-45k4chq-2/?back=ZHViYWkuZHViaXp6bGUuY29tL3Byb3BlcnR5LWZvci1yZW50L3Jlc2lkZW50aWFsL2FwYXJ0bWVudGZsYXQv&pos=1', 'longitude': -1, 'property_reference': '', 'yearly_cost': 45000.0, 'agent_mobile': -1, 'posting_date': '2015-02-19'}>

I am trying to store a dictionary on a couchbase. i am using this code

connection.set(fileName, dict(item))

to transfer the item to dictionary. as you see from the error message. i have a unicode values, which is according to python sdk couchbase is fine, could you help me please?

回答1:

Your values are not unicode. Keep in mind that a str object containing valid unicode escape sequences does not automatically make it "Unicode" in Python parlance. You need to ensure the strings are properly Unicode.

This seems to work with the normal json.dumps() function (without any arguments); whereas the python client passes (by default) the ensure_ascii=False parameter to decrease the data size (JSON itself can be in UTF-8 encoding, and is not limited to ASCII).

Thus, a workaround may be to set your own encoding function for JSON which does not pass the ensure_ascii parameter; like so:

import json
import couchbase

couchbase.set_json_converters(json.dumps, json.loads)

Though this workaround is not recommended as it may inflate your document size slightly.

来源：https://stackoverflow.com/questions/28593263/couchbase-python-sdk-ascii-exception