TypeError: Object of type 'bytes' is not JSON serializable

Question

I just started programming Python. I want to use scrapy to create a bot，and it showed TypeError: Object of type \'bytes\' is not JSON serializable when I run the project. <

Answer 1

I was dealing with this issue today, and I knew that I had something encoded as a bytes object that I was trying to serialize as json with json.dump(my_json_object, write_to_file.json). my_json_object in this case was a very large json object that I had created, so I had several dicts, lists, and strings to look at to find what was still in bytes format.

The way I ended up solving it: the write_to_file.json will have everything up to the bytes object that is causing the issue.

In my particular case this was a line obtained through

for line in text:
    json_object['line'] = line.strip()

I solved by first finding this error with the help of the write_to_file.json, then by correcting it to:

for line in text:
    json_object['line'] = line.strip().decode()

Answer 2

You are creating those bytes objects yourself:

item['title'] = [t.encode('utf-8') for t in title]
item['link'] = [l.encode('utf-8') for l in link]
item['desc'] = [d.encode('utf-8') for d in desc]
items.append(item)

Each of those t.encode(), l.encode() and d.encode() calls creates a bytes string. Do not do this, leave it to the JSON format to serialise these.

Next, you are making several other errors; you are encoding too much where there is no need to. Leave it to the json module and the standard file object returned by the open() call to handle encoding.

You also don't need to convert your items list to a dictionary; it'll already be an object that can be JSON encoded directly:

class W3SchoolPipeline(object):    
    def __init__(self):
        self.file = open('w3school_data_utf8.json', 'w', encoding='utf-8')

    def process_item(self, item, spider):
        line = json.dumps(item) + '\n'
        self.file.write(line)
        return item

I'm guessing you followed a tutorial that assumed Python 2, you are using Python 3 instead. I strongly suggest you find a different tutorial; not only is it written for an outdated version of Python, if it is advocating line.decode('unicode_escape') it is teaching some extremely bad habits that'll lead to hard-to-track bugs. I can recommend you look at Think Python, 2nd edition for a good, free, book on learning Python 3.

Answer 3

I guess the answer you need is referenced here Python sets are not json serializable

Not all datatypes can be json serialized . I guess pickle module will serve your purpose.