Dump to JSON adds additional double quotes and escaping of quotes

Question

I am retrieving Twitter data with a Python tool and dump these in JSON format to my disk. I noticed an unintended escaping of the entire data-string for a tweet being enclos

Answer 1

Another situation where this unwanted escaping can happen is if you try to use json.dump() on the pre-processed output of json.dumps(). For example

import json, sys
json.dump({"foo": json.dumps([{"bar": 1}, {"baz": 2}])},sys.stdout)

will result in

{"foo": "[{\"bar\": 1}, {\"baz\": 2}]"}

To avoid this, you need to pass dictionaries rather than the output of json.dumps(), e.g.

json.dump({"foo": [{"bar": 1}, {"baz": 2}]},sys.stdout)

which outputs the desired

{"foo": [{"bar": 1}, {"baz": 2}]}

(Why would you pre-process the inner list with json.dumps(), you ask? Well, I had another function that was creating that inner list out of other stuff, and I thought it would make sense to return a json object from that function... Wrong.)

Answer 2

You are double encoding your JSON strings. data is already a JSON string, and doesn't need to be encoded again:

>>> import json
>>> not_encoded = {"created_at":"Fri Aug 08 11:04:40 +0000 2014"}
>>> encoded_data = json.dumps(not_encoded)
>>> print encoded_data
{"created_at": "Fri Aug 08 11:04:40 +0000 2014"}
>>> double_encode = json.dumps(encoded_data)
>>> print double_encode
"{\"created_at\": \"Fri Aug 08 11:04:40 +0000 2014\"}"

Just write these directly to your file:

with open('data{}.txt'.format(self.timestamp), 'a') as f:
    f.write(data + '\n')