Python 2.6 JSON decoding performance

Question

I\'m using the json module in Python 2.6 to load and decode JSON files. However I\'m currently getting slower than expected performance. I\'m using a test case

Answer 1

For those who are parsing output from a request using the requests package, e.g.:

res = requests.request(...)

text = json.loads(res.text)

This can be very slow for larger response contents, say ~45 seconds for 6 MB on my 2017 MacBook. It is not caused by a slow json parser, but instead by a slow character set determination by the res.text call.

You can solve this by setting the character set before you are calling res.text, and using the cchardet package (see also here):

if res.encoding is None:
    res.encoding = cchardet.detect(res.content)['encoding']

This makes the response text json parsing almost instant!