What's the best way to load large JSON lists in Python? [duplicate]

Answer 1

There is already a similar post here. Here is the solution they proposed:

import json
with open('file.json') as infile:
  o = json.load(infile)
  chunkSize = 1000
  for i in xrange(0, len(o), chunkSize):
    with open('file_' + str(i//chunkSize) + '.json', 'w') as outfile:
      json.dump(o[i:i+chunkSize], outfile)

Answer 2

The module pandas 0.21.0 now supports chunksize as part of read_json. You can load and manipulate one chunk at a time:

import pandas as pd
chunks = pd.read_json(file, lines=True, chunksize = 100)
for c in chunks:
    print(c)

Answer 3

If you're trying to avoid loading the whole list into memory, you could process the file as text first:

Use a stack to keep track of open and closing of brackets/quotes. Scan through the string for any of the openers, or the current closer. When scanning through text, look only for the text closer. Push one on when you read the opener, and pop it off when you find the closer.

The full set for JSON is [ -> ], { -> } and " -> ". You should exclude \" though. You can check the spec at http://www.json.org/

Then whenever a ] is encountered and the stack only has one item (the top level '[') after popping the matching [, then you know it's time to start a new line.

Finally, you should ensure the first [ and last ] don't appear in your output.

That will give you separate JSON objects for each item of the list, each on a separate line of the file.

If you dig into the python JSON library, there should be some functions that parse JSON too. You could leverage those, even though they aren't part of the public interface.

Of course, you can achieve the same by loading the string using the JSON library and then dumping it item by item (or multiple items) as per the other answer.