Compressing A Series of JSON Objects While Maintaining Serial Reading?

Question

I have a bunch of json objects that I need to compress as it\'s eating too much disk space, approximately 20 gigs worth for a few million of them.

Answer 1

You might want to try an incremental json parser, such as jsaone.

That is, create a single json with all your objects, and parse it like

with gzip.GzipFile(file_path, 'r') as f_in:
    for key, val in jsaone.load(f_in):
        ...

This is quite similar to Martin's answer, wasting slightly more space but maybe slightly more comfortable.

EDIT: oh, by the way, it's probably fair to clarify that I wrote jsaone.