Parse large JSON file [duplicate]

Answer 1

try this lib https://github.com/shevron/ext-jsonreader

The existing ext/json which is shipped with PHP is very convenient and simple to use - but it is inefficient when working with large ammounts of JSON data, as it requires reading the entire JSON data into memory (e.g. using file_get_contents()) and then converting it into a PHP variable at once - for large data sets, this takes up a lot of memory.

JSONReader is designed for memory efficiency - it works on streams and can read JSON data from any PHP stream without loading the entire data into memory. It also allows the developer to extract specific values from a JSON stream without decoding and loading all data into memory.

Answer 2

Best possible solution:

Use some sort of delimiter (pagination, timestamp, object ID etc) that allows you to read the data in smaller chunks over multiple requests. This solution assumes that you have some sort of control of how these JSON files are generated. I'm basing my assumption on:

This would be fine except for the fact that my JSON files will usually range from 250MB-1GB+.

Reading in and processing 1GB of JSON data is simply ridiculous. A better approach is most definitely needed.

Answer 3

This really depends on what the json files contain.

If opening the file one shot into memory is not an option, your only other option, as you eluded to, is fopen/fgets.

Reading line by line is possible, and if these json objects have a consistent structure, you can easily detect when a json object in a file starts, and ends.

Once you collect a whole object, you insert it into a db, then go on to the next one.

There isn't much more to it. the algorithm to detect the beginning and end of a json object may get complicating depending on your data source, but I hvae done something like this before with a far more complex structure (xml) and it worked fine.