Fast & Efficient Way To Read Large JSON Files Line By Line in Java

Question

I have 100 millions of records in JSON file, need an efficient and fastest method to read the array of arrays from a JSON file in java.

JSON file look like:

[["XYZ",...,"ABC"],["XYZ",...,"ABC"],["XYZ",...,"ABC"],...,["XYZ",...,"ABC"],
 ["XYZ",...,"ABC"],["XYZ",...,"ABC"],["XYZ",...,"ABC"],...,["XYZ",...,"ABC"],
 ...
 ...
 ...
 ,["XYZ",...,"ABC"],["XYZ",...,"ABC"],["XYZ",...,"ABC"]]

I want to read this JSON file line by line as:

read first:

["XYZ",...,"ABC"]

then:

["XYZ",...,"ABC"]

so on:'

...
...
...
["XYZ",...,"ABC"]

How do I read a JSON file like this, I know it does not completely look like a JSON file but I need to read this file in this format which is saved as.JSON

Answer 1

You can use JSON Processing API (JSR 353), to process your data in a streaming fashion:

import javax.json.Json;
import javax.json.stream.JsonParser;

...

String dataPath = "data.json";

try(JsonParser parser = Json.createParser(new FileReader(dataPath))) {
     List<String> row = new ArrayList<>();

     while(parser.hasNext()) {
         JsonParser.Event event = parser.next();
         switch(event) {
             case START_ARRAY:
                 continue;
             case VALUE_STRING:
                 row.add(parser.getString());
                 break;
             case END_ARRAY:
                 if(!row.isEmpty()) {
                     //Do something with the current row of data 
                     System.out.println(row);

                     //Reset it (prepare for the new row) 
                     row.clear();
                 }
                 break;
             default:
                 throw new IllegalStateException("Unexpected JSON event: " + event);
         }
     }
}

Answer 2

Please take a look on Jackson Streaming API,

I guess you are looking something like this - https://www.ngdata.com/parsing-a-large-json-file-efficiently-and-easily/

and this - https://stackoverflow.com/a/24838392/814304

Main thing - if you have a big file you need to read and process file lazy, piece by piece.

Answer 3

You can use JsonSurfer to extract all inner JSON array by a JsonPath: $[*]

    JsonSurfer surfer = JsonSurferJackson.INSTANCE;
    surfer.configBuilder().bind("$[*]", new JsonPathListener() {
        @Override
        public void onValue(Object value, ParsingContext context) {
            System.out.println(value);
        }
    }).buildAndSurf(json);

It won't load entire Json into memory. JSON array will be processed one by one.