JSON parser read an entry by entry from large JSON file

问题

I have a huge JSON file (1GB) which is basically an array of objects in the below format

[{"x":"y", "p":"q"}, {"x1":"y1", "p1":"q1"},....]

I want to parse this file such the all the data is not loaded in memory.
Basically I want to get for eg: first 1000 objects in the array to memory process it and then get the next 1000 objects into the memory process it and so on util all data is read.
Is there any JSON library that supports this use case? I currently use Gson. However it loads all the data to memory when I call gson.fromJson()

Thanks in advance for the help.

回答1:

It looks like Gson has a streaming API, which is what you want: https://sites.google.com/site/gson/streaming

回答2:

With Jackson you can use a SAX-like approach (streaming) using a JsonParser object, in your case it would be something like this:

JsonFactory jsonFactory = new JsonFactory();
JsonParser parser = jsonFactory.createParser(new File("/path/to/my/jsonFile"));

// Map where to store your field-value pairs per object
Map<String, String> fields = new HashMap<String, String>();

JsonToken token;
while ((token = parser.nextToken()) != JsonToken.END_ARRAY) {
    switch (token) {

        // Starts a new object, clear the map
        case START_OBJECT:
            fields.clear();
            break;

        // For each field-value pair, store it in the map 'fields'
        case FIELD_NAME:
            String field = parser.getCurrentName();
            token = parser.nextToken();
            String value = parser.getValueAsString();
            fields.put(field, value);
            break;

        // Do something with the field-value pairs
        case END_OBJECT:
            doSomethingWithTheObject(fields)
            break;
        }
    }
    parser.close();

来源：https://stackoverflow.com/questions/18230744/json-parser-read-an-entry-by-entry-from-large-json-file