“OutOfMemoryError: GC overhead limit exceeded”: parse large json file with java

Question

I try to parse large json file (more 600Mo) with Java. My json file look like that:

{
    \"0\" : {\"link_id\": \"2381317\", \"overview\": \"mjklmk         


        

Answer 1

If you have to read huge JSON Files you can't mantain in memory all informations. Extending memory can be a solution for a file of 1 Gb. If the files tomorrow is a 2 Gb Files?

The right approach to this problem is to parse the json element by element using a streaming parser. Basically instead of loading the whole json in memory and creating a whole big object representing it you need to read single elements of the json and converting them to objects step by step.

Here you find a nice article explaing how to do it with jackson library.