Best way of writing Big JSON files avoiding OutOfMemory issues

问题

First, please note today is my first day with GSON. I am trying to write a Json file using GSON library. I have thousands of JsonObjects inside an ArrayList. When the Json file is written, it should look like similar to this.

[
    {
        "hash_index": "00102x05h06l0aj0dw",
        "body": "Who's signing up for Obamacare?",
        "_type": "ArticleItem",
        "title": "Who's signing up for Obamacare? - Jan. 13, 2014",
        "source": "money.cnn.com",
        "primary_key": 0,
        "last_crawl_date": "2014-01-14",
        "url": "http://money.cnn.com/2014/01/13/news/economy/obamacare-enrollment/index.html"
    },
    {
        "hash_index": "00102x05h06l0aj0dw0iz0kn0l@0t#0",
        "body": "Who's signing up for Obamacare?",
        "_type": "ArticleItem",
        "title": "Who's signing up for Obamacare? - Jan. 13, 2014",
        "source": "money.cnn.com",
        "primary_key": 1,
        "last_crawl_date": "2014-01-14",
        "url": "http://money.cnn.com/2014/01/13/news/economy/obamacare-enrollment/index.html"
    }
]

Right now, I write the JSOn by using the below code.

 private void writeNewJsonFile() throws IOException
    {
        System.out.println("Starting to write the JSON File");
        //Add everything into a JSONArray
        JsonArray jsonArrayNew = new JsonArray();

        for(int i=0;i<jsonObjectHolder.size();i++)
        {
            System.out.println("inside array");
            jsonArrayNew.add(jsonObjectHolder.get(i));
        }


        //Write it to the File
    /*  File file= new File("items_Articles_4_1.json");

        FileWriter fw = new FileWriter(file);;
        fw.write(jsonArrayNew.toString());
        fw.flush();
        fw.close();*/

        System.out.println("outside array");

        ByteArrayInputStream input = new ByteArrayInputStream(jsonArrayNew.toString().getBytes());

        Long contentLength = Long.valueOf(jsonArrayNew.toString().getBytes().length);

        ObjectMetadata metaData = new ObjectMetadata();
        metaData.setContentLength(contentLength);

        s3.putObject(outputBucket,outputFile,input,metaData);


    }

Here I am converting the JsonArray into a String and do the writing. I have a fear that this will soon crash with Big Json arrays and give me the OutOfMemoryException. Just like I read the Json files part by part using GSON, is there is any way I can write the Json file piece by piece or something, which can avoid the OutOfMemoryException issues?

回答1:

I am using next code:

WriteJsonArrayByParts<Cache> write = new WriteJsonArrayByParts<Cache>(fileNameTest, " ");
write.writeStart();
for(Cache cache : listOfObjects()) {
    write.writeObject(cache, Cache.class);
}
write.writeEnd();
write.close();

...

public static class WriteJsonArrayByParts<T> {
    Gson gson = new Gson();
    JsonWriter writer;

    public WriteJsonArrayByParts(String fileNameWithPath, String indent) throws Exception {
        OutputStream os = new FileOutputStream(fileNameWithPath, false);
        BufferedOutputStream osb = new BufferedOutputStream(os, 8 * 1024);

        writer = new JsonWriter(new OutputStreamWriter(osb, StringUtil.UTF_8));
        writer.setIndent(indent);
    }

    public void writeStart() throws IOException {
        writer.beginArray();
    }

    @SuppressWarnings("unchecked")
    public void writeObject(T t, Class<?> resultClass) throws IOException {
        ((TypeAdapter<Object>) gson.getAdapter(resultClass)).write(writer, t);
    }

    public void writeEnd() throws IOException {
        writer.endArray();
    }

    public void close() throws IOException {
        writer.close();
    }
}

来源：https://stackoverflow.com/questions/21553285/best-way-of-writing-big-json-files-avoiding-outofmemory-issues