Java/Android: java.lang.OutOfMemoryError while building a JSON object

问题

I am importing JSON data from a public database URI http://data.seattle.gov/api/views/3k2p-39jp/rows.json and the rows go as far as 445454. Using the following code I am constructing the JSON object of the entire data.

   HttpGet get = new HttpGet(uri);
   HttpClient client = new DefaultHttpClient();
   HttpResponse response = client.execute(get);
   BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent(), "UTF-8"));
    StringBuilder builder=new StringBuilder();
for(String line=null;(line = reader.readLine()) != null;){
      builder.append(line).append("\n");
    }
  JSONTokener jsonTokener=new JSONTokener(builder.toString());
  JSONObject finalJson=new JSONObject(jsonTokener);
  JSONArray data=finalJson.getJSONArray("data");

Because the data is too large, i am getting 03-21 03:41:49.714: E/AndroidRuntime(666): Caused by: java.lang.OutOfMemoryError pointing the source of error at buildr.append(line).append("\n"). Is there anyway I can handle large datasets without getting memory allocation issues?

回答1:

That JSON is huge!

You definitely need to use a streaming JSON parser. There are two out there for Android: GSON and Jackson.

GSON Streaming is explained at: https://sites.google.com/site/gson/streaming

I like how GSON explains the problem you're having:

Most applications should use only the object model API. JSON streaming is useful in just a few situations:

When it is impossible or undesirable to load the entire object model into memory. This is most relevant on mobile platforms where memory is limited.

Jackson Streaming is documented at: http://wiki.fasterxml.com/JacksonInFiveMinutes#Streaming_API_Example

回答2:

If possible only request parts of the data - this also reduces time for network io and thus saves battery.

Otherwise you could try to not keep the incoming data in memory, but to 'stream' it onto the sd-card. When it is stored there you can then iterate over it. Most likely this will mean to use your own JSON tokenizer that does not build a full tree, but which is able to (like a SAX parser) only look at a part of the object tree at a time.

You may have a look at Jackson, which has a streaming mode, which may be applicable.

回答3:

Streaming pull parser is the way. I recommend GSON, as this has small memory footpring (just pull parsing is about 16K , jackson is way bigger)

Your code is problematic because you allocate:

buffer to hold all the string data coming from service
all the JSON DOM objects

and this is slow, and gives you memory meltdown.

In case you need java objects out of your JSON data , you may try my small databinding library building on GSON (shameles self advertising off):

https://github.com/ko5tik/jsonserializer

回答4:

I did it a bit differently, My JSON code was waiting for status, which comes towards the end. So I modified the code to return earlier.

// try to get formattedAddress without reading the entire JSON
        String formattedAddress;
        while ((read = in.read(buff)) != -1) {
            jsonResults.append(buff, 0, read);
            formattedAddress = ((String) ((JSONObject) new JSONObject(
                    jsonResults.toString()).getJSONArray("results").get(0))
                    .get("formatted_address"));
            if (formattedAddress != null) {
                Log.i("Taxeeta", "Saved memory, returned early from json") ;
                return formattedAddress;
            }               
        }
JSONObject statusObj = new JSONObject(jsonResults.toString());
        String status = (String) (statusObj.optString("status"));
        if (status.toLowerCase().equals("ok")) {
            formattedAddress = ((String) ((JSONObject) new JSONObject(
                    jsonResults.toString()).getJSONArray("results").get(0))
                    .get("formatted_address"));
            if (formattedAddress != null) {
                Log.w("Taxeeta", "Did not saved memory, returned late from json") ;
                return formattedAddress;
            }           
        }

来源：https://stackoverflow.com/questions/9799549/java-android-java-lang-outofmemoryerror-while-building-a-json-object

标签

java

android

json

webrequest