java outOfMemoryError with stringbuilder

问题

I'm getting a java outOfMemoryError when I call this method - i'm using it in a loop to parse many large files in sequence. my guess is that result.toString() is not getting garbage collected properly during the loop. if so, how should i fix it?

private String matchHelper(String buffer, String regex, String method){
    Pattern abbrev_p = Pattern.compile(regex);//norms U.S.A., B.S., PH.D, PH.D.
    Matcher abbrev_matcher = abbrev_p.matcher(buffer);
    StringBuffer result = new StringBuffer();
    while (abbrev_matcher.find()){
            abbrev_matcher.appendReplacement(result, abbrevHelper(abbrev_matcher));
    }
    abbrev_matcher.appendTail(result);
    String tempResult = result.toString(); //ERROR OCCURS HERE
  return tempResult;

}

回答1:

Written this way, you'll need roughly 6 bytes of memory for every character in the file.

Each character is two bytes. You have the raw input, the substituted output (in the buffer), and you are asking for a third copy when you run out of memory.

If the file is encoded in something like ASCII or ISO-8859-1 (a single-byte character encoding), that means it will be six times larger in memory than on disk.

You could allocate more memory to the process, but a better solution might be to process the input "streamwise"—read, scan, and write the data without loading it all into memory at once.

回答2:

If your files to be processed are all very large, say more than a few hundred MB, then you really should go with stream processing instead of this "loading all into memory" way, just as @erickson suggested.

Otherwise, there are a few things you could try, all to reduce memory usage as much as possible:

Try properly enlarge your heap size if not yet (when applicable).
Give StringBuffer an initial size same as the lenght of the given String buffer. This should reduce the unnecessary memory usage while expanding the StringBuffer in the process. I assume it is only replacing certain words of the original string and should be more or less the same in length.
If possible, maybe you could return the generated StringBuffer object instead. Calling its toString() only after you get rid of the original String object.

回答3:

I reckon the problem with StringBuilder.append(). When Matcher appends sequence of characters to the Builder.

As explained in article about OutOfMemoryError with StringBuilder/StringBuffer, it is a known issue that append() will double the capacity if internal buffer chars if the capacity is not sufficient. Go for streams as suggested by Erickson.

回答4:

I agree with the other responses ... but ... simply because the exception occurs there doesn't necessarily mean it's the problem. You may very well be leaking memory elsewhere and that just happens to be the place that it's revealed. You should run a profiler to examine memory usage and verify exactly what objects aren't being collected.