java.lang.OutOfMemoryError even though plenty of

问题

I am trying to read a 2.5GB txt file into my application. I am running Win7 x64 and have 43GB of mem available (out of 64GB). I tried playing around with -Xmx -XX:MaxParmSize -XX:ParmSize etc. None of these affect the error. What else could I try? This error seems very odd as I certainly have enough heap space available.

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
    at java.util.Arrays.copyOf(Unknown Source)
    at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
    at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
    at java.lang.AbstractStringBuilder.append(Unknown Source)
    at java.lang.StringBuilder.append(Unknown Source)
    at j.utilities.IO.loadString(IO.java:187)
    at j.utilities.IO.loadString(IO.java:169)
    at city.PreProcess.main(PreProcess.java:78)

I am running

java version "1.7.0_09"
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)

Thanks a lot in advance.

============== ANSWER ==============

OK, I just tested it with

StringBuilder sb = new StringBuilder();
for ( int i=1; i<Integer.MAX_VALUE; i++ )
    sb.append("x");

and got

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
at java.util.Arrays.copyOf(Unknown Source)
...

Thus, it really is StringBuilder which tries to build an array bigger than Integer.MAX_VALUE.

In case interested

StringBuilder sb = new StringBuilder();
int i=1;
try {
    for ( ; i<Integer.MAX_VALUE; i++ )
        sb.append("x");
} catch ( OutOfMemoryError e ) {
    System.out.println(i); // OUTPUT: 1207959551
    System.out.println(Integer.MAX_VALUE); // OUTPUT: 2147483647
}

With StringBuilder you can accumulate 1,207,959,550 chars - far less than Integer.MAX_VALUE.

回答1:

You're trying to allocate an array that is too large. This is because you're trying to create a very long String. Since arrays are indexed by an integer, an array cannot have more than Integer.MAX_VALUE elements. Even if the your heap size is very large, you won't be able to allocate an array that has more than Integer.MAX_VALUE elements, simply because you cannot index its elements using an Integer. See Do Java arrays have a maximum size? for more details.

回答2:

You could create a new StringBuilder with a size, e.g.

    StringBuilder sb = new StringBuilder(Integer.MAX_VALUE);

The problem is that you're trying to read a file that's larger than what StringBuilder can have in its array. You have several options, e.g.:

1) Do you really need to read the entire file into memory at once? If so, you'll have to read it into several StringBuilders.

2) Process the file sequentially.

3) Read it into a compressed structure, and uncompress the parts you need when you need them.

回答3:

You should look into the -Xmsn option for the java command.

It specifies the initial size of the memory allocation pool.

Edit: I see you've already done that.

回答4:

You can hold the data in string buffer data is List<String> in define interval and clear the StringBuffer.

来源：https://stackoverflow.com/questions/13772802/java-lang-outofmemoryerror-even-though-plenty-of

标签

java

out-of-memory