Byte[] and java.lang.OutOfMemoryError reading file by bits

Question

I am trying to write a reader which reads files by bits but I have a problem with large files. I tried to read file with 100 mb and it took over 3 minutes but it worked.

Answer 1

I suggest you try memory mapping.

FileChannel fc = new FileInputStream(file).getChannel();
MappedByteBuffer mbb = fc.map(FileChannel.MapMode.READ_ONLY, 0, (int) fc.size());

This will make the whole file available almost immediately (about 10 ms) and uses next to no heap. BTW The file has to be less than 2 GB.

Answer 2

I suggest you use a RandomAccessFile.

Answer 3

If you really need to load the entire file to memory at once, I can only suggest increasing the memory available to Java. Try invoking your program using the Xmx (max heap size) or Xms (initial heap size) arguments (use the latter if you know beforehand how much memory you'll need, otherwise the former might be best).

java -Xms512m -Xmx1g BigApp

As an alternative, you can use NIO's Memory-mapped files.

Answer 4

You shouldn't open the whole file into memory. You need to create a byte array buffer with a fixed size, then you open your file from chunks of the size you defined.