What is the fastest way to read a large number of small files into memory?

Question

I need to read ~50 files on every server start and place each text file\'s representation into memory. Each text file will have its own string (which is the best type to use

Answer 1

The most efficient way is:

• Determine the length of the file (File.length())
• Create a char buffer with the same size (or slightly larger)
• Determine the encoding of the file
• Use new InputStreamReader (new FileInputStream(file), encoding) to read
• Read the while file into the buffer with a single call to read(). Note that read() might return early (not having read the whole file). In that case, call it again with an offset to read the next batch.
• Create the string: new String(buffer)

If you need to search&replace once at startup, use String.replaceAll().

If you need to do it repeatedly, you may consider using StringBuilder. It has no replaceAll() but you can use it to manipulate the character array in place (-> no allocation of memory).

That said:

1. Make your code as short and simple as possible.
2. Measure the performance
3. It it's too slow, fix it.

There is no reason to waste a lot of time into making this code run fast if it takes just 0.1s to execute.

If you still have a performance problem, consider to put all the text files into a JAR, add it into the classpath and use Class.getResourceAsStream() to read the files. Loading things from the Java classpath is highly optimized.