What is the fastest way to read a large number of small files into memory?

Question

I need to read ~50 files on every server start and place each text file\'s representation into memory. Each text file will have its own string (which is the best type to use

Answer 1

Any conventional approach is going to be limited in speed. I'm not sure you'll see much of a difference from one approach to the next.

I would concentrate on business tricks that could make the entire operation faster.

For instance, if you read all the files and stored them in a single file with the timestamps from each of your original file, then you could check to see if any of the files have changed without actually opening them. (a simple cache, in other words).

If your problem was getting a GUI up quickly, you might find a way to open the files in a background thread after your first screen was displayed.

The OS can be pretty good with files, if this is part of a batch process (no user I/O), you could start with a batch file that appends all the files into one big one before launching java, using something like this:

echo "file1" > file.all
type "file1" >> file.all
echo "file2" >> file.all
type "file2" >> file.all

Then just open file.all (I'm not sure how much faster this will be, but it's probably the fastest approach for the conditions I just stated)

I guess I'm just saying that more often than not, a solution to a speed issue often requires expanding your viewpoint a little and completely rethinking the solution using new parameters. Modifications of an existing algorithm usually only give minor speed enhancements at the cost of readability.