How do you deal with lots of small files?

Question

A product that I am working on collects several thousand readings a day and stores them as 64k binary files on a NTFS partition (Windows XP). After a year in production the

Answer 1

Having hundreds of thousands of files in a single directory will indeed cripple NTFS, and there is not really much you can do about that. You should reconsider storing the data in a more practical format, like one big tarball or in a database.

If you really need a separate file for each reading, you should sort them into several sub directories instead of having all of them in the same directory. You can do this by creating a hierarchy of directories and put the files in different ones depending on the file name. This way you can still store and load your files knowing just the file name.

The method we use is to take the last few letters of the file name, reversing them, and creating one letter directories from that. Consider the following files for example:

1.xml
24.xml
12331.xml
2304252.xml

you can sort them into directories like so:

data/1.xml
data/24.xml
data/1/3/3/12331.xml
data/2/5/2/4/0/2304252.xml

This scheme will ensure that you will never have more than 100 files in each directory.