How do you deal with lots of small files?

Question

A product that I am working on collects several thousand readings a day and stores them as 64k binary files on a NTFS partition (Windows XP). After a year in production the

Answer 1

To create a folder structure that will scale to a large unknown number of files, I like the following system:

Split the filename into fixed length pieces, and then create nested folders for each piece except the last.

The advantage of this system is that the depth of the folder structure only grows as deep as the length of the filename. So if your files are automatically generated in a numeric sequence, the structure is only is deep is it needs to be.

12.jpg -> 12.jpg
123.jpg -> 12\123.jpg
123456.jpg -> 12\34\123456.jpg

This approach does mean that folders contain files and sub-folders, but I think it's a reasonable trade off.

And here's a beautiful PowerShell one-liner to get you going!

$s = '123456'

-join  (( $s -replace '(..)(?!$)', '$1\' -replace '[^\\]*$','' ), $s )