Large scale image storage

Question

I will likely be involved in a project where an important component is a storage for a large number of files (in this case images, but it should just act as a file storage).

Answer 1

Store the images in a series of SQLite databases. Sounds crazy at first but it seriously is faster than storing them on the file system directly, and take up less space.

SQLite is extremely efficient at storing binary data and by storing the files in an aggregated database instead of individual OS files it saves overhead when the images don't fit into exact block sizes (which is significant for this many files). Also the paged data in SQLite can give you faster throughput overall than you would get with plain OS files.

SQLite has concurrency limitations on writes but well within the limits you're talking about and can be mitigated even further by clever use of multiple (hundreds) of SQLite databases.

Try it out, you'll be pleasantly surprised.