Storage of many log files

Question

I have a system which is receiving log files from different places through http (>10k producers, 10 logs per day, ~100 lines of text each).

I would like to store them t

Answer 1

Since you would like to store them to be able to compute misc. statistics over them nightly , export them (ordered by date of arrival or first line content) ... You're expecting 100,000 files a day, at a total of 10,000,000 lines:

I'd suggest:

1. Store all the files as regular textfiles using the following format : yyyymmdd/producerid/fileno.
2. At the end of the day, clear the database, and load all the textfiles for the day.
3. After loading the files, it would be easy to get the stats from the database, and post them in any format needed. (maybe even another "stats" database). You could also generate graphs.
4. To save space ,you could compress the daily folder. Since they're textfiles, they would compress well.

So you would only be using the database to be able to easily aggregate the data. You could also reproduce the reports for an older day if the process didn't work, by going through the same steps.