Not really as a solution, but just out of interest, one way to do it like this:
- First break the file down into 1GB files
- Then reading 2 files at a time, load the contents into a list of string and sort it
- Write it back down to the individual files.
The problem is that you would need to read/write 100 files on each pass and do 100 passes to make sure that the data is sorted.
If my maths is correct: That is 10 000 GB read and 10 000 GB write, at an average 10MB/sec that is 20 000 000 sec which is 231 days
One way that is might work is that you scan the file once and write to smaller files, one for each time period for example day or hour. Then sort these individual files.
