Advice on handling large data volumes

Question

So I have a \"large\" number of \"very large\" ASCII files of numerical data (gigabytes altogether), and my program will need to process the entirety of it sequentially at l

Answer 1

You might want to have a look at the entries in the Wide Finder Project (do a google search for "wide finder" java).

The Wide finder involves reading over lots of lines in log files, so look at the Java implementations and see what worked and didn't work there.