Advice on handling large data volumes

Question

So I have a \"large\" number of \"very large\" ASCII files of numerical data (gigabytes altogether), and my program will need to process the entirety of it sequentially at l

Answer 1

I've found Informatica to be an exceptionally useful data processing tool. The good news is that the more recent versions even allow Java transformations. If you're dealing with terabytes of data, it might be time to pony up for the best-of-breed ETL tools.

I'm assuming you want to do something with the results of the processing here, like store it somewhere.