Advice on handling large data volumes

Question

So I have a \"large\" number of \"very large\" ASCII files of numerical data (gigabytes altogether), and my program will need to process the entirety of it sequentially at l

Answer 1

You really haven't given us enough info to help you. Do you need to load each file in its entiretly in order to process it? Or can you process it line by line?

Loading an entire file at a time is likely to result in poor performance even for files that aren't terribly large. Your best bet is to define a buffer size that works for you and read/process the data a buffer at a time.