TStringList of objects taking up tons of memory in Delphi XE

Question

I\'m working on a simulation program.

One of the first things the program does is read in a huge file (28 mb, about 79\'000 lines,), parse each line (about 150 field

Answer 1

I also read in a lot of strings in my program that can approach a couple of GB for large files.

Short of waiting for 64-bit XE2, here is one idea that might help you:

I found storing individual strings in a stringlist to be slow and wasteful in terms of memory. I ended up blocking the strings together. My input file has logical records, which may contain between 5 and 100 lines. So instead of storing each line in the stringlist, I store each record. Processing a record to find the line I need adds very little time to my processing, so this is possible for me.

If you don't have logical records, you might just want to pick a blocking size, and store every (say) 10 or 100 strings together as one string (with a delimiter separating them).

The other alternative, is to store them in a fast and efficient on-disk file. The one I'd recommend is the open source Synopse Big Table by Arnaud Bouchez.