mmap problem, allocates huge amounts of memory

Question

I got some huge files I need to parse, and people have been recommending mmap because this should avoid having to allocate the entire file in-memory.

But looking at

Answer 1

You need to specify a size smaller than the total size of the file in the mmap call, if you don't want the entire file mapped into memory at once. Using the offset parameter, and a smaller size, you can map in "windows" of the larger file, one piece at a time.

If your parsing is a single pass through the file, with minimal lookback or look-forward, then you won't actually gain anything by using mmap instead of standard library buffered I/O. In the example you gave of counting the newlines in the file, it'd be just as fast to do that with fread(). I assume that your actual parsing is more complex, though.

If you need to read from more than one part of the file at a time, you'll have to manage multiple mmap regions, which can quickly get complicated.