Why doesn't Python's mmap work with large files?

Question

[Edit: This problem applies only to 32-bit systems. If your computer, your OS and your python implementation are 64-bit, then mmap-ing huge files works reliably and

Answer 1

Use a 64-bit computer, with a 64-bit OS and a 64-bit python implementation, or avoid mmap()

mmap() requires CPU hardware support to make sense with large files bigger than a few GiB.

It uses the CPU's MMU and interrupt subsystems to allow exposing the data as if it were already loaded ram.

The MMU is hardware which will generate an interrupt whenever an address corresponding to data not in physical RAM is accessed, and the OS will handle the interrupt in a way that makes sense at runtime, so the accessing code never knows (or needs to know) that the data doesn't fit in RAM.

This makes your accessing code simple to write. However, to use mmap() this way, everything involved will need to handle 64 bit addresses.

Or else it may be preferable to avoid mmap() altogether and do your own memory management.