Alternatives to keeping large lists in memory (python)

Question

If I have a list(or array, dictionary....) in python that could exceed the available memory address space, (32 bit python) what are the options and there relative speeds? (o

Answer 1

Did you check shelve python module which is based on pickle?

http://docs.python.org/library/shelve.html

Answer 2

Well, if you are looking for speed and your data is numerical in nature, you could consider using numpy and PyTables or h5py. From what I remember, the interface is not as nice as simple lists, but the scalability is fantastic!

Answer 3

Modern operating systems will handle this for you without you having to worry about it. It's called virtual memory.