问题
I have a program for learning Artificial Neural Network and it takes a 2-d numpy array as training data. The size of the data array I want to use is around 300,000 x 400 floats. I can't use chunking here because the library I am using (DeepLearningTutorials) takes a single numpy array as training data.
The code shows MemoryError when the RAM usage is around 1.6Gb by this process(I checked it in system monitor) but I have a total RAM of 8GB. Also, the system is Ubuntu-12.04 32-bit.
I checked for the answers ofor other similar questions but somewhere it says that there is nothing like allocating memory to your python program and somewhere the answer is not clear as to how to increase the process memory.
One interesting thing is I am running the same code on a different machine and it can take a numpy array of almost 1,500,000 x 400 floats without any problem. The basic configurations are similar except that the other machine is 64-bit and this one is 32-bit.
Could someone please give some theoretical answer as to why there is so much difference in this or is this the only reason for my problem?
回答1:
A 32-bit OS can only address up to aroung 4gb of ram, while a 64-bit OS can take advantage of a lot more ram (theoretically 16.8 million terabytes). Since your OS is 32-bit, your OS can only take advantage of 4gb, so your other 4gb isn't used.
The other 64-bit machine doesn't have the 4gb ram limit, so it can take advantage of all of its installed ram.
These limits come from the fact that a 32-bit machine can only store memory address (pointers) of 32-bytes, so there are 2^32 different possible memory locations that the computer can identify. Similarly, a 64-bit machine can identify 2^64 different possible memory locations, so it can address 2^64 different bytes.
来源：https://stackoverflow.com/questions/17756791/python-process-using-only-1-6-gb-ram-ubuntu-32-bit-in-numpy-array