Upper memory limit?

Question

Is there a limit to memory for python? I\'ve been using a python script to calculate the average values from a file which is a minimum of 150mb big.

Depending on the

Answer 1

No, there's no Python-specific limit on the memory usage of a Python application. I regularly work with Python applications that may use several gigabytes of memory. Most likely, your script actually uses more memory than available on the machine you're running on.

In that case, the solution is to rewrite the script to be more memory efficient, or to add more physical memory if the script is already optimized to minimize memory usage.

Edit:

Your script reads the entire contents of your files into memory at once (line = u.readlines()). Since you're processing files up to 20 GB in size, you're going to get memory errors with that approach unless you have huge amounts of memory in your machine.

A better approach would be to read the files one line at a time:

for u in files:
     for line in u: # This will iterate over each line in the file
         # Read values from the line, do necessary calculations