问题
I am computing the solution of a linear system Ax=b with A a large (typically 200,000 lines and columns for the associated dense matrix) sparse matrix and b a sparse matrix of about 100 columns.
When I run my code on Windows systems (Python 2.7, scipy 0.14.0), the following command
from scipy.sparse.linalg import spsolve
...
Temp = spsolve(A.tocsc(),b.tocsc())
runs smoothly and requires about 7 GB of ram memory.
Running the exact same code with the exact same matrices on Linux systems (same CPU, same amount of RAM memory: 64 GB, Linux Mint 17.3, python 2.7, scipy 0.13.3) requires more than 20 GB of ram memory and it crashes with the following error message:
<function umfpack_di_numeric at ...> failed with UMFPACK_ERROR_out_of_memory (see 1)
Because this error is os dependent, I ruled out any issue regarding the matrices A and b (contrary to some of the mentioned solutions in this post), and I am trying to find a fix specific to Linux... But I do not know where to start... Does anyone would have any idea of what is going on ? And why would such a problem be specific to Linux systems ?
Please find below the full error message:
Exception in Tkinter callback
Traceback (most recent call last):
File "/usr/lib/python2.7/lib-tk/Tkinter.py", line 1489, in __call__
return self.func(*args)
File "...", line 1533, in mmvConstruction
...
File "...", line 1555, in modes_cb
Temp = spsolve(k[inter][:,inter].tocsc(),k[inter][:,exter].tocsc())
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 151, in spsolve
Afactsolve = factorized(A)
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/linsolve.py", line 352, in factorized
umf.numeric(A)
File "/usr/lib/python2.7/dist-packages/scipy/sparse/linalg/dsolve/umfpack/umfpack.py", line 450, in numeric
umfStatus[status]))
RuntimeError:<function umfpack_di_numeric at ...> failed with UMFPACK_ERROR_out_of_memory
Update: still trying to find out a solution... it seems that the latest version of BLAS on Linux Mint is quite old: 1.8.2. On Windows, I use BLAS 1.9.1. When using the test_numpy.py file available here: https://gist.github.com/osdf/3842524#file-test_numpy-py I notice very significant differences between Linux and Windows: Linux : version 1.8.2, maxint 9223372036854775807, dot: 0.76 s - Windows : version 1.9.1, maxint 2147483647, dot: 0,037 s. I am investigating if OPENBLAS on Linux could be a solution to this problem...
Update 2: I realized that the problem may be hardware related. Indeed, an older PC, with the exact same libraries on the same Linux Mint distribution (Rosa 17.3) provides much more satisfying results. The benchmark mentioned in the first update gives on this old PC: Linux : version 1.8.2, maxint 9223372036854775807, dot: 0,054 s.
回答1:
Well, after thorough investigations, I am now convinced that the issue I had is related to the fact that Linux Mint (Rosa 17.3) may not be optimized for the most recent processors.
The comparisons I mentioned in the updates of my post underlined that the software installation was correct. I then installed Fedora 23 on my PC, installing sequentially:
- libblas
- python
- python-scipy
Then I ran the exact same code with the exact same matrices and there was no issue whatsoever: RAM consumption was limited to about 7 GB, similarly to what was observed on Windows systems.
来源：https://stackoverflow.com/questions/36344049/scipy-sparse-linalg-spsolve-surprising-behaviour-for-large-sparse-matrices-on-li