Numpy Cholesky decomposition LinAlgError

Question

In my attempt to perform cholesky decomposition on a variance-covariance matrix for a 2D array of periodic boundary condition, under certain parameter combinations, I always get LinAlgError: Matrix is not positive definite - Cholesky decomposition cannot be computed. Not sure if it's a numpy.linalg or implementation issue, as the script is straightforward:

sigma = 3.
U = 4

def FromListToGrid(l_):
    i = np.floor(l_/U)
    j = l_ - i*U
    return np.array((i,j))

Ulist = range(U**2)

Cov = []
for l in Ulist:
    di = np.array([np.abs(FromListToGrid(l)[0]-FromListToGrid(i)[0]) for i, x in enumerate(Ulist)])
    di = np.minimum(di, U-di)

    dj = np.array([np.abs(FromListToGrid(l)[1]-FromListToGrid(i)[1]) for i, x in enumerate(Ulist)])
    dj = np.minimum(dj, U-dj)

    d = np.sqrt(di**2+dj**2)
    Cov.append(np.exp(-d/sigma))
Cov = np.vstack(Cov)

W = np.linalg.cholesky(Cov)

Attempts to remove potential singularies also failed to resolve the problem. Any help is much appreciated.

Answer 1

Digging a bit deeper in problem, I tried printing the Eigenvalues of the Cov matrix.

print np.linalg.eigvalsh(Cov)

And the answer turns out to be this

[-0.0801339  -0.0801339   0.12653595  0.12653595  0.12653595  0.12653595 0.14847999  0.36269785  0.36269785  0.36269785  0.36269785  1.09439988 1.09439988  1.09439988  1.09439988  9.6772531 ]

Aha! Notice the first two negative eigenvalues? Now, a matrix is positive definite if and only if all its eigenvalues are positive. So, the problem with the matrix is not that it's close to 'zero', but that it's 'negative'. To extend @duffymo analogy, this is linear algebra equivalent of trying to take square root of negative number.

Now, let's try to perform same operation, but this time with scipy.

scipy.linalg.cholesky(Cov, lower=True)

And that fails saying something more

numpy.linalg.linalg.LinAlgError: 12-th leading minor not positive definite

That's telling something more, (though I couldn't really understand why it's complaining about 12-th minor).

Bottom line, the matrix is not quite close to 'zero' but is more like 'negative'

Answer 2

The problem is the data you're feeding to it. The matrix is singular, according to the solver. That means a zero or near-zero diagonal element, so inversion is impossible.

It'd be easier to diagnose if you could provide a small version of the matrix.

Zero diagonals aren't the only way to create a singularity. If two rows are proportional to each other then you don't need both in the solution; they're redundant. It's more complex than just looking for zeroes on the diagonal.

If your matrix is correct, you have a non-empty null space. You'll need to change algorithms to something like SVD.

See my comment below.