sparse-matrix

I have a badly conditioned matrix, whose rcond() is close to zero, and therefore, the inverse of that matrix does not come out to be correct. I have tried using pinv() but that does not solve the problem. This is how I am taking the inverse: X = (A)\(b); I looked up for a solution to this problem and found this link (last solution) for improving the matrix. The solution there suggests to use this: A_new = A_old + c*eye(size(A_old)); Where c > 0 . So far employing this technique works in making the matrix A better conditioned and the resultant solution looks better. However, I investigated

I am trying to apply PCA on huge sparse matrix, in the following link it says that randomizedPCA of sklearn can handle sparse matrix of scipy sparse format. Apply PCA on very large sparse matrix However, I always get error. Can someone point out what I am doing wrong. Input matrix 'X_train' contains numbers in float64: >>>type(X_train) <class 'scipy.sparse.csr.csr_matrix'> >>>X_train.shape (2365436, 1617899) >>>X_train.ndim 2 >>>X_train[0] <1x1617899 sparse matrix of type '<type 'numpy.float64'>' with 81 stored elements in Compressed Sparse Row format> I am trying to do: >>>from sklearn

I have a NxM matrix and I want to compute the NxN matrix of Euclidean distances between the M points. In my problem, N is about 100,000. As I plan to use this matrix for a k-nearest neighbor algorithm, I only need to keep the k smallest distances, so the resulting NxN matrix is very sparse. This is in contrast to what comes out of dist() , for example, which would result in a dense matrix (and probably storage problems for my size N ). The packages for kNN that I've found so far ( knnflex , kknn , etc) all appear to use dense matrices. Also, the Matrix package does not offer a pairwise

I have a matrix of factors in R and want to convert it to a matrix of dummy variables 0-1 for all possible levels of each factors. However this "dummy" matrix is very large (91690x16593) and very sparse. I need to store it in a sparse matrix, otherwise it does not fit in my 12GB of ram. Currently, I am using the following code and it works very fine and takes seconds: library(Matrix) X_factors <- data.frame(lapply(my_matrix, as.factor)) #encode factor data in a sparse matrix X <- sparse.model.matrix(~.-1, data = X_factors) However, I want to use the e1071 package in R, and eventually save this

I have a Sqlite database that contains following type of schema: termcount(doc_num, term , count) This table contains terms with their respective counts in the document. like (doc1 , term1 ,12) (doc1, term 22, 2) . . (docn,term1 , 10) This matrix can be considered as sparse matrix as each documents contains very few terms that will have a non-zero value. How would I create a dense matrix from this sparse matrix using numpy as I have to calculate the similarity among documents using cosine similarity. This dense matrix will look like a table that have docid as the first column and all the terms