sparse-matrix

R: sparse matrix conversion

不羁岁月 提交于 2019-11-30 11:08:53
问题 I have a matrix of factors in R and want to convert it to a matrix of dummy variables 0-1 for all possible levels of each factors. However this "dummy" matrix is very large (91690x16593) and very sparse. I need to store it in a sparse matrix, otherwise it does not fit in my 12GB of ram. Currently, I am using the following code and it works very fine and takes seconds: library(Matrix) X_factors <- data.frame(lapply(my_matrix, as.factor)) #encode factor data in a sparse matrix X <- sparse.model

Determining the byte size of a scipy.sparse matrix?

自古美人都是妖i 提交于 2019-11-30 08:16:01
Is it possible to determine the byte size of a scipy.sparse matrix? In NumPy you can determine the size of an array by doing the following: import numpy as np print(np.zeros((100, 100, 100).nbytes) 8000000 A sparse matrix is constructed from regular numpy arrays, so you can get the byte count for any of these just as you would a regular array. If you just want the number of bytes of the array elements: >>> from scipy.sparse import csr_matrix >>> a = csr_matrix(np.arange(12).reshape((4,3))) >>> a.data.nbytes 88 If you want the byte counts of all arrays required to build the sparse matrix, then

What is the fastest way to slice a scipy.sparse matrix?

生来就可爱ヽ(ⅴ<●) 提交于 2019-11-30 06:50:55
I normally use matrix[:, i:] It seems not work as fast as I expected. Saullo G. P. Castro If you want to obtain a sparse matrix as output the fastest way to do row slicing is to have a csr type, and for columns slicing csc , as detailed here . In both cases you just have to do what you are currently doing: matrix[l1:l2,c1:c2] If you want another type as output there maybe faster ways. In this other answer it is explained many methods for slicing a matrix and their different timings compared. For example, if you want a ndarray as output the fastest slicing is: matrix.A[l1:l2,c1:c2] or: matrix

Is there support for sparse matrices in Python?

▼魔方 西西 提交于 2019-11-30 06:39:18
Is there support for sparse matrices in python? Possibly in numpy or in scipy? Steve Tjoa Yes. SciPi provides scipy.sparse , a "2-D sparse matrix package for numeric data". There are seven available sparse matrix types: csc_matrix: Compressed Sparse Column format csr_matrix: Compressed Sparse Row format bsr_matrix: Block Sparse Row format lil_matrix: List of Lists format dok_matrix: Dictionary of Keys format coo_matrix: COOrdinate format (aka IJV, triplet format) dia_matrix: DIAgonal format CVXOPT - Sparse matrices Pysparse 来源: https://stackoverflow.com/questions/4623800/is-there-support-for

Efficient way to normalize a Scipy Sparse Matrix

痴心易碎 提交于 2019-11-30 06:03:51
I'd like to write a function that normalizes the rows of a large sparse matrix (such that they sum to one). from pylab import * import scipy.sparse as sp def normalize(W): z = W.sum(0) z[z < 1e-6] = 1e-6 return W / z[None,:] w = (rand(10,10)<0.1)*rand(10,10) w = sp.csr_matrix(w) w = normalize(w) However this gives the following exception: File "/usr/lib/python2.6/dist-packages/scipy/sparse/base.py", line 325, in __div__ return self.__truediv__(other) File "/usr/lib/python2.6/dist-packages/scipy/sparse/compressed.py", line 230, in __truediv__ raise NotImplementedError Are there any reasonably

scipy.sparse dot extremely slow in Python

℡╲_俬逩灬. 提交于 2019-11-30 05:44:57
问题 The following code will not even finish on my system: import numpy as np from scipy import sparse p = 100 n = 50 X = np.random.randn(p,n) L = sparse.eye(p,p, format='csc') X.T.dot(L).dot(X) Is there any explanation why this matrix multiplication is hanging? 回答1: X.T.dot(L) is not, as you may think, a 50x100 matrix, but an array of 50x100 sparse matrices of 100x100 >>> X.T.dot(L).shape (50, 100) >>> X.T.dot(L)[0,0] <100x100 sparse matrix of type '<type 'numpy.float64'>' with 100 stored

sparse matrix svd in python

谁说我不能喝 提交于 2019-11-30 05:42:41
Does anyone know how to perform svd operation on a sparse matrix in python? It seems that there is no such functionality provided in scipy.sparse.linalg. You can use the Divisi library to accomplish this; from the home page: It is a library written in Python, using a C library (SVDLIBC) to perform the sparse SVD operation using the Lanczos algorithm. Other mathematical computations are performed by NumPy. Sounds like sparsesvd is what you're looking for! SVDLIBC efficiently wrapped in Python (no extra data copies made in RAM). Simply run "easy_install sparsesvd" to install. You can try scipy

How to convert sparse matrix to dense matrix in Eigen

与世无争的帅哥 提交于 2019-11-30 04:57:24
Is there some easy and fast way to convert a sparse matrix to a dense matrix of doubles? Because my SparseMatrix is not sparse any more, but became dense after some matrix products. Another question I have: The Eigen library has excellent performance, how is this possible? I don't understand why, because there are only header files, no compiled source. ggael Let's declare two matrices: SparseMatrix<double> spMat; MatrixXd dMat; Sparse to dense: dMat = MatrixXd(spMat); Dense to sparse: spMat = dMat.sparseView(); 来源: https://stackoverflow.com/questions/15484622/how-to-convert-sparse-matrix-to

python matplotlib plot sparse matrix pattern

北城余情 提交于 2019-11-30 04:08:52
问题 Given a sparse binary matrix A (csr, coo, whatever) I want to make a plot such that I can see the position (i,j) = white in the figure if A(i,j) = 1, and (i,j) = black if A(i,j) = 0; For a dense numpy array, matshow will do the job. However, the dimension of my sparse matrix (say 100000 x 1000000) is to big to be converted to a dense array. I wonder how could I plot the pattern in my sparse matrix. Thanks 回答1: You can get a nice result using a coo_matrix , plot() and some adjustments: import

Get constraints in matrix format from gurobipy

♀尐吖头ヾ 提交于 2019-11-30 04:02:58
I coded my model in gurobipy and I want to get the matrix of constraints and vector of cost. Is there any way to access those? From the python API, there's no single function to get the matrix coefficients from a Gurobi model, but it's not to hard to write one yourself. It is convenient to have lists of your variables and constraints. If you have a gurobi model in variable m dvars = m.getVars() constrs = m.getConstrs() will give you the list of variables and constraints. You can then use m.getAttr to retrieve attributes related to the variables. To obtain the objective function coefficients,