sparse-matrix

问题 My data can be viewed as a matrix of 10B entries (100M x 100), which is very sparse (< 1/100 * 1/100 of entries are non-zero). I would like to feed the data into into a Keras Neural Network model which I have made, using a Tensorflow backend. My first thought was to expand the data to be dense, that is, write out all 10B entries into a series of CSVs, with most entries zero. However, this is quickly overwhelming my resources (even doing the ETL overwhelmed pandas and is causing postgres to

问题 alpha = csr_matrix((1000,1000),dtype=np.float32) beta = csr_matrix((1,1000),dtype=np.float32) alpha[0,:] = beta After initiation, alpha and beta should be sparse matrixes with no element stored there. But after assigning beta to the first row of alpha, alpha become non-sparse, with 1000 zeros stored in alpha. I know I can use eliminate_zeros() to turn alpha back to sparse matrix but is there any better way to do this? 回答1: When I copy your steps I get In [131]: alpha[0,:]=beta /usr/lib

问题 I've got a large matrix stored as a scipy.sparse.csc_matrix and want to subtract a column vector from each one of the columns in the large matrix. This is a pretty common task when you're doing things like normalization/standardization, but I can't seem to find the proper way to do this efficiently. Here's an example to demonstrate: # mat is a 3x3 matrix mat = scipy.sparse.csc_matrix([[1, 2, 3], [2, 3, 4], [3, 4, 5]]) #vec is a 3x1 matrix (or a column vector) vec = scipy.sparse.csc_matrix([1

问题 This question is based on a discussion in this question. I have been working with sparse matrices earlier and my belief is that the way I've been working with these is efficient. My question is twofold: In the below, A = full(S) where S is a sparse matrix. What's the "correct" way to access an element in a sparse matrix? That is, what would the sparse equivalent to var = A(row, col) be? My view on this topic: You wouldn't do anything different. var = S(row, col) is as efficient as it gets. I

问题 I have a Markov chain given as a large sparse scipy matrix A . (I've constructed the matrix in scipy.sparse.dok_matrix format, but converting to other ones or constructing it as csc_matrix are fine.) I'd like to know any stationary distribution p of this matrix, which is an eigenvector to the eigenvalue 1 . All entries in this eigenvector should be positive and add up to 1, in order to represent a probability distribution. This means I want any solution for the system (A-I) p = 0 , p.sum()=1

问题 I have a Markov chain given as a large sparse scipy matrix A . (I've constructed the matrix in scipy.sparse.dok_matrix format, but converting to other ones or constructing it as csc_matrix are fine.) I'd like to know any stationary distribution p of this matrix, which is an eigenvector to the eigenvalue 1 . All entries in this eigenvector should be positive and add up to 1, in order to represent a probability distribution. This means I want any solution for the system (A-I) p = 0 , p.sum()=1

问题 I need to pass a scipy.sparse CSR matrix to a cython function. How do I specify the type, as one would for a numpy array? 回答1: Here is an example about how to quickly access the data from a coo_matrix using the properties row , col and data . The purpose of the example is just to show how to declare the data types and create the buffers (also adding the compiler directives that will usually give you a considerable boost)... #cython: boundscheck=False #cython: wraparound=False #cython:

问题 I need to pass a scipy.sparse CSR matrix to a cython function. How do I specify the type, as one would for a numpy array? 回答1: Here is an example about how to quickly access the data from a coo_matrix using the properties row , col and data . The purpose of the example is just to show how to declare the data types and create the buffers (also adding the compiler directives that will usually give you a considerable boost)... #cython: boundscheck=False #cython: wraparound=False #cython:

问题 I'm trying to use SparseTensor to represent weight variables in a fully-connected layer. However, it seems that TensorFlow 0.8 doesn't allow to use SparseTensor as tf.Variable. Is there any way to go around this? I've tried import tensorflow as tf a = tf.constant(1) b = tf.SparseTensor([[0,0]],[1],[1,1]) print a.__class__ # shows <class 'tensorflow.python.framework.ops.Tensor'> print b.__class__ # shows <class 'tensorflow.python.framework.ops.SparseTensor'> tf.Variable(a) # Variable is

问题 The SciPy Sparse Matrix tutorial is very good -- but it actually leaves the section on slicing un(der)developed (still in outline form -- see section: "Handling Sparse Matrices"). I will try and update the tutorial, once this question is answered. I have a large sparse matrix -- currently in dok_matrix format. import numpy as np from scipy import sparse M = sparse.dok_matrix((10**6, 10**6)) For various methods I want to be able to slice columns and for others I want to slice rows. Ideally I