How to perform efficient sparse matrix multiplication by using tf.matmul?

Question

I'm trying to perform a sparse matrix multiplication by using tf.matmul().

However, the inference speed is much more slower than dense matrix multiplication.

According to the description in tf.sparse_matmul() :

• The breakeven for using this versus a dense matrix multiply on one platform was 30% zero values in the sparse matrix.

Thus , I make the sparse matrix with 7/8 zero values.

Here is my code:

import tensorflow as tf
import numpy as np
import time
a = tf.Variable(np.arange(1000).reshape(250,4) ,dtype=tf.float32) #dense matrix
b = tf.Variable(np.array([0,0,0,0,0,0,0,1],dtype=np.float32).reshape(4,2),dtype=tf.float32) # sparse matrix
c = tf.matmul(a,b,b_is_sparse=True) # do the sparse matrix multiplication

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_iteration = 5000
    num_burnin = 50
    duration = 0

    for i in range(num_iteration+num_burnin):
        startTime  = time.time()
        result = sess.run(c)
        endTime = time.time()
        if i > num_burnin :
            duration+= endTime-startTime

   print(" Average Inference Time = %.3f  ms"%(duration*1000/num_iteration))

I set "b_is_sparse=True" to do a sparse matrix multiplication , and it takes about 0.380 ms on my GeForce GTX 960M.

However , if I set "b_is_sparse=False" to do a dense matrix multiplication , it takes about 0.280 ms.

I have tried to use tf.sparse_tensor_dense_matmul and tf.embedding_lookup_sparse to perform sparse matrix multiplication , but the inference speed is still slower than dense matrix multiplication.

Is there something wrong in my code or other way to perform sparse matrix multiplication ?

Any advice will be greatly appreciated!!

Answer 1

The relative performance depends on many factor. Sparse multiplication can be faster than dense multiplication with dense matrix (hopefully), but you are right that it can also be slower.

For one thing, it depends on the size of your matrix.

Here is the result of the multiplication of two square matrices, one random and one filled with zeros, and recorded the computation time for dense and spare multiplication.

As you can see, even with a completely zero matrix, sparse multiplication can be slower than dense multiplication for smaller matrix size -- in fact almost three times slower for matrices about 120x120. In this experiment on my computer, sparse matrix multiplication starts taking over for sizes of about 700x700 and ends up being about 2 times faster. Of course YMMV depending on your configuration.