Tensorflow dense gradient explanation?

Question

I recently implemented a model and when I ran it I received this warning:

UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. 
T         


        

Answer 1

A dense Tensor can be thought of like a standard python array. A sparse one can be thought of as a collection of indices and values e.g.

# dense
array = ['a', None, None, 'c']

# sparse
array = [(0, 'a'), (3, 'c')]

So as you can see if you have a lot of empty entries a sparse array will be much more efficient than a dense one. But if all entries are filled in, dense is far more efficient. In your case somewhere in the tensor flow graph a sparse array is being converted to a dense one of indeterminate size. The warning is just saying it is possible that you can waste a lot of memory like this. But it might not be a problem at all if the sparse array is not too big/already quite dense.

If you want to diagnose it I would advise naming your various tensor objects then it will print exactly which ones are being used in this conversion and you can work out what you might be able to adjust to remove it.