TensorFlow memory use while running on GPU: why does it look like not all memory is used?

Question

This is a follow-up to my question posted here: Memory error with larger images when running convolutional neural network using TensorFlow on AWS instance g2.2xlarge

Answer 1

Looking the error log it looks like or you are running out of GPU memory or the tensor is not initiated at that point. You can try to insert Tensor::IsInitialized before the line that start the problem (99) to make sure it's the GPU, if it is, you may have some code still running in the GPU, from previous tries, make sure that is not happening. There is two discussions that I think may be relevant for your problem, here: https://github.com/tensorflow/tensorflow/issues/7025 and here: https://github.com/aymericdamien/TensorFlow-Examples/issues/38 Good luck