tensorflow running error with cublas

Question

when I successfully install tensorflow on cluster, I immediately running mnist demo to check if it\'s going well, but here I came up with a problem. I don\'t know what is th

Answer 1

This was a nightmare to find a fix for - but the fix is somewhat simple

https://www.tensorflow.org/guide/using_gpu

# add to the top of your code under import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config....)

Answer 2

This problem re-surfaced for me using the latest stack (tensorflow 2.5, Cuda 11.1, Nvidia 3080). The fix above (as amended for Tensorflow 2) worked like a charm:

config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.compat.v1.Session(config=config)

Answer 3

Make sure to use sess.close() between each session to free the resources otherwise you'll have to kill the process in the task manager

Answer 4

I had exactly same error because in LD_LIBRARY_PATH I have cuda 5.5 in front of 7.5. After I moved 7.5 in front of 5.5 everything works fine now.

Answer 5

Aside from the mentioned solutions, this error also gets thrown when the CUBLAS version isn't compatible with the CUDA version. In my case, libclubas10 version 10.2.2.89-1 was incompatible with CUDA 10.1, so I had to downgrade:

sudo apt-get install libcublas10=10.2.1.243-1 libcublas-dev=10.2.1.243-1 cuda-libraries-10-1 cuda-libraries-dev-10-1

Answer 6

The compatibility issue between CUDA version and TensorFlow version. In my case, My CUDA version is 10.0 and TensorFlow version is 2.1.0, and this issue occurs. After changing TensorFlow 2.1.0 to TensorFlow 2.0.0, this issue disappears.