ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory

Question

I have installed Cuda 10.1 and cudnn on Ubuntu 18.04 and it seems to be installed properly as type nvcc and nvidia-smi, I get proper response:

user:~$ nvcc -V         


        

Answer 1

Amin,

I'm getting the same error when I try to run imagenet tutorial from tensorflow models package -- https://github.com/tensorflow/models/tree/master/tutorials/image/imagenet

 python3 classify_image.py
 ...
 2019-07-21 22:29:58.367858: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
 2019-07-21 22:29:58.367982: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
 2019-07-21 22:29:58.368112: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
 2019-07-21 22:29:58.368234: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
 2019-07-21 22:29:58.368369: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
 2019-07-21 22:29:58.368498: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
 2019-07-21 22:29:58.374333: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7

I think there's a version incompatibility somewhere and likely tensorflow, still relies on the old version of binaries provided by cuda libraries. Going to the place where binaries are stored and creating a link that's named 10.0 but either targets 10.1 or the default version of the library, seems to solve the problem for me.

 # cd /usr/lib/x86_64-linux-gnu
 # ln -s libcudart.so.10.1 libcudart.so.10.0
 # ln -s libcublas.so libcublas.so.10.0
 # ln -s libcufft.so libcufft.so.10.0
 # ln -s libcurand.so libcurand.so.10.0
 # ln -s libcusolver.so libcusolver.so.10.0
 # ln -s libcusparse.so libcusparse.so.10.0

Now I'm able to run tutorial successfully

 2019-07-24 21:43:21.172908: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
 2019-07-24 21:43:21.174653: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
 2019-07-24 21:43:21.175826: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
 2019-07-24 21:43:21.182305: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
 2019-07-24 21:43:21.183970: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
 2019-07-24 21:43:21.206796: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
 2019-07-24 21:43:21.210685: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
 2019-07-24 21:43:21.212694: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
 2019-07-24 21:43:21.213060: I tensorflow/core/platform/cpu_feature_guard.cc:142]      
 Your CPU supports instructions that this TensorFlow binary was not compiled to use: FMA
 2019-07-24 21:43:21.238541: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3214745000 Hz
 2019-07-24 21:43:21.240096: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557e2b682ce0 executing computations on platform Host. Devices:
 2019-07-24 21:43:21.240162: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): , 
 2019-07-24 21:43:21.355158: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557e2b652000 executing computations on platform CUDA. Devices:
 2019-07-24 21:43:21.355234: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce GTX 1060 6GB, Compute Capability 6.1
 2019-07-24 21:43:21.357074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
 name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7715
 pciBusID: 0000:01:00.0
 2019-07-24 21:43:21.357151: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
 2019-07-24 21:43:21.357207: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
 2019-07-24 21:43:21.357245: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
 2019-07-24 21:43:21.357283: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
 2019-07-24 21:43:21.357321: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
 2019-07-24 21:43:21.357358: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
 2019-07-24 21:43:21.357395: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
 2019-07-24 21:43:21.360449: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
 2019-07-24 21:43:21.380616: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
 2019-07-24 21:43:21.385223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
 2019-07-24 21:43:21.385272: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
 2019-07-24 21:43:21.385299: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
 2019-07-24 21:43:21.388647: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5250 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
 2019-07-24 21:43:32.001598: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
 2019-07-24 21:43:32.532105: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
 W0724 21:43:34.981204 140284114071872 deprecation_wrapper.py:119] From classify_image.py:85: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.