ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory

Question

I have installed Cuda 10.1 and cudnn on Ubuntu 18.04 and it seems to be installed properly as type nvcc and nvidia-smi, I get proper response:

user:~$ nvcc -V         


        

Answer 1

CUDA 10.1 (installed as per tensorflow docs) throws can't find libcublas.so.10.0 errors. The libs exist in /usr/local/cuda-10.1/targets/x86_64-linux/lib/ but are misnamed.

There was another (lost) stackoverflow post saying this was a pinned dependency issue with the package and could be fixed with an extra cli flag to apt. This didn't seem to fix the issue for me.

Tested workaround is to modify instructions to downgrade to CUDA 10.0

# Uninstall packages from tensorflow installation instructions 
sudo apt-get remove cuda-10-1 \
    libcudnn7 \
    libcudnn7-dev \
    libnvinfer6 \
    libnvinfer-dev \
    libnvinfer-plugin6

# WORKS: Downgrade to CUDA-10.0
sudo apt-get install -y --no-install-recommends \
    cuda-10-0 \
    libcudnn7=7.6.4.38-1+cuda10.0  \
    libcudnn7-dev=7.6.4.38-1+cuda10.0;
sudo apt-get install -y --no-install-recommends \
    libnvinfer6=6.0.1-1+cuda10.0 \
    libnvinfer-dev=6.0.1-1+cuda10.0 \
    libnvinfer-plugin6=6.0.1-1+cuda10.0;

Upgrading to CUDA-10.2 also seems to suffer from the same problem

# BROKEN: Upgrade to CUDA-10.2 
# use `apt show -a libcudnn7 libnvinfer7` to find 10.2 compatable version numbers
sudo apt-get install -y --no-install-recommends \
    cuda-10-2 \
    libcudnn7=7.6.5.32-1+cuda10.2  \
    libcudnn7-dev=7.6.5.32-1+cuda10.2;
sudo apt-get install -y --no-install-recommends \
    libnvinfer7=7.0.0-1+cuda10.2 \
    libnvinfer-dev=7.0.0-1+cuda10.2 \
    libnvinfer-plugin7=7.0.0-1+cuda10.2;

Test GPU Visibility in Python

python3
>>> import tensorflow as tf
>>> tf.test.is_gpu_available()

FutureWarnings on tensorflow import

https://github.com/tensorflow/tensorflow/issues/30427

two solutions:

• pip3 install tf-nightly-gpu
• pip3 install "numpy<1.17"

Update:

You also need the correct tensorflow version to match with your CUDA version

Tensorflow / CUDA version combinations:

• Tensorflow v2.x does not support CUDA 9 (Ubuntu 18.4 default)
• Tensorflow v2.1.0 works with CUDA 10.1
• Tensorflow v2.0.0 works with CUDA 10.0

See for the full list: https://www.tensorflow.org/install/source#tested_build_configurations

You may potentually need to reinstall tensorflow with a named version matching your CUDA

pip uninstall tensorflow tensorflow-gpu
pip install tensorflow==2.1.0 tensorflow-gpu==2.1.0

Then add CUDA to $PATH and $LD_LIBRARY_PATH in ~/.bashrc

~/.bashrc

# CUDA Environment Setup: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#environment-setup
for CUDA_BIN_DIR in `find /usr/local/cuda-*/bin   -maxdepth 0`; do export PATH="$PATH:$CUDA_BIN_DIR"; done;
for CUDA_LIB_DIR in `find /usr/local/cuda-*/lib64 -maxdepth 0`; do export LD_LIBRARY_PATH="${LD_LIBRARY_PATH:+${LD_LIBRARY_PATH}:}$CUDA_LIB_DIR"; done;

export            PATH=`echo $PATH            | tr ':' '\n' | awk '!x[$0]++' | tr '\n' ':' | sed 's/:$//g'` # Deduplicate $PATH
export LD_LIBRARY_PATH=`echo $LD_LIBRARY_PATH | tr ':' '\n' | awk '!x[$0]++' | tr '\n' ':' | sed 's/:$//g'` # Deduplicate $LD_LIBRARY_PATH