After building TensorFlow from source, seeing libcudart.so and libcudnn errors

Question

I\'m building TensorFlow from source code. The build appears to succeed; however, when my TensorFlow program invokes import tensorflow, one or both of the foll

Answer 1

On MacOS, this issue is often caused by bazel running in a sandbox environment, thus not respecting the LD_LIBRARY_PATH set in your local shell. I wouldn't bother going into the merit of deep integration of sandboxing in a build tool.

The simple workaround is to symlink the libraries into /usr/local/lib.

cd /usr/local/lib && ln -s ../cuda/lib/libcudart.8.0.dylib

Answer 2

First, for the following error:

ImportError: libcudart.so.8.0: cannot open shared object file: No such file or directory

make sure your LD_LIBRARY_PATH includes your lib64 directory in whichever path you installed your cuda package in. You can do this by adding an export line in your .bashrc. For Omar, it looked like the following:

I fixed this just adding the cuda path to my .bashrc

export LD_LIBRARY_PATH=/usr/local/cuda/lib64/

---

For me, I had to do Omar's line and also: export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64/ because I have two directories involving cuda (probably not the best).

---

Second, are you sure you installed cuDNN? Note that this is different from the regular cuda package. You will need to register, then download and install the package from the following page: https://developer.nvidia.com/cudnn

---

Third, I had this same problem:

ImportError: libcudnn.5: cannot open shared object file: No such file or directory

It turns out there is no libcudnn.5 in my /usr/local/cuda/lib64 or /usr/local/cuda-8.0/lib64 directories. However, I do have a libcudnn.so.6.* file. To solve the problem, I created a soft link:

ln -s libcudnn.so.6.* libcudnn.so.5

in my /usr/local/cuda/lib64 directory. Now everything works for me. Your directory might be different if you already had cuDNN, and your libcudnn.so.6.* might be a different version, so check that.

Answer 3

I came across the same issue

In [1]: import tensorflow
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/usr/local/lib/python3.5/site-packages/tensorflow/python/pywrap_tensorflow.py in <module>()
     40     sys.setdlopenflags(_default_dlopen_flags | ctypes.RTLD_GLOBAL)
---> 41   from tensorflow.python.pywrap_tensorflow_internal import *
     42   from tensorflow.python.pywrap_tensorflow_internal import __version__

/usr/local/lib/python3.5/site-packages/tensorflow/python/pywrap_tensorflow_internal.py in <module>()
     27             return _mod
---> 28     _pywrap_tensorflow_internal = swig_import_helper()
     29     del swig_import_helper

/usr/local/lib/python3.5/site-packages/tensorflow/python/pywrap_tensorflow_internal.py in swig_import_helper()
     23             try:
---> 24                 _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
     25             finally:

/usr/local/lib/python3.5/imp.py in load_module(name, file, filename, details)
    241         else:
--> 242             return load_dynamic(name, filename, file)
    243     elif type_ == PKG_DIRECTORY:

/usr/local/lib/python3.5/imp.py in load_dynamic(name, path, file)
    341             name=name, loader=loader, origin=path)
--> 342         return _load(spec)
    343

ImportError: libcudnn.so.5: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

ImportError                               Traceback (most recent call last)
<ipython-input-1-a649b509054f> in <module>()
----> 1 import tensorflow

/usr/local/lib/python3.5/site-packages/tensorflow/__init__.py in <module>()
     22
     23 # pylint: disable=wildcard-import
---> 24 from tensorflow.python import *
     25 # pylint: enable=wildcard-import
     26

/usr/local/lib/python3.5/site-packages/tensorflow/python/__init__.py in <module>()
     49 import numpy as np
     50
---> 51 from tensorflow.python import pywrap_tensorflow
     52
     53 # Protocol buffers

/usr/local/lib/python3.5/site-packages/tensorflow/python/pywrap_tensorflow.py in <module>()
     50 for some common reasons and solutions.  Include the entire stack trace
     51 above this error message when asking for help.""" % traceback.format_exc()
---> 52   raise ImportError(msg)
     53
     54 # pylint: enable=wildcard-import,g-import-not-at-top,unused-import,line-too-long

ImportError: Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/usr/local/lib/python3.5/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
  File "/usr/local/lib/python3.5/imp.py", line 242, in load_module
    return load_dynamic(name, filename, file)
  File "/usr/local/lib/python3.5/imp.py", line 342, in load_dynamic
    return _load(spec)
ImportError: libcudnn.so.5: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

I have installed cudnn 6.0 while it needs libcudnn.so.5, apparently it couldn't find libcudnn.so.5. It seems that your tensorflow needs cudnn 5.x, so install cudnn 5.x

Make sure you have already installed cuda 8.0 and exported the PATH and LD_LIBRARY_PATH

To install cudnn 5.x, try the following commands

Extract tgz files

$ tar -zxvf cudnn-8.0-linux-x64-v5.1.tgz

Check the files

$ cd cuda/lib64/
$ ls -l
total 150908
lrwxrwxrwx 1 doom doom       13 Nov  7  2016 libcudnn.so -> libcudnn.so.5
lrwxrwxrwx 1 doom doom       18 Nov  7  2016 libcudnn.so.5 -> libcudnn.so.5.1.10
-rwxr-xr-x 1 doom doom 84163560 Nov  7  2016 libcudnn.so.5.1.10
-rw-r--r-- 1 doom doom 70364814 Nov  7  2016 libcudnn_static.a

Here you will see 2 symbolic link files, and just copy libcudnn.so.5.1.10 and libcudnn_static.a to /usr/local/cuda/lib64

Make symbolic link files

$ cd /usr/local/cuda/lib64/
$ sudo ln -s libcudnn.so.5.1.10 libcudnn.so.5
$ sudo ln -s libcudnn.so.5 libcudnn.so
$ ls -l libcudnn*
lrwxrwxrwx 1 root root       13 May 24 09:24 libcudnn.so -> libcudnn.so.5
lrwxrwxrwx 1 root root       18 May 24 09:24 libcudnn.so.5 -> libcudnn.so.5.1.10
-rwxr-xr-x 1 root root 84163560 May 24 09:23 libcudnn.so.5.1.10
-rw-r--r-- 1 root root 70364814 May 24 09:23 libcudnn_static.a

Copy cudnn.h in include directory to /usr/local/cuda/include

$ sudo cp cudnn.h /usr/local/cuda/include/

Hope it will help you!

Answer 4

I have seen a similar error (bottom of this post), but complaining about libcudnn.so.6 instead of libcudart.so.8.0 (see a note below).

Solution:

1. Download 'cuDNN v6.0 Library for Linux':
   • Go to https://developer.nvidia.com/rdp/cudnn-download
   • Click on 'Download cuDNN v6.0 (April 27, 2017), for CUDA 8.0'
   • Then click on 'cuDNN v6.0 Library for Linux'. A file (named 'cudnn-8.0-linux-x64-v6.0.tgz') will be downloaded.
2. Follow the instructions of Alexander Yau above to install the cuDNN v6.0 library.

Note:

the Tensorflow installation instructions (as of 20/Aug/2017) require installing cuDNN v5.1, but my Tensorflow installation (following the instructions for installing in a virtualenv) required cuDNN v6.x (as indicated by the error). I don't know if it is a mistake on my side or a Tensorflow documentation one. Nevertheless, above solution worked for me.

Encountered error:

In [1]: import tensorflow as tf
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-41389fad42b5> in <module>()
----> 1 import tensorflow as tf

/home/haseeb/.virtualenvs/attention_transformer/local/lib/python2.7/site-packages/tensorflow/__init__.py in <module>()
     22 
     23 # pylint: disable=wildcard-import
---> 24 from tensorflow.python import *
     25 # pylint: enable=wildcard-import
     26 

/home/haseeb/.virtualenvs/attention_transformer/local/lib/python2.7/site-packages/tensorflow/python/__init__.py in <module>()
     47 import numpy as np
     48 
---> 49 from tensorflow.python import pywrap_tensorflow
     50 
     51 # Protocol buffers

/home/haseeb/.virtualenvs/attention_transformer/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py in <module>()
     50 for some common reasons and solutions.  Include the entire stack trace
     51 above this error message when asking for help.""" % traceback.format_exc()
---> 52   raise ImportError(msg)
     53 
     54 # pylint: enable=wildcard-import,g-import-not-at-top,unused-import,line-too-long

ImportError: Traceback (most recent call last):
  File "/home/haseeb/.virtualenvs/attention_transformer/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "/home/haseeb/.virtualenvs/attention_transformer/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "/home/haseeb/.virtualenvs/attention_transformer/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
ImportError: libcudnn.so.6: cannot open shared object file: No such file or directory


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

Answer 5

TensorFlow 1.2.1 is compatible with cuDNN 5.1, but not yet with 6.0. So just install cuDNN 5.1. Besides that you seem to be missing CUDA 8.0.

Answer 6

Common workaround general problem related to GPU, CUDA, and Docker:

A. If you are dealing with machine learning/ deep learning related deployment, use nvidia-docker and not native-docker. To install nvidia-docker follow these simple steps.

Docker containers are platform-agnostic, but also hardware-agnostic. This presents a problem when using specialized hardware such as NVIDIA GPUs which require kernel modules and user-level libraries to operate. As a result, Docker does not natively support NVIDIA GPUs within containers.

B. If you want to access GPU with docker, never build a container from scratch, you will be fried with a number of errors. Instead just use any container from Nvidia-Docker hub. Choose any specific image, copy its Dockerfile and run sudo nvidia-docker build -t happyapp .. [happyapp is your new app name]. In 5 min you will get your container ready (depends on network speed :p).

C. Never download a nvidia-docker having any cuda/cudnn version If you want to install and run Tensorflow on it. If you do so you will get libcudnn.so.6 or libcudnn.so.9 or libcusolver.so.8.0 related errors and you will hardly get around this errors. Instead just use pre-built Tensorflow Docker image: sudo nvidia-docker run -it tensorflow/tensorflow:latest-gpu /bin/bash