Cannot find dynamic library when running a Python script from Bazel

问题

I am trying to setup CUDA enabled Python & TensorFlow environment on OSx 10.11.6

Everything went quite smoothly. First I installed following:

CUDA - 7.5
cuDNN - 5.1

I ensured that the LD_LIBRARY_PATH and CUDA_HOME are set properly by adding following into my ~/.bash_profile file:

export CUDA_HOME=/usr/local/cuda 
export DYLD_LIBRARY_PATH="$CUDA_HOME/lib:$DYLD_LIBRARY_PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib:$LD_LIBRARY_PATH"
export PATH="$CUDA_HOME/bin:$PATH"

Then I used Brew to install following:

python - 2.7.12_2
bazel - 0.3.2
protobuf - 3.1.0

Then I used Pip to install CPU only TensorFlow from: https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-0.11.0rc0-py2-none-any.whl

I checked out the Magenta project from: https://github.com/tensorflow/magenta and run all the test using:

bazel test //magenta/...

And all of them have passed.

So far so good. So I decided to give the GPU enabled version of TensorFlow a shot and installed it from: https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow-0.11.0rc0-py2-none-any.whl

Now all the tests fail with the following error:

import tensorflow as tf
  File "/usr/local/lib/python2.7/site-packages/tensorflow/__init__.py", line 23, in <module>
    from tensorflow.python import *
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 28, in <module>
    _pywrap_tensorflow = swig_import_helper()
  File "/usr/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
    _mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
ImportError: dlopen(/usr/local/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow.so, 10): Library not loaded: @rpath/libcudart.7.5.dylib
  Referenced from: /usr/local/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow.so
  Reason: image not found

So obviously the script run from Bazel has trouble locating the libcudart.7.5.dylib library.

I did try running GPU computations from Python without Bazel and everything seems to be fine.

I also did create a test script and run it using Bazel and it seems that the directory containing libcudart.7.5.dylib library is reachable, however the LD_LIBRARY_PATH is not set.

I searched the documentation and found --action_env and --test_env flags, but none of them actually seems to set the LD_LIBRARY_PATH for the execution.

These are the options from loaded from .bazelrc files.

Inherited 'common' options: --isatty=1 --terminal_columns=80
Inherited 'build' options: --define=allow_oversize_protos=true --copt -funsigned-char -c opt --spawn_strategy=standalone
'run' options: --spawn_strategy=standalone

What is the correct way to let Bazel know about the runtime dependencies?

UPDATE

The trouble seems to be caused by the fact that "env" command is part of the execution chain and it does seem to clear both LD_LIBRARY_PATH and DYLD_LIBRARY_PATH environmental variables. Is there a workaround different than disabling the SIP?

回答1:

It looks like SIP affects the behavior of how the DYLD_LIBRARY_PATH gets propagated to the child processes. I found a similar problem and another similar problem.

I didn't want to turn the SIP off, so I just created symlinks for the CUDA library into a standard location.

ln -s /usr/local/cuda/lib/* /usr/local/lib

Not sure if this is the best solution, but it does work and it does not require the SIP to be disabled.

回答2:

Use

export LD_LIBRARY_PATH=/usr/local/cuda/lib64/

before launching bazel. Double check in the directory above if there is such a file.

ls /usr/local/cuda/lib64/libcudart.7.5.dylib

Note that in Macosx the name is different:

export DYLD_LIBRARY_PATH=/usr/local/cuda/lib/

See this answer for more information on SuperUser

回答3:

The problem is indeed SIP, and the solution is to pass --action_env DYLD_LIBRARY_PATH=$CUDA_HOME/lib to the bazel command, e.g.:

bazel build -c opt --config=cuda --action_env DYLD_LIBRARY_PATH=$CUDA_HOME/lib //tensorflow/tools/pip_package:build_pip_package

来源：https://stackoverflow.com/questions/40105625/cannot-find-dynamic-library-when-running-a-python-script-from-bazel

标签

python

macos

tensorflow

bazel