Set LD_LIBRARY_PATH before importing in python

Question

Python uses the PYTHONPATH environment-variable to determine in which folders it should look for modules. You can play around with it by modifying sys.pat

Answer 1

Since coreutils 8.30 it is possible to use env -S to split shebang line to separate arguments:

#!/usr/bin/env -S LD_LIBRARY_PATH=/path/to/lib python options

For compatibility with older systems, you can use the fact that the shell allows all commands to be enclosed in quotes, whereas in python it will be just strings:

#!/bin/sh
"export" "LD_LIBRARY_PATH=/path/to/lib:$LD_LIBRARY_PATH"
"exec" "python3" "$0" "$@"
# Further python program
import somemodule

Answer 2

My solution to this problem is to put this as the first line of a Python script (instead of the usual shebang):

exec env LD_LIBRARY_PATH=/some/path/to/lib /path/to/specific/python -x "$0" "$@"

And here is how this works:

• with no shebang the current shell treats the file as a shell script,
• "exec" ensures that this first line is also the last command from this file executed by the shell,
• "env" is used here to set any environment variables, e.g. LD_LIBRARY_PATH,
• an exact path to Python's interpreter can specified or "env" can find one in PATH,
• "-x" is a Python's option which causes the first line to be ignored by the Python interpreter,
• "$0" is the script name, "$@" is substituted by positional parameters.

Answer 3

UPDATE: see the EDIT below.

I would use:

import os

os.environ['LD_LIBRARY_PATH'] = os.getcwd()  # or whatever path you want

This sets the LD_LIBRARY_PATH environment variable for the duration/lifetime of the execution of the current process only.

EDIT: it looks like this needs to be set before starting Python: Changing LD_LIBRARY_PATH at runtime for ctypes

So I'd suggest going with a wrapper .sh (or .py if you insist) script. Also, as @chepner pointed out, you might want to consider installing your .so files in a standard location (within the virtualenv).

See also Setting LD_LIBRARY_PATH from inside Python

Answer 4

The solution works fine if the env is reinit

import os

os.environ['LD_LIBRARY_PATH'] = os.getcwd()  # or whatever path you want 

The code need to be taken in place....

os.execv(sys.argv[0], sys.argv)

Answer 5

Python, when gets the values of environment variables as in os.environ[‘LD_LIBRARY_PATH’] or os.environ[‘PATH’], it copies the values, into a dictionary, from it's parent process's environment, generally bash (bash process's environment get's carried to the child process, the python running instance).

you can see this environment variable section with env command output from bash.

you can also see/read this env data from /proc/<pid>/environ, by introducing an infinite loop(while 1: pass) after modifying any environment variable.

If you see/read this variable value/data from /proc/<pid>/environ after modifying it inside the python script, you would get to see that the real variable's data doesn't get modified, though the python script shows a modified dictionary key value, updated.

What actually happens when you modify an env variable inside python script, as in os.environ['LD_LIBRARY_PATH']='/<new_location>', is that it just updates the value in local dictionary, which is not mapped to process's env variable section. Hence it won't propagate all the way back to reflect in current process's environment, because ONLY a local dictionary was modified/updated/populated.

Hence if we want to have the new environment variable to be reflected, we should overwrite the memory image of the process with new environment variable data, using execv.

Example:

new_lib = '/<new_location>'
if not new_lib in os.environ['LD_LIBRARY_PATH']:
    os.environ['LD_LIBRARY_PATH'] += ':'+new_lib
    try:
        os.execv(sys.argv[0], sys.argv)
    except Exception as e:
        sys.exit('EXCEPTION: Failed to Execute under modified environment, '+e)

import xyz
#do something else

Limitation: Ideally, python should not allow such modification of os.environ variables. But because there is no constant dictionary data type, it allows modification of the data variable. There is absolutely no use of modifying the values, as it does nothing useful to reflect in running process's real environment, unless execv is used.