Shared library dependencies with distutils

Question

I'm a newbie to distutils and I have a problem that really has me stuck. I am compiling a package that requires an extension, so I make the extension thus:

    a_module = Extension(
          "amodule",
          ["initmodule.cpp"],
          library_dirs=libdirs,
          extra_objects = [
                    "unix/x86_64/lib/liba.so"
                    "unix/x86_64/lib/lib.so",
                    "unix/x86_64/lib/libc.so"],
    )

I then run the setup method:

    setup(name="apackage", version="7.2",
      package_dir = {'':instdir+'/a/b/python'},
      packages=['apackage','package.tests'],
      ext_modules=[hoc_module]
)

The package distribution is made properly and I can "python setup.py install" fine but when I try and import my package I get an error ImportError: liba.so.0: cannot open shared object file: No such file or directory

I realise that when I add the location of liba.so.0 to my LD_LIBRARY_PATH the program runs fine. Unfortunately I haven't written these modules and don't have a good understanding of compilation. I've been trying to figure this out for several days to no avail.

UPDATE:I tried passing the liba.a, libb.a etc files to extra_objects but this didn't work, generating the following errror: liba.a: could not read symbols: Bad value collect2: ld returned 1 exit status. What I'm trying to do is package a python module which requires a library to be compiled which itself depends on other libraries which I need to somehow include in the package .I suspect that my problem is very similar to this one: http://mail.python.org/pipermail/distutils-sig/2009-February/010960.html but that one was not resolved, I thought perhaps since it's two years old a resolution has been found?

UPDATE 2: For now I have solved this by doing:

      data_files=[('/usr/local/lib', glob.glob('unix/x86_64/lib/*'))]

That is to say, I am copying the libraries I need into /usr/local/lib. I'm not hugely happy with this solution however, not least because it requires my users to have root privileges and also because this may still not work Redhat distros. So if anyone can suggest something better than this fix do please let me know.

Answer 1

You can pass flags to the compiler or linker to let it know where to find libraries at runtime, alleviating the need to have LD_LIBRARY_PATH set properly. I'll illustrate with a few examples:

# Will link just fine, then fail to find libpcap.so unless it's in LD_LIBRARY_PATH
gcc -o blah blah.o -lpcap -L/opt/csw/lib

# If libpcap is in LD_LIBRARY_PATH, it'll link fine.  Other people who may not have
# LD_LIBRARY_PATH set properly can still run it without fixing their environment
gcc -o blah blah.o -lpcap -R/opt/csw/lib

# This will allow me to link and execute the binary without having LD_LIBRARY_PATH
# setup properly
gcc -o blah blah.o -lpcap -{L,R}/opt/csw/lib

# This makes it possible to use relative paths.  The literal string `$ORIGIN/../lib/`
# gets stored in the binary (`readelf -d binary_name` if you want to see the effect
# it has), which causes `$ORIGIN` to resolve to the directory containing the binary
# when it was executed.  In a makefile, you'll see that written as `$$ORIGIN/../lib/`
# to prevent `make` from expanding it.
gcc -o blah blah.o -lsomelib -L/whatever/path/floats/your/boat -R'$ORIGIN/../lib/'

In the way of explanation, in case it wasn't obvious (since I hate answers without an explanation):

• paths given with -L are only used to find libraries when you're linking
• paths given with -R are only used to find libraries when the binary is executed

Answer 2

The extra_objects argument to the Extension class is not so much a list of libraries to link into your extension, but a list of object files that will be passed to the linker (and the filenames shouldn't include extensions, since distutils will add those.) It doesn't do what you seem to want.

If you want to link against specific shared libraries, as the names of those files suggest you want, you have to do two things: tell distutils to tell the compiler to link against those shared libraries, and tell the dynamic linker (usually ld.so) where to find those shared libraries. You can tell distutils to tell the compiler to link against the libraries by using the libraries argument to Extension, which should be a list of library names (without the lib prefix and .so suffix.) In your example that seems to be ['a', 'b', 'c'] (although it looks like the 'b' fell off of 'lib.so', and 'c' would actually clash with the system libc.)

Telling the linker where to find these shared libraries can be done by setting the LD_LIBRARY_PATH environment variable, as you did, or by changing a system-wide configuration setting (with ldconfig or by editing /etc/ld.so.conf), or by hardcoding the search path in the extension module; you can do the latter by passing the runtime_library_dirs argument to Extension. Hardcoding the path does have its own issues, though -- you have to keep those libraries in the same place, and accessible to all users of the extension module.

(Alternatively, you can use static instead of dynamic linking, for example by only providing the libraries in static form, liba.a archives (in which case distutils will automatically link to them statically.) That basically means the whole library is included in the extension module, which has various downsides and upsides.)