How to deal with Linux/Python dependencies?

Question

Due to lack of support for some libraries I want to use, I moved some Python development from Windows to Linux development. I\'ve spent most of the day messing about getting

Answer 1

I see two separate problems here:

1. Keeping track of all the python modules you need for your project.

2. Keeping track of all the dynamic libraries you need for the python modules in your project.

For the first problem, I have found that buildout is good help, althought it takes a litle while to grasp.

In your case, I would start by creating a directory for my new project. I would then go into that directory and download bootstrap.py

wget http://python-distribute.org/bootstrap.py 

I would then create a buildout.cfg file:

[buildout]
parts = qrproject
        python
eggs = pyqrencode

[qrproject]
recipe = z3c.recipe.scripts
eggs = ${buildout:eggs}
entry-points= qrproject=qrprojectmodule:run
extra-paths = ${buildout:directory}

# This is a simple way of creating an interpreter that will have
# access to all the eggs / modules that this project uses.
[python]
recipe = z3c.recipe.scripts
interpreter = python
eggs = ${buildout:eggs}
extra-paths = ${buildout:directory}

In this buildout.cfg I'm referencing the module qrprojectmodule (in entry-points under [qrproject]. This will create a bin/qrproject that runs the function run in the module qrprojectmodule. So I will also create the file qrprojectmodule.py

import qrencode

def run():
    print "Entry point for qrproject. Happily imports qrencode module"

Now it's time to run bootstrap.py with the python binary you want to use:

python bootstrap.py

Then run the generated bin/buildout

bin/buildout

This will create two additional binaries in the bin/ directory - bin/qrproject and bin/python. The former is your project's main binary. It will be created automatically each time you run buildout and will have all the modules and eggs you want loaded. The second is simply a convenient way to get a python prompt where all your modules and eggs are loaded, for easy debugging. The fine thing here is that bin/buildout will automatically install any python eggs that the eggs (in your case pyqrencode) have specified as dependencies.

Actually, you will probably get a compilation error in the step where you run bin/buildout. This is because you need to address problem 2: All dynamic libraries being available on your system. On Linux, it's usually best to get help from your packaging system. I'm going to assume you're using a Debian derivate such as Ubuntu here.

The pyqrencode web site specifies that you need the libqrencode library for pyqrencode to work. So I used my package manager to search for that:

$ apt-cache search libqrencode
libqrencode-dev - QR Code encoding library -- development
libqrencode3 - QR Code encoding library
qrencode - QR Code encoder into PNG image

In this case, I want the -dev package, as that installs linkable libraries and header files required to compile python C-modules. Also, the dependency system in the package manager will make sure that if I install libqrencode-dev, I will also get libqrencode3, as that is required at runtime, i.e. after compilation of the module.

So, I install the package:

sudo apt-get install libqrencode-dev

Once that has completed, rerun bin/buildout and the pyqrencode module will (hopefully) compile and install successfully. Now try to run bin/qrproject

$ bin/qrproject 
Entry point for qrproject. Happily imports qrencode module

Success! :-)

So, in summary:

1. Use buildout to automatically download and install all the python modules/eggs you need for your project.

2. Use your system's package manager to install any dynamic (C) libraries required by the python modules you use.

Be aware that in many cases there are already packaged versions of your python modules available in the package system. For example, pil is available by installing the python-imaging package on Ubuntu. In this case, you don't need to install it via buildout, and you don't need to worry about libraries being available - the package manager will install all dependencies required for the module to run. Doing it via buildout can however make it easier to distribute your project and make it run on other systems.

Answer 2

Your story reminds me of my early experiences with Linux, and why I love APT.

There is no universal solution to your general problem; the best you can do is to take advantage of the work or others. The Debian packagers do a great job of flagging the dependencies of packages, so apt-get will pull in what you need. So, my strategy is simply to avoid building and installing stuff on my own, and use apt-get wherever possible.

Note that Ubuntu is based on Debian and thus gains the benefit of the work of the Debian packagers. I haven't used Fedora but I hear that the packages are not as well-organized as the ones from Debian.