How to get virtualenv to use dist-packages on Ubuntu?

Question

I know that virtualenv, if not passed the --no-site-packages argument when creating a new virtual environment, will link the packages in /usr/local/lib/python2.7/site-packages (for Python 2.7) with a newly-created virtual environment. On Ubuntu 12.04 LTS, I have three locations where Python 2.7 packages can be installed (using the default, Ubuntu-supplied Python 2.7 installation):

1. /usr/lib/python2.7/dist-packages: this has my global installation of ipython, scipy, numpy, matplotlib – packages that I would find difficult and time-consuming to install individually (and all their dependences) if they were not available via the scipy stack.
2. /usr/local/lib/python2.7/site-packages: this is empty, and I think it will stay that way on Ubuntu unless I install a package from source.
3. /usr/local/lib/python2.7/dist-packages: this has very important local packages for astronomy, notably those related to PyRAF, STScI, etc., and they are extremely difficult and time-consuming to install individually.

Note that a global directory such as /usr/lib/python2.7/site-packages does not exist on my system. Note also that my global installation of ipython, scipy, etc. lets me use those packages on-the-fly without having to source/activate a virtual environment every time.

Naturally, I now want to use virtualenv to create one virtual environment in my user home directory which I will source/activate for my future projects. However, I would like this virtual environment, while being created, to link/copy all of my packages in locations (1) and (3) in the list above. The main reason for this is that I don't want to go through the pip install process (if it is even possible) to re-install ipython, scipy, the astro-packages, etc. for this (and maybe other) virtual environments.

Here are my questions:

1. Is there a way for me to specify to virtualenv that I would like it to link/copy packages in these two dist-packages directories for virtual environments that are created in the future?
2. When I eventually update my global installation of scipy, ipython, etc. in the two dist-packages directories, will this also update/change the packages that my virtual environment uses (and which it originally got during virtualenv creation)?
3. If I ever install a package from source on Ubuntu, will it go in /usr/local/lib/python2.7/dist-packages, or /usr/local/lib/python2.7/site-packages?

Thanks in advance for your help!

Answer 1

This might be a legitimate use of PYTHONPATH - an environmental variable that virtualenv doesn't touch, which uses the same syntax as the environmental variable PATH, in bash PYTHONPATH=/usr/lib/python2.7/dist-packages:/usr/local/lib/python2.7/dist-packages in a .bashrc or similar. If you followed this path,

1. You don't have to tell your virtual environment about this at all, it won't try to change it.

2. No relinking will be required, and

3. That will still go wherever it would have gone (pip install always uses /usr/local/lib/python2.7/dist-packages/ for my Ubuntu) if you install them outside of your virtual environment. If you install them from within your virtual environment (while it's activated) then of course it'll be put in the virtualenvironment.

Answer 2

I'm just getting my head around virtualenv, but there seems to be an easier way than mentioned so far.

1. Since virtualenv 1.7 --no-site-packages has been the default behavior. Therefore using the --system-site-packages flag to virtualenv is all that is needed to get dist-packages in your path - if you use the tweaked virtualenv shipped by Ubuntu. (This answer and this one give some useful history). I've tested this and it does work.

   $ virtualenv --system-site-packages .

2. I agree with Thomas here - I can't see any action required in virtualenv to see the effect of updates in dist-packages.

3. Having tested that with python setup.py install, it does (again as Thomas said) still go to dist-packages. You could change that by building your own python, but that's a bit extreme.

Answer 3

PYTHONPATH works for me.

vim ~/.bashrc

add this line below:

export PYTHONPATH=$PYTHONPATH:/usr/lib/python2.7/dist-packages:/usr/local/lib/python2.7/dist-packages
source ~/.bashrc

Answer 4

In the directory site-packages, create a file dist.pth In the file dist.path, put the following: ../dist-packages

Now deactivate and activate your virtualenv. You should be set.

Answer 5

What you want to achieve here is essentially add specific folder (dist-packages) to Python search path. You have a number of options for this:

1. Use path configuration (.pth) file, entries will be appended to the system path.
2. Modify PYTHONPATH (entries from it go to the beginning of system path).
3. Modify sys.path directly from your Python script, i.e. append required folders to it.

I think that for this particular case (enable global dist-packages folder) third option is better, because with first option you have to create .pth file for every virtualenv you'll be working in (with some external shell script?). It's easy to forget it when you distribute your package. Second option requires run-time setup (add a envvar), which is, again, easy to miss.

And only third option doesn't require any prerequisites at configure- or run-time and can be distributed without issues (on the same-type system, of course).

You can use function like this:

def enable_global_distpackages():
    import sys
    sys.path.append('/usr/lib/python2.7/dist-packages')
    sys.path.append('/usr/local/lib/python2.7/dist-packages')

And then in __init__.py file of your package:

enable_global_distpackages()