tell pip to install the dependencies of packages listed in a requirement file

Question

Developing a Django web app, I have a list of packages I need to install in a virtualenv. Say:

Django==1.3.1
--extra-index-url=http://dist.pinaxproject.com/dev/
Pinax==0.9b1.dev10
git+git://github.com/pinax/pinax-theme-bootstrap.git@cff4f5bbe9f87f0c67ee9ada9aa8ae82978f9890
# and other packages

Initially I installed them manually, one by one, along the development. This installed the required dependencies and I finally used pip freeze before deploying the app.

Problem is, as I upgraded some packages, some dependencies are no longer used nor required but they keep being listed by pip freeze.

Now, I'd like to set up a new virtualenv this way:

• put the needed packages (without their dependencies) in a requirement file,
  like manual-requirements.txt
• install them with their dependencies
  pip install -r manual-requirement.txt (← problem, this does not install the dependencies)
• freeze the full virtualenv
  pip freeze -r manual-requirements.txt > full-requirements.txt
  and deploy.

Any way to do this without manually re-installing the packages in a new virtualenv to get their dependencies ? This would be error-prone and I'd like to automate the process of cleaning the virtualenv from no-longer-needed old dependencies.

edit: actually, pip does install dependencies not explicitly listed in the requirement file, even if the documentation tells us that such files are flat. I was wrong about which dependencies i expected to be installed. I'll let this question for anyone in doubt about pip not installing all dependencies.

Answer 1

Given your comment to the question (where you say that executing the install for a single package works as expected), I would suggest looping over your requirement file. In bash:

#!/bin/sh
while read p; do
  pip install $p
done < requirements.pip

HTH!

Answer 2

simplifily, use:

pip install -r requirement.txt

it can install all listed in requirement file.

Answer 3

Any way to do this without manually re-installing the packages in a new virtualenv to get their dependencies ? This would be error-prone and I'd like to automate the process of cleaning the virtualenv from no-longer-needed old dependencies.

That's what pip-tools package is for (from https://github.com/nvie/pip-tools):

Installation

$ pip install --upgrade pip  # pip-tools needs pip==6.1 or higher (!)
$ pip install pip-tools

Example usage for pip-compile

Suppose you have a Flask project, and want to pin it for production. Write the following line to a file:

# requirements.in
Flask

Now, run pip-compile requirements.in:

$ pip-compile requirements.in
#
# This file is autogenerated by pip-compile
# Make changes in requirements.in, then run this to update:
#
#    pip-compile requirements.in
#
flask==0.10.1
itsdangerous==0.24        # via flask
jinja2==2.7.3             # via flask
markupsafe==0.23          # via jinja2
werkzeug==0.10.4          # via flask

And it will produce your requirements.txt, with all the Flask dependencies (and all underlying dependencies) pinned. Put this file under version control as well and periodically re-run pip-compile to update the packages.

Example usage for pip-sync

Now that you have a requirements.txt, you can use pip-sync to update your virtual env to reflect exactly what's in there. Note: this will install/upgrade/uninstall everything necessary to match the requirements.txt contents.

$ pip-sync
Uninstalling flake8-2.4.1:
  Successfully uninstalled flake8-2.4.1
Collecting click==4.1
  Downloading click-4.1-py2.py3-none-any.whl (62kB)
    100% |████████████████████████████████| 65kB 1.8MB/s
  Found existing installation: click 4.0
    Uninstalling click-4.0:
      Successfully uninstalled click-4.0
Successfully installed click-4.1