Proper way to parse requirements file after pip upgrade to pip 10.x.x?

Question

So today I did found out that with the release of pip 10.x.x the req package changed its directory and can now be found under pip._internal.r

Answer 1

The solution of Scrotch only works until pip 19.0.3, in the pip >= 20 versions the PipSession module was refactored. Here is a solution for the imports that works for all pip versions:

try:
    # pip >=20
    from pip._internal.network.session import PipSession
    from pip._internal.req import parse_requirements
except ImportError:
    try:
        # 10.0.0 <= pip <= 19.3.1
        from pip._internal.download import PipSession
        from pip._internal.req import parse_requirements
    except ImportError:
        # pip <= 9.0.3
        from pip.download import PipSession
        from pip.req import parse_requirements

Answer 2

First, I believe parsing requirements.txt from within setup.py is not a good idea. It should be the other way around, install_requires in setup.py or setup.cfg should be considered as some kind of source of truth, and files such as requirements.txt should be generated from there. But everyone has different needs, that lead to different workflows.

So with that said...

It is possible to parse a relatively simple requirements.txt file from a setuptools setup.py script without pip. The setuptools project already contains necessary tools in its top level package pkg_resources.

It could more or less look like this:

#!/usr/bin/env python3

import pathlib

import pkg_resources
import setuptools

with pathlib.Path('requirements.txt').open() as requirements_txt:
    install_requires = [
        str(requirement)
        for requirement
        in pkg_resources.parse_requirements(requirements_txt)
    ]

setuptools.setup(
    install_requires=install_requires,
)

Again, this will work only for simple requirements.txt files. See Requirements parsing in the documentation page for pkg_resources to get details about what is handled. In short, each line should be a valid PEP 508 requirement. Notations that are really specific to pip are not handled.

---

Notes:

• See also this other answer: https://stackoverflow.com/a/59971469/11138259

Answer 3

with open("requirements.txt") as f:
    dependencies = [line for line in f if "==" in line]

setup(
    install_requires=dependencies
)

Answer 4

I don't agree with the accepted answer. The setup.py file can get ugly real fast if you have a large project with a lot of dependencies. It is always good practice to keep your requirements in a separate .txt file. I would do something like this -

try:  # for pip >= 10
    from pip._internal.req import parse_requirements
    from pip._internal.download import PipSession
except ImportError:  # for pip <= 9.0.3
    from pip.req import parse_requirements
    from pip.download import PipSession

requirements = parse_requirements(os.path.join(os.path.dirname(__file__), 'requirements.txt'), session=PipSession())

if __name__ == '__main__':
    setup(
        ...
        install_requires=[str(requirement.requirement) for requirement in requirements],
        ...
    )

Throw in all your requirements in requirements.txt under project root directory.

Answer 5

What I figured out the right way to do is adding the dependencies in the setup.py like:

REQUIRED_PACKAGES = [
    'boto3==1.7.33'
]

if __name__ == '__main__':
    setup(
        ...
        install_requires=REQUIRED_PACKAGES,
        ...
    )

and just have a . in your requirements.txt. It will then automatically install all packages from the setup.py if you install from the file.