Is it possible to include subdirectories using dist utils (setup.py) as part of package data?

Question

Basically my python package is setup like:

module
\_examples
  \_folder1
     \_file1.py
     \_file2.py
  \_folder2
    \_file1.py
    \_file2.py

Basically I want to just use:

package_data  = { 
            'module': ['examples/*'],
  },

because my project always has people adding examples and I want it to be easy to list them from within my application. I can get it work for any FILE within examples, but not re-curse down through sub-directories. Is this possible?

Answer 1

I believe what you're looking for is something like this for you setup.py, which will recursively find any packages in the project, also be sure and include __init__.py files to subdirectories for each package you want.

from setuptools import setup, find_packages

setup(name='MySoftware',
      packages=find_packages()
)

Answer 2

You'll have to use a MANIFEST.in file for that.

I believe you'll want something like this:

$ cat MANIFEST.in
recursive-include examples/ *.py

Answer 3

Introduction

I came across this post and spent some time figuring out how to add specific sub-modules to my package, so I will post my solution here.

Solution

In my package root folder, I have a setup.py file see doc
In this file, I have the following code:

from setuptools import setup

with open("README.md", "r") as fh:
    long_description = fh.read()

setup(
    name='package name',
    version='0.4.1',
    description='short description',
    long_description=long_description,
    long_description_content_type="text/markdown",
    url='repository url',
    author='My name',
    author_email='my@e.mail',
    license='MIT',
    packages=['PackageName','PackageName.SubModule'],
    zip_safe=False,
    install_requires=[
        'dependecy1',
    ],
    classifiers=[
        'Development Status :: 3 - Alpha',
        'License :: OSI Approved :: MIT License',
        'Programming Language :: Python :: 3.7'
    ]
)

The interesting part to answer the question, here is : packages=['PackageName','PackageName.SubModule'],

By following this syntax, you can include sub-modules to your main package distribution.

More info about all the others arguments can be found in the doc.

Answer 4

Yes, you can include all the subdirectories.

You just need to pass the below args to setup() function:

packages=find_packages()

include_package_data=True

Along with this you need to have a MANIFEST.in file, with contents as

recursive-include examples *

This ensures all the files are recursively included.

If you want to exclude certain extensions specifically, you can do so by specifying exclude array in the find_packages() argument.

Ex: To exclude .txt files

packages=find_packages(exclude=['.txt'])

You can read more about include_package_data here.

And also here is another link which tells you when you should not use include_package_data

Answer 5

Following what David Wolever said, just to make it a little more clear. If you want to include everything under a sub-directory folder you have to explicitly specify each file in the MANIFEST.in,

recursive-include examples/ *.py *.png *.sh etc.....

It would be nice if the manifest.in would just understand examples/ and include everything but oh well.

Answer 6

None of the suggested answers worked for me in a similar situation.

I needed to make a distribution with my package, which included several sub-modules in a sub-directory, so that these were the files I needed to go into sdist:

ipyexperiments/*py
ipyexperiments/utils/*py

and no matter what I tried, the subdir utils's modules were not getting included in sdist.

What worked for me is leaving config.py's default:

# config.py
from setuptools import setup, find_packages
[...]
setup(
    packages = find_packages(),
    [...]
)

but adding to MANIFEST.in:

# MANIFEST.in
graft ipyexperiments

and everything under ipyexperiments was included.

I also added to MANIFEST.in

prune tests
global-exclude *.py[co]

to exclude all of tests directory and any unwanted *pyc and *.pyo files anywhere.