How to add package data recursively in Python setup.py?

问题

I have a new library that has to include a lot of subfolders of small datafiles, and I'm trying to add them as package data. Imagine I have my library as so:

 library
    - foo.py
    - bar.py
 data
   subfolderA
      subfolderA1
      subfolderA2
   subfolderB
      subfolderB1 
      ...

I want to add all of the data in all of the subfolders through setup.py, but it seems like I manually have to go into every single subfolder (there are 100 or so) and add an init.py file. Furthermore, will setup.py find these files recursively, or do I need to manually add all of these in setup.py like:

package_data={
  'mypackage.data.folderA': ['*'],
  'mypackage.data.folderA.subfolderA1': ['*'],
  'mypackage.data.folderA.subfolderA2': ['*']
   },

I can do this with a script, but seems like a super pain. How can I achieve this in setup.py?

PS, the hierarchy of these folders is important because this is a database of material files and we want the file tree to be preserved when we present them in a GUI to the user, so it would be to our advantage to keep this file structure intact.

回答1:

Use Setuptools instead of distutils.
Use data files instead of package data. These do not require __init__.py.

Generate the lists of files and directories using standard Python code, instead of writing it literally:

data_files = []
directories = glob.glob('data/subfolder?/subfolder??/')
for directory in directories:
    files = glob.glob(directory+'*')
    data_files.append((directory, files))
# then pass data_files to setup()

回答2:

The problem with the glob answer is that it only does so much. I.e. it's not fully recursive. The problem with the copy_tree answer is that the files that are copied will be left behind on an uninstall.

The proper solution is a recursive one which will let you set the package_data parameter in the setup call.

I've written this small method to do this:

import os

def package_files(directory):
    paths = []
    for (path, directories, filenames) in os.walk(directory):
        for filename in filenames:
            paths.append(os.path.join('..', path, filename))
    return paths

extra_files = package_files('path_to/extra_files_dir')

setup(
    ...
    packages = ['package_name'],
    package_data={'': extra_files},
    ....
)

You'll notice that when you do a pip uninstall package_name, that you'll see your additional files being listed (as tracked with the package).

回答3:

If you don't have any problem with getting your setup.py code dirty use distutils.dir_util.copy_tree.
The whole problem is how to exclude files from it.
Heres some the code:

import os.path
from distutils import dir_util
from distutils import sysconfig
from distutils.core import setup

__packagename__ = 'x' 
setup(
    name = __packagename__,
    packages = [__packagename__],
)

destination_path = sysconfig.get_python_lib()
package_path = os.path.join(destination_path, __packagename__)

dir_util.copy_tree(__packagename__, package_path, update=1, preserve_mode=0)

Some Notes:

This code recursively copy the source code into the destination path.

You can just use the same setup(...) but use copy_tree() to extend the directory you want into the path of installation.

The default paths of distutil installation can be found in it's API.

More information about copy_tree() module of distutils can be found here.

回答4:

Use glob to select all subfolders in your setup.py

...
packages=['your_package'],
package_data={'your_package': ['data/**/*']},
...

回答5:

I can suggest a little code to add data_files in setup():

data_files = []

start_point = os.path.join(__pkgname__, 'static')
for root, dirs, files in os.walk(start_point):
    root_files = [os.path.join(root, i) for i in files]
    data_files.append((root, root_files))

start_point = os.path.join(__pkgname__, 'templates')
for root, dirs, files in os.walk(start_point):
    root_files = [os.path.join(root, i) for i in files]
    data_files.append((root, root_files))

setup(
    name = __pkgname__,
    description = __description__,
    version = __version__,
    long_description = README,
    ...
    data_files = data_files,
)

回答6:

To add all the subfolders using package_data in setup.py: add the number of * entries based on you subdirectory structure

package_data={
  'mypackage.data.folderA': ['*','*/*','*/*/*'],
}

来源：https://stackoverflow.com/questions/27664504/how-to-add-package-data-recursively-in-python-setup-py

标签

python

distutils

setup.py