MANIFEST.in ignored on “python setup.py install” - no data files installed?

Question

Here's my stripped-down setup.py script with non-code stuff removed:

#!/usr/bin/env python

from distutils.core import setup
from whyteboard.misc import meta


setup(
    name = 'Whyteboard',
    version = meta.version,

    packages = ['whyteboard', 'whyteboard.gui', 'whyteboard.lib', 'whyteboard.lib.pubsub',
                'whyteboard.lib.pubsub.core', 'whyteboard.lib.pubsub.utils', 'whyteboard.misc'],

    py_modules = ['whyteboard'],
    scripts = ['whyteboard.py'],
)

MANIFEST.in:

include *.txt
include whyteboard-help/*.*
recursive-include locale *.mo
recursive-include images *.png

When I run "python setup.py install sdist" I get a nice .tar.gz with a "whyteboard-0.41" root folder, with my locale/ images/ and whyteboard-help/ folders inside. This also has my whyteboard.py script that launches my program from inside the whyteboard source package.

So:

whyteboard/
 ├── locale/
 ├── images
 ├── whyteboard-help/
 ├── whyteboard/
 │  ├── __init__.py
 │  └── other packages etc
 ├── whyteboard.py
 ├── README
 ├── setup.py
 └── CHANGELOG

This mirrors the source of my program, is how everything should be, and is correct.

However when I run "python setup.py install" none of my data files are written - only the "whyteboard" source package, and the whyteboard.py is placed in /usr/local/lib/python2.6/dist-packages/.

Ideally, I'd like the same directory structure as what's been generated in the .tar.gz file to be created in dist-packages, as this is how my program expects to look for its resources.

How can I get "install" to create this directory structure? It seems to be ignoring my manifest file, as far as I can tell.

Answer 1

Some notes in addition to Ned's answer (which hits on the core problem):

Distutils does not install Python packages and modules inside a per-project subdirectory within site-packages (or dist-packages on Debian/Ubuntu): they are installed directly into site-packages, as you've seen. So the containing whyteboard-xx directory in your sdist will not exist in the final installed form.

One implication of this is that you should be careful to name your data_files in a way that clarifies what project they belong to, because those files/directories are installed directly into the global site-packages directory, not inside any containing whyteboard directory.

Or you could instead make your data package_data of the whyteboard package (which means it needs to live inside that package, i.e. next to __init__.py), and then this isn't a problem.

Lastly, it doesn't make much sense to have both a whyteboard.py module in py_modules and a whyteboard/__init__.py package in packages. The two are mutually exclusive, and if you have both, the whyteboard.py module will be ignored by imports in favor of the package of the same name.

If whyteboard.py is just a script, and is not intended to be imported, then you should use the scripts option for it, and remove it from py_modules.

Answer 2

MANIFEST.in tells Distutils what files to include in the source distribution but it does not directly affect what files are installed. For that you need to include the appropriate files in the setup.py file, generally either as package data or as additional files.

Answer 3

I couldn't figure out why my MANIFEST.in file was being ignored when I ran python setup.py install - turns out include_package_data=True solves the problem. The package_data option isn't actually required.

Answer 4

Running python 2.6.1 on Mac OSX, I had absolutely no luck except by using the data_files parameter in setup.py. Everything with MANIFEST.in simply resulted in files being included in the dist package, but never installed. I checked some other packages and they were indeed using data_files to specify additional files.

I created a short function to help enumerate all the files from a directory tree in the

(target_dir, [file list]) format that data_files expects:

def gen_data_files(*dirs):
    results = []

    for src_dir in dirs:
        for root,dirs,files in os.walk(src_dir):
            results.append((root, map(lambda f:root + "/" + f, files)))
    return results

Now I can just call this inside my setup call:

setup(... data_files = gen_data_files("docs", "lib") ...

And everything in those trees gets installed.

Answer 5

You should use setuptools:

#!/usr/bin/env python

from setuptools import setup, find_packages
from whyteboard.misc import meta


setup(
  name = 'Whyteboard',
  version = meta.version,

  packages = find_packages(),
  include_package_data=True,

  py_modules = ['whyteboard'],
  scripts = ['whyteboard.py'],
)

This is not actually using the MANIFEST file to do the job, but it includes all the needed files.