How do I write a setup.py for a twistd/twisted plugin that works with setuptools, distribute, etc?

Question

The Twisted Plugin System is the preferred way to write extensible twisted applications.

However, due to the way the plugin system is structured (plugins go into a twisted/plugins directory which should not be a Python package), writing a proper setup.py for installing those plugins appears to be non-trivial.

I've seen some attempts that add 'twisted.plugins' to the 'packages' key of the distutils setup command, but since it is not really a package, bad things happen (for example, an __init__.py is helpfully added by some tools).

Other attempts seem to use 'package_data' instead (eg, http://bazaar.launchpad.net/~glyph/divmod.org/trunk/view/head:/Epsilon/epsilon/setuphelper.py), but that can also fail in weird ways.

The question is: has anyone successfully written a setup.py for installing twisted plugins which works in all cases?

Answer 1

I document a setup.py below that is needed only if you have users with pip < 1.2 (e.g. on Ubuntu 12.04). If everyone has pip 1.2 or newer, the only thing you need is packages=[..., 'twisted.plugins'].

By preventing pip from writing the line "twisted" to .egg-info/top_level.txt, you can keep using packages=[..., 'twisted.plugins'] and have a working pip uninstall that doesn't remove all of twisted/. This involves monkeypatching setuptools/distribute near the top of your setup.py. Here is a sample setup.py:

from distutils.core import setup

# When pip installs anything from packages, py_modules, or ext_modules that
# includes a twistd plugin (which are installed to twisted/plugins/),
# setuptools/distribute writes a Package.egg-info/top_level.txt that includes
# "twisted".  If you later uninstall Package with `pip uninstall Package`,
# pip <1.2 removes all of twisted/ instead of just Package's twistd plugins.
# See https://github.com/pypa/pip/issues/355 (now fixed)
#
# To work around this problem, we monkeypatch
# setuptools.command.egg_info.write_toplevel_names to not write the line
# "twisted".  This fixes the behavior of `pip uninstall Package`.  Note that
# even with this workaround, `pip uninstall Package` still correctly uninstalls
# Package's twistd plugins from twisted/plugins/, since pip also uses
# Package.egg-info/installed-files.txt to determine what to uninstall,
# and the paths to the plugin files are indeed listed in installed-files.txt.
try:
    from setuptools.command import egg_info
    egg_info.write_toplevel_names
except (ImportError, AttributeError):
    pass
else:
    def _top_level_package(name):
        return name.split('.', 1)[0]

    def _hacked_write_toplevel_names(cmd, basename, filename):
        pkgs = dict.fromkeys(
            [_top_level_package(k)
                for k in cmd.distribution.iter_distribution_names()
                if _top_level_package(k) != "twisted"
            ]
        )
        cmd.write_file("top-level names", filename, '\n'.join(pkgs) + '\n')

    egg_info.write_toplevel_names = _hacked_write_toplevel_names

setup(
    name='MyPackage',
    version='1.0',
    description="You can do anything with MyPackage, anything at all.",
    url="http://example.com/",
    author="John Doe",
    author_email="jdoe@example.com",
    packages=['mypackage', 'twisted.plugins'],
    # You may want more options here, including install_requires=,
    # package_data=, and classifiers=
)

# Make Twisted regenerate the dropin.cache, if possible.  This is necessary
# because in a site-wide install, dropin.cache cannot be rewritten by
# normal users.
try:
    from twisted.plugin import IPlugin, getPlugins
except ImportError:
    pass
else:
    list(getPlugins(IPlugin))

I've tested this with pip install, pip install --user, and easy_install. With any install method, the above monkeypatch and pip uninstall work fine.

You might be wondering: do I need to clear the monkeypatch to avoid messing up the next install? (e.g. pip install --no-deps MyPackage Twisted; you wouldn't want to affect Twisted's top_level.txt.) The answer is no; the monkeypatch does not affect another install because pip spawns a new python for each install.

Related: keep in mind that in your project, you must not have a file twisted/plugins/__init__.py. If you see this warning during installation:

package init file 'twisted/plugins/__init__.py' not found (or not a regular file)

it is completely normal and you should not try to fix it by adding an __init__.py.

Answer 2

Here is a blog entry which describes doing it with 'package_data':

http://chrismiles.livejournal.com/23399.html

In what weird ways can that fail? It could fail if the installation of the package doesn't put the package data into a directory which is on the sys.path. In that case the Twisted plugin loader wouldn't find it. However, all installations of Python packages that I know of will put it into the same directory where they are installing the Python modules or packages themselves, so that won't be a problem.

Answer 3

Maybe you could adapt the package_data idea to use data_files instead: it wouldn’t require you to list twisted.plugins as package, as it uses absolute paths. It would still be a kludge, though.

My tests with pure distutils have told me that its is possible to overwrite files from another distribution. I wanted to test poor man’s namespace packages using pkgutil.extend_path and distutils, and it turns out that I can install spam/ham/__init__.py with spam.ham/setup.py and spam/eggs/__init__.py with spam.eggs/setup.py. Directories are not a problem, but files will be happily overwritten. I think this is actually undefined behavior in distutils which trickles up to setuptools and pip, so pip could IMO close as wontfix.

What is the usual way to install Twisted plugins? Drop-it-here by hand?

Answer 4

I use this approach:

1. Put '.py' and '.pyc' versions of your file to "twisted/plugins/" folder inside your package. Note that '.pyc' file can be empty, it just should exist.

2. In setup.py specify copying both files to a library folder (make sure that you will not overwrite existing plugins!). For example:

   # setup.py
   
   from distutils import sysconfig
   
   LIB_PATH = sysconfig.get_python_lib()
   
   # ...
   
   plugin_name = '<your_package>/twisted/plugins/<plugin_name>'
   # '.pyc' extension is necessary for correct plugins removing
   data_files = [
     (os.path.join(LIB_PATH, 'twisted', 'plugins'),
      [''.join((plugin_name, extension)) for extension in ('.py', '.pyc')])
   ]
   
   setup(
         # ...
         data_files=data_files
   )