How do I write a setup.py for a twistd/twisted plugin that works with setuptools, distribute, etc?

Question

The Twisted Plugin System is the preferred way to write extensible twisted applications.

However, due to the way the plugin system is structured (plugins go into a

Answer 1

I document a setup.py below that is needed only if you have users with pip < 1.2 (e.g. on Ubuntu 12.04). If everyone has pip 1.2 or newer, the only thing you need is packages=[..., 'twisted.plugins'].

By preventing pip from writing the line "twisted" to .egg-info/top_level.txt, you can keep using packages=[..., 'twisted.plugins'] and have a working pip uninstall that doesn't remove all of twisted/. This involves monkeypatching setuptools/distribute near the top of your setup.py. Here is a sample setup.py:

from distutils.core import setup

# When pip installs anything from packages, py_modules, or ext_modules that
# includes a twistd plugin (which are installed to twisted/plugins/),
# setuptools/distribute writes a Package.egg-info/top_level.txt that includes
# "twisted".  If you later uninstall Package with `pip uninstall Package`,
# pip <1.2 removes all of twisted/ instead of just Package's twistd plugins.
# See https://github.com/pypa/pip/issues/355 (now fixed)
#
# To work around this problem, we monkeypatch
# setuptools.command.egg_info.write_toplevel_names to not write the line
# "twisted".  This fixes the behavior of `pip uninstall Package`.  Note that
# even with this workaround, `pip uninstall Package` still correctly uninstalls
# Package's twistd plugins from twisted/plugins/, since pip also uses
# Package.egg-info/installed-files.txt to determine what to uninstall,
# and the paths to the plugin files are indeed listed in installed-files.txt.
try:
    from setuptools.command import egg_info
    egg_info.write_toplevel_names
except (ImportError, AttributeError):
    pass
else:
    def _top_level_package(name):
        return name.split('.', 1)[0]

    def _hacked_write_toplevel_names(cmd, basename, filename):
        pkgs = dict.fromkeys(
            [_top_level_package(k)
                for k in cmd.distribution.iter_distribution_names()
                if _top_level_package(k) != "twisted"
            ]
        )
        cmd.write_file("top-level names", filename, '\n'.join(pkgs) + '\n')

    egg_info.write_toplevel_names = _hacked_write_toplevel_names

setup(
    name='MyPackage',
    version='1.0',
    description="You can do anything with MyPackage, anything at all.",
    url="http://example.com/",
    author="John Doe",
    author_email="jdoe@example.com",
    packages=['mypackage', 'twisted.plugins'],
    # You may want more options here, including install_requires=,
    # package_data=, and classifiers=
)

# Make Twisted regenerate the dropin.cache, if possible.  This is necessary
# because in a site-wide install, dropin.cache cannot be rewritten by
# normal users.
try:
    from twisted.plugin import IPlugin, getPlugins
except ImportError:
    pass
else:
    list(getPlugins(IPlugin))

I've tested this with pip install, pip install --user, and easy_install. With any install method, the above monkeypatch and pip uninstall work fine.

You might be wondering: do I need to clear the monkeypatch to avoid messing up the next install? (e.g. pip install --no-deps MyPackage Twisted; you wouldn't want to affect Twisted's top_level.txt.) The answer is no; the monkeypatch does not affect another install because pip spawns a new python for each install.

Related: keep in mind that in your project, you must not have a file twisted/plugins/__init__.py. If you see this warning during installation:

package init file 'twisted/plugins/__init__.py' not found (or not a regular file)

it is completely normal and you should not try to fix it by adding an __init__.py.