Python distutils error: “[directory]… doesn't exist or not a regular file”

Question

Let's take the following project layout:

$ ls -R .
.:
package  setup.py

./package:
__init__.py  dir  file.dat  module.py

./package/dir:
tool1.dat  tool2.dat

And the following content for setup.py:

$ cat setup.py 
from distutils.core import setup


setup(name='pyproj',
      version='0.1',

      packages=[
          'package',
      ],
      package_data={
          'package': [
              '*',
              'dir/*',
          ],
      },
     )

As you can see, I want to include all non-Python files in package/ and package/dir/ directories. However, running setup.py install would raise the following error:

$ python setup.py install
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/package
copying package/module.py -> build/lib/package
copying package/__init__.py -> build/lib/package
error: can't copy 'package/dir': doesn't exist or not a regular file

What gives?

Answer 1

In your package_data, your '*' glob will match package/dir itself, and try to copy that dir as a file, resulting in a failure. Find a glob that won't match the directory package/dir, rewriting your setup.py along these lines:

from distutils.core import setup

setup(name='pyproj',
      version='0.1',

      packages=[
          'package',
      ],
      package_data={
          'package': [
              '*.dat',
              'dir/*'
          ],
      },
     )

Given your example, that's just changing '*' to '*.dat', although you'd probably need to refine your glob more than that, just ensure it won't match 'dir'

Answer 2

You could use Distribute instead of distutils. It works basically the same (for the most part, you will not have to change your setup.py) and it gives you the exclude_package_data option:

from distribute_setup import use_setuptools
use_setuptools()

from setuptools import setup

setup(name='pyproj',
      version='0.1',

      packages=[
          'package',
      ],
      package_data={
          'package': [
              '*.dat',
              'dir/*'
          ],
      },
      exclude_package_data={
          'package': [
              'dir'
          ],
      },
     )

Answer 3

I created a function that gives me all the files that I need

def find_files(directory, strip):
  """
  Using glob patterns in ``package_data`` that matches a directory can
  result in setuptools trying to install that directory as a file and
  the installation to fail.

  This function walks over the contents of *directory* and returns a list
  of only filenames found. The filenames will be stripped of the *strip*
  directory part.
  """

  result = []
  for root, dirs, files in os.walk(directory):
    for filename in files:
      filename = os.path.join(root, filename)
      result.append(os.path.relpath(filename, strip))
  return result

And used that as arugment for package_data

Answer 4

Not quite sure why, but after some troubleshooting I realised that renaming the directories that had dots in their names solved the problem. E.g.

chart.js-2.4.0 => chart_js-2_4_0

Note: I'm using Python 2.7.10, SetupTools 12.2