How do you organise a python project that contains multiple packages so that each file in a package can still be run individually?

Question

TL;DR

Here\'s an example repository that is set up as described in the first diagram (below): https://github.com/Poddster/package_problems

I

Answer 1

Once you move to your desired configuration, the absolute imports you are using to load the modules that are specific to my_tool no longer work.

You need three modifications after you create the my_tool subdirectory and move the files into it:

1. Create my_tool/__init__.py. (You seem to already do this but I wanted to mention it for completeness.)

2. In the files directly under in my_tool: change the import statements to load the modules from the current package. So in my_tool.py change:

   import c
   import d
   import k
   import s
   

   to:

   from . import c
   from . import d
   from . import k
   from . import s
   

   You need to make a similar change to all your other files. (You mention having tried setting __package__ and then doing a relative import but setting __package__ is not needed.)

3. In the files located in my_tool/tests: change the import statements that import the code you want to test to relative imports that load from one package up in the hierarchy. So in test_my_tool.py change:

   import my_tool
   

   to:

   from .. import my_tool
   

   Similarly for all the other test files.

With the modifications above, I can run modules directly:

$ python -m my_tool.my_tool
C!
D!
F!
V!
K!
T!
S!
my_tool!
my_tool main!
|main tool!||detected||tar edit!||installed||keys||LOL||ssl connect||parse ASN.1||config|

$ python -m my_tool.k
F!
V!
K!
K main!
|keys||LOL||ssl connect||parse ASN.1|

and I can run tests:

$ nosetests 
........
----------------------------------------------------------------------
Ran 8 tests in 0.006s

OK

Note that I can run the above both with Python 2.7 and Python 3.

---

Rather than make the various modules under my_tool be directly executable, I suggest using a proper setup.py file to declare entry points and let setup.py create these entry points when the package is installed. Since you intend to distribute this code, you should use a setup.py to formally package it anyway.

1. Modify the modules that can be invoked from the command line so that, taking my_tool/my_tool.py as example, instead of this:

   if __name__ == "__main__":
       print("my_tool main!")
       print(do_something())
   

   You have:

   def main():
       print("my_tool main!")
       print(do_something())
   
   if __name__ == "__main__":
       main()
   

2. Create a setup.py file that contains the proper entry_points. For instance:

   from setuptools import setup, find_packages
   
   setup(
       name="my_tool",
       version="0.1.0",
       packages=find_packages(),
       entry_points={
           'console_scripts': [
               'my_tool = my_tool.my_tool:main'
           ],
       },
       author="",
       author_email="",
       description="Does stuff.",
       license="MIT",
       keywords=[],
       url="",
       classifiers=[
       ],
   )
   

   The file above instructs setup.py to create a script named my_tool that will invoke the main method in the module my_tool.my_tool. On my system, once the package is installed, there is a script located at /usr/local/bin/my_tool that invokes the main method in my_tool.my_tool. It produces the same output as running python -m my_tool.my_tool, which I've shown above.

Answer 2

Point 1

I believe it's working, so I don't comment on it.

Point 2

I always used tests at the same level as my_tool, not below it, but they should work if you do this at the top of each tests files (before importing my_tool or any other py file in the same directory)

import os
import sys

sys.path.insert(0, os.path.abspath(__file__).rsplit(os.sep, 2)[0])

Point 3

In my_second_package.py do this at the top (before importing my_tool)

import os
import sys

sys.path.insert(0,
                os.path.abspath(__file__).rsplit(os.sep, 2)[0] + os.sep
                + 'my_tool')

Best regards,

JM

Answer 3

To run it from both command line and act like library while allowing nosetest to operate in a standard manner, I believe you will have to do a double up approach on Imports.

For example, the Python files will require:

try:
    import f
except ImportError:
    import tools.f as f

I went through and made a PR off the github you linked with all test cases working.

https://github.com/Poddster/package_problems/pull/1

Edit: Forgot the imports in __init__.py to be properly usable in other packages, added. Now should be able to do:

import tools
tools.c.do_something()