How do I implement a simple cross platform Python daemon?

Question

I would like to have my Python program run in the background as a daemon, on either Windows or Unix. I see that the python-daemon package is for Unix only; is there an alternati

Answer 1

This question is 6 years old, but I had the same problem, and the existing answers weren't cross-platform enough for my use case. Though Windows services are often used in similar ways as Unix daemons, at the end of the day they differ substantially, and "the devil's in the details". Long story short, I set out to try and find something that allows me to run the exact same application code on both Unix and Windows, while fulfilling the expectations for a well-behaved Unix daemon (which is better explained elsewhere) as best as possible on both platforms:

1. Close open file descriptors (typically all of them, but some applications may need to protect some descriptors from closure)
2. Change the working directory for the process to a suitable location to prevent "Directory Busy" errors
3. Change the file access creation mask (os.umask in the Python world)
4. Move the application into the background and make it dissociate itself from the initiating process
5. Completely divorce from the terminal, including redirecting STDIN, STDOUT, and STDERR to different streams (often DEVNULL), and prevent reacquisition of a controlling terminal
6. Handle signals, in particular, SIGTERM.

The fundamental problem with cross-platform daemonization is that Windows, as an operating system, really doesn't support the notion of a daemon: applications that start from a terminal (or in any other interactive context, including launching from Explorer, etc) will continue to run with a visible window, unless the controlling application (in this example, Python) has included a windowless GUI. Furthermore, Windows signal handling is woefully inadequate, and attempts to send signals to an independent Python process (as opposed to a subprocess, which would not survive terminal closure) will almost always result in the immediate exit of that Python process without any cleanup (no finally:, no atexit, no __del__, etc).

Windows services (though a viable alternative in many cases) were basically out of the question for me: they aren't cross-platform, and they're going to require code modification. pythonw.exe (a windowless version of Python that ships with all recent Windows Python binaries) is closer, but it still doesn't quite make the cut: in particular, it fails to improve the situation for signal handling, and you still cannot easily launch a pythonw.exe application from the terminal and interact with it during startup (for example, to deliver dynamic startup arguments to your script, say, perhaps, a password, file path, etc), before "daemonizing".

In the end, I settled on using subprocess.Popen with the creationflags=subprocess.CREATE_NEW_PROCESS_GROUP keyword to create an independent, windowless process:

import subprocess

independent_process = subprocess.Popen(
    '/path/to/pythonw.exe /path/to/file.py',
    creationflags=subprocess.CREATE_NEW_PROCESS_GROUP
)

However, that still left me with the added challenge of startup communications and signal handling. Without going into a ton of detail, for the former, my strategy was:

1. pickle the important parts of the launching process' namespace
2. Store that in a tempfile
3. Add the path to that file in the daughter process' environment before launching
4. Extract and return the namespace from the "daemonization" function

For signal handling I had to get a bit more creative. Within the "daemonized" process:

1. Ignore signals in the daemon process, since, as mentioned, they all terminate the process immediately and without cleanup
2. Create a new thread to manage signal handling
3. That thread launches daughter signal-handling processes and waits for them to complete
4. External applications send signals to the daughter signal-handling process, causing it to terminate and complete
5. Those processes then use the signal number as their return code
6. The signal handling thread reads the return code, and then calls either a user-defined signal handler, or uses a cytpes API to raise an appropriate exception within the Python main thread
7. Rinse and repeat for new signals

That all being said, for anyone encountering this problem in the future, I've rolled a library called daemoniker that wraps both proper Unix daemonization and the above Windows strategy into a unified facade. The cross-platform API looks like this:

from daemoniker import Daemonizer

with Daemonizer() as (is_setup, daemonizer):
    if is_setup:
        # This code is run before daemonization.
        do_things_here()

    # We need to explicitly pass resources to the daemon; other variables
    # may not be correct
    is_parent, my_arg1, my_arg2 = daemonizer(
        path_to_pid_file,
        my_arg1,
        my_arg2
    )

    if is_parent:
        # Run code in the parent after daemonization
        parent_only_code()

# We are now daemonized, and the parent just exited.
code_continues_here()