Run background process in Python and do NOT wait

Question

My goal is simple: kick off rsync and DO NOT WAIT.

Python 2.7.9 on Debian

Sample code:

rsync_cmd = "/usr/bin/rsync -a -e 'ssh -i /home/myuser/.ssh/id_rsa' {0}@{1}:'{2}' {3}".format(remote_user, remote_server, file1, file1)
rsync_cmd2 = "/usr/bin/rsync -a -e 'ssh -i /home/myuser/.ssh/id_rsa' {0}@{1}:'{2}' {3} &".format(remote_user, remote_server, file1, file1)
rsync_path = "/usr/bin/rsync"
rsync_args = shlex.split("-a -e 'ssh -i /home/mysuser/.ssh/id_rsa' {0}@{1}:'{2}' {3}".format(remote_user, remote_server, file1, file1))
#subprocess.call(rsync_cmd, shell=True)     # This isn't supposed to work but I tried it
#subprocess.Popen(rsync_cmd, shell=True)    # This is supposed to be the solution but not for me
#subprocess.Popen(rsync_cmd2, shell=True)   # Adding my own shell "&" to background it, still fails
#subprocess.Popen(rsync_cmd, shell=True, stdin=None, stdout=None, stderr=None, close_fds=True)  # This doesn't work
#subprocess.Popen(shlex.split(rsync_cmd))   # This doesn't work
#os.execv(rsync_path, rsync_args)           # This doesn't work
#os.spawnv(os.P_NOWAIT, rsync_path, rsync_args) # This doesn't work
#os.system(rsync_cmd2)                      # This doesn't work
print "DONE"

(I've commented out the execution commands only because I'm actually keeping all of my trials in my code so that I know what I've done and what I haven't done. Obviously, I would run the script with the right line uncommented.)

What happens is this...I can watch the transfer on the server and when it's finished, then I get a "DONE" printed to the screen.

What I'd like to have happen is a "DONE" printed immediately after issuing the rsync command and for the transfer to start.

Seems very straight-forward. I've followed details outlined in other posts, like this one and this one, but something is preventing it from working for me.

Thanks ahead of time.

(I have tried everything I can find in StackExchange and don't feel like this is a duplicate because I still can't get it to work. Something isn't right in my setup and need help.)

Answer 1

Here is verified example for Python REPL:

>>> import subprocess
>>> import sys
>>> p = subprocess.Popen([sys.executable, '-c', 'import time; time.sleep(100)'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT); print('finished')
finished

How to verify that via another terminal window:

$ ps aux | grep python

Output:

user           32820   0.0  0.0  2447684   3972 s003  S+   10:11PM   0:00.01 /Users/user/venv/bin/python -c import time; time.sleep(100)

Answer 2

Popen() starts a child process—it does not wait for it to exit. You have to call .wait() method explicitly if you want to wait for the child process. In that sense, all subprocesses are background processes.

On the other hand, the child process may inherit various properties/resources from the parent such as open file descriptors, the process group, its control terminal, some signal configuration, etc—it may lead to preventing ancestors processes to exit e.g., Python subprocess .check_call vs .check_output or the child may die prematurely on Ctrl-C (SIGINT signal is sent to the foreground process group) or if the terminal session is closed (SIGHUP).

To disassociate the child process completely, you should make it a daemon. Sometimes something in between could be enough e.g., it is enough to redirect the inherited stdout in a grandchild so that .communicate() in the parent would return when its immediate child exits.