Dynamic communication between main and subprocess in Python

Question

I work in Python, and I want to find a workflow for enabling two processes (main-process and sub-process) to communicate with each other. By that, I mean t

Answer 1

It seems like pipe might be a suitable choice for your use case. Beware though that under normal circumstance both reading and writing end expect data to be written or consumed resp. Also make sure you do not get surprised by buffering (nothing come through because buffer would not get automatically flushed unless on expected boundary unless set accordingly).

A basic example of how two pipe (they are unidirectional) can be used between two processes:

import os

def child():
    """
    This function is executed in a child process.
    """
    infile = os.fdopen(r1)
    outfile = os.fdopen(w2, 'w', buffering=1)
    for line in infile:
        if line.rstrip() == 'quit':
            break
        outfile.write(line.upper())

def parent():
    """
    This function is executed in a parent process.
    """
    outfile = os.fdopen(w1, 'w', buffering=1)
    infile = os.fdopen(r2)
    print('Foo', file=outfile)
    print(infile.readline(), end='')
    print('bar', file=outfile)
    print(infile.readline(), end='')
    print('quit', file=outfile)

(r1, w1) = os.pipe()  # for parent -> child writes
(r2, w2) = os.pipe()  # for child -> parent writes

pid = os.fork()
if pid == 0:
    child()  # child code runs here.
elif pid > 0:
    parent()  # parent code runs here.
    os.waitpid(pid, 0)  # wait for child
else:
    raise RuntimeError("This should not have happened.")
# Once returned from corresponding function, both processes exit

Indeed it'd be easier and more practical to use subprocess and you likely want to exec another binary/file. Former would need to be told to not close (at least the relevant pipe) file descriptors, latter would require the pipe file descriptors to be inheritable (not have O_CLOEXEC flag set). Otherwise same as above.

Child code:

import os
import sys

infile = os.fdopen(int(sys.argv[1]))
outfile = os.fdopen(int(sys.argv[2]), 'w', buffering=1)

for line in infile:
    if line.rstrip() == 'quit':
        break
    outfile.write(line.upper())

Parent script:

import os
import subprocess

(r1, w1) = os.pipe2(0)  # for parent -> child writes
(r2, w2) = os.pipe2(0)  # for child -> parent writes

child = subprocess.Popen(['./child.py', str(r1), str(w2)], pass_fds=(r1, w2))
outfile = os.fdopen(w1, 'w', buffering=1)
infile = os.fdopen(r2)
print('Foo', file=outfile)
print(infile.readline(), end='')
print('bar', file=outfile)
print(infile.readline(), end='')
print('quit', file=outfile)
child.wait()

Come to think of that, I forgot to ask, whether child needs stdin/out for anything, or it could be used to get information in/out of it. That would be even simpler:

Child:

import sys

for line in sys.stdin:
    if line.rstrip() == 'quit':
        break
    print(line.upper(), end='', flush=True)

Parent:

import os
import subprocess

(r1, w1) = os.pipe2(0)  # for parent -> child writes
(r2, w2) = os.pipe2(0)  # for child -> parent writes

child = subprocess.Popen(['./child.py'], stdin=r1, stdout=w2)
outfile = os.fdopen(w1, 'w', buffering=1)
infile = os.fdopen(r2)
print('Foo', file=outfile)
print(infile.readline(), end='')
print('bar', file=outfile)
print(infile.readline(), end='')
print('quit', file=outfile)
child.wait()

As stated, it is not really python specific and these are just rough hints on how pipes as one option could be used.

Answer 2

You want to make a Popen object with subprocess.PIPE for standard input and output and use its file objects to communicate—rather than using one of the cantrips like run (and the older, more specific ones like check_output). The challenge is avoiding deadlock: it’s easy to land in a situation where each process is trying to write, the pipe buffers fill (because no one is reading from them), and everything hangs. You also have to remember to flush in both processes, to avoid having a request or response stuck in a file object’s buffer.

Popen.communicate is provided to avoid these issues, but it supports only a single string (rather than an ongoing conversation). The traditional solution is select, but it also works to use separate threads to send requests and read results. (This is one of the reasons to use CPython threads in spite of the GIL: each exists to run while the other is blocked, so there’s very little contention.) Of course, synchronization is then an issue, and you may need to do some work to make the multithreaded client act like a simple, synchronous function call on the outside.

Note that both processes need to flush, but it’s enough if either implements such non-blocking I/O; one normally does that job in the process that starts the other because that’s where it’s known to be necessary (and such programs are the exception).