Dynamic communication between main and subprocess in Python

Question

I work in Python, and I want to find a workflow for enabling two processes (main-process and sub-process) to communicate with each other. By that, I mean t

Answer 1

It seems like pipe might be a suitable choice for your use case. Beware though that under normal circumstance both reading and writing end expect data to be written or consumed resp. Also make sure you do not get surprised by buffering (nothing come through because buffer would not get automatically flushed unless on expected boundary unless set accordingly).

A basic example of how two pipe (they are unidirectional) can be used between two processes:

import os

def child():
    """
    This function is executed in a child process.
    """
    infile = os.fdopen(r1)
    outfile = os.fdopen(w2, 'w', buffering=1)
    for line in infile:
        if line.rstrip() == 'quit':
            break
        outfile.write(line.upper())

def parent():
    """
    This function is executed in a parent process.
    """
    outfile = os.fdopen(w1, 'w', buffering=1)
    infile = os.fdopen(r2)
    print('Foo', file=outfile)
    print(infile.readline(), end='')
    print('bar', file=outfile)
    print(infile.readline(), end='')
    print('quit', file=outfile)

(r1, w1) = os.pipe()  # for parent -> child writes
(r2, w2) = os.pipe()  # for child -> parent writes

pid = os.fork()
if pid == 0:
    child()  # child code runs here.
elif pid > 0:
    parent()  # parent code runs here.
    os.waitpid(pid, 0)  # wait for child
else:
    raise RuntimeError("This should not have happened.")
# Once returned from corresponding function, both processes exit

Indeed it'd be easier and more practical to use subprocess and you likely want to exec another binary/file. Former would need to be told to not close (at least the relevant pipe) file descriptors, latter would require the pipe file descriptors to be inheritable (not have O_CLOEXEC flag set). Otherwise same as above.

Child code:

import os
import sys

infile = os.fdopen(int(sys.argv[1]))
outfile = os.fdopen(int(sys.argv[2]), 'w', buffering=1)

for line in infile:
    if line.rstrip() == 'quit':
        break
    outfile.write(line.upper())

Parent script:

import os
import subprocess

(r1, w1) = os.pipe2(0)  # for parent -> child writes
(r2, w2) = os.pipe2(0)  # for child -> parent writes

child = subprocess.Popen(['./child.py', str(r1), str(w2)], pass_fds=(r1, w2))
outfile = os.fdopen(w1, 'w', buffering=1)
infile = os.fdopen(r2)
print('Foo', file=outfile)
print(infile.readline(), end='')
print('bar', file=outfile)
print(infile.readline(), end='')
print('quit', file=outfile)
child.wait()

Come to think of that, I forgot to ask, whether child needs stdin/out for anything, or it could be used to get information in/out of it. That would be even simpler:

Child:

import sys

for line in sys.stdin:
    if line.rstrip() == 'quit':
        break
    print(line.upper(), end='', flush=True)

Parent:

import os
import subprocess

(r1, w1) = os.pipe2(0)  # for parent -> child writes
(r2, w2) = os.pipe2(0)  # for child -> parent writes

child = subprocess.Popen(['./child.py'], stdin=r1, stdout=w2)
outfile = os.fdopen(w1, 'w', buffering=1)
infile = os.fdopen(r2)
print('Foo', file=outfile)
print(infile.readline(), end='')
print('bar', file=outfile)
print(infile.readline(), end='')
print('quit', file=outfile)
child.wait()

As stated, it is not really python specific and these are just rough hints on how pipes as one option could be used.