Python program using os.pipe and os.fork() issue

Question

I\'ve recently needed to write a script that performs an os.fork() to split into two processes. The child process becomes a server process and passes data b

Answer 1

Using

fcntl.fcntl(readPipe, fcntl.F_SETFL, os.O_NONBLOCK)

Before invoking the read() solved both problems. The read() call is no longer blocking and the data is appearing after just a flush() on the writing end.

Answer 2

I see you have solved the problem of blocking i/o and buffering.

A note if you decide to try a different approach: subprocess is the equivalent / a replacement for the fork/exec idiom. It seems like that's not what you're doing: you have just a fork (not an exec) and exchanging data between the two processes -- in this case the multiprocessing module (in Python 2.6+) would be a better fit.

Answer 3

Are you using read() without specifying a size, or treating the pipe as an iterator (for line in f)? If so, that's probably the source of your problem - read() is defined to read until the end of the file before returning, rather than just read what is available for reading. That will mean it will block until the child calls close().

In the example code linked to, this is OK - the parent is acting in a blocking manner, and just using the child for isolation purposes. If you want to continue, then either use non-blocking IO as in the code you posted (but be prepared to deal with half-complete data), or read in chunks (eg r.read(size) or r.readline()) which will block only until a specific size / line has been read. (you'll still need to call flush on the child)

It looks like treating the pipe as an iterator is using some further buffer as well, for "for line in r:" may not give you what you want if you need each line to be immediately consumed. It may be possible to disable this, but just specifying 0 for the buffer size in fdopen doesn't seem sufficient.

Heres some sample code that should work:

import os, sys, time

r,w=os.pipe()
r,w=os.fdopen(r,'r',0), os.fdopen(w,'w',0)

pid = os.fork()
if pid:          # Parent
    w.close()
    while 1:
        data=r.readline()
        if not data: break
        print "parent read: " + data.strip()
else:           # Child
    r.close()
    for i in range(10):
        print >>w, "line %s" % i
        w.flush()
        time.sleep(1)

Answer 4

The "parent" vs. "child" part of fork in a Python application is silly. It's a legacy from 16-bit unix days. It's an affectation from a day when fork/exec and exec were Important Things to make the most of a tiny little processor.

Break your Python code into two separate parts: parent and child.

The parent part should use subprocess to run the child part.

A fork and exec may happen somewhere in there -- but you don't need to care.