Real-time intercepting of stdout from another process in Python

Question

I\'d like to run a system process, intercept the output, and modify it real-time, line by line, in a Python script.

My best attempt, which waits for the process to c

Answer 1

Looping over a file unavoidably buffers things in pretty large chunks -- a known issue with all Python 2.* implementations. It works as you intend in Python 3.1, with the final loop being slightly different:

for line in proc.stdout:
    print(">>> " + str(line.rstrip()))

If upgrading to Python 3.1 is impractical (and I know it will often be!), go the other way and write the loop in an old-fashioned manner -- the following version of the loop does work as you intend in Python 2.*:

while True:
    line = proc.stdout.readline()
    if not line:
        break
    print ">>> " + line.rstrip()

Answer 2

This whole thing can be encapsulated in an iterator as:

def subprocess_readlines(out):
    while True:
        line = out.readline()
        if not line:
            return
        yield line

And called as:

for line in subprocess_readlines(proc.stdout):
    print ">>>", line.rstrip()