Reading/writing to a Popen() subprocess

Question

I'm trying to talk to a child process using the python subprocess.Popen() call. In my real code, I'm implementing a type of IPC, so I want to write some data, read the response, write some more data, read the response, and so on. Because of this, I cannot use Popen.communicate(), which otherwise works well for the simple case.

This code shows my problem. It never even gets the first response, hangs at the first "Reading result". Why? How can I make this work as I expect?

import subprocess
p = subprocess.Popen(["sed", 's/a/x/g'],
                     stdout = subprocess.PIPE,
                     stdin = subprocess.PIPE)

p.stdin.write("abc\n")
print "Reading result:"
print p.stdout.readline()

p.stdin.write("cat\n")
print "Reading result:"
print p.stdout.readline()

Answer 1

I would try to use Popen().communicate() if you can as it does a lot of nice things for you, but if you need to use Popen() exactly as you described, you'll need to set sed to flush its buffer after newlines with the -l option:

p = subprocess.Popen(['sed', '-l', 's/a/x/g'],
                     stdout=subprocess.PIPE,
                     stdin=subprocess.PIPE)

and your code should work fine

Answer 2

sed's output is buffered and only outputs its data until enough has been cumulated or the input stream is exhausted and closed.

Try this:

import subprocess
p = subprocess.Popen(["sed", 's/a/x/g'],
                     stdout = subprocess.PIPE,
                     stdin = subprocess.PIPE)

p.stdin.write("abc\n")
p.stdin.write("cat\n")
p.stdin.close()

print "Reading result 1:"
print p.stdout.readline()

print "Reading result 2:"
print p.stdout.readline()

Be aware that this cannot be done reliably which huge data as wriring to stdin blocks once the buffer is full. The best way to do is using communicate().