Printing output in realtime from subprocess

Question

I'm trying to print stdout in realtime for a subprocess but it looks like stdout is buffered even with bufsize=0 and I can't figure out how to make it work, I always have a delay.

The code I tried :

p = subprocess.Popen(cmd, 
                     stdout=subprocess.PIPE, 
                     stderr=subprocess.STDOUT, 
                     bufsize=0)
line = p.stdout.readline()
while line:
    sys.stdout.write(line)
    sys.stdout.flush()
    # DO OTHER STUFF
    line = p.stdout.readline()

Also tried with for line in iter(p.stdout.readline, b'') instead of the while loop and with read(1) instead of readline(). Always the same result, the output gets delayed by a lot of seconds or minutes and multiple lines appear suddenly at once.

What I think happens :

bufsize is set to 0 ( it is set to 0 by default according to the docs ) so the lines piped top.stdout should be available immediately. But since p.stdout.readline() doesn't return immediately when a new line is piped, that means that it IS buffered, hence the multiple lines at once when the buffer is finally flushed to p.stdout.

What can I do to make it work ?

Answer 1

Thanks to pobrelkey who found the source of the problem. Indeed, the delay is due to the fact that the child is buffering its write to stdout because it is not writing to a tty. The child uses stdio which is line buffered when writing to a tty, else it is fully buffered.

I managed to get it to work by using pexpect instead of subprocess. pexpect uses a pseudo-tty and that's exactly what we need here :

p = pexpect.spawn(cmd,args,timeout=None) 
line = p.readline() 
while line:
    sys.stdout.write(line)
    sys.stdout.flush()
    # DO OTHER STUFF
    line = p.readline()

Or even better in my case :

p = pexpect.spawn(cmd,args,timeout=None,logfile=sys.stdout)
line = p.readline() 
while line:
    # DO OTHER STUFF
    line = p.readline()

No more delay !

More infos about pexpect : wiki

Answer 2

I would first make sure the subprocess itself doesn't buffer its output. If the subprocess is in turn a Python program, proceed to the paragraph below to see how to disable output buffering for Python processes.

As per Python, usually the problem is that Python by default buffers stderr and stdout even if you explicitly .flush() it from the code. The solution is to pass -u to Python when starting your program.

Also, you can just do for line in p.stdout instead of the tricky while loop.

P.S. actually I tried running your code (with cmd = ['cat', '/dev/urandom']) and without -u and it outputted everything in real time already; this is on OS X 10.8.

Answer 3

If you just want stdout of your child process to go to your stdout, why not just have the child process inherit stdout from your process?

subprocess.Popen(cmd, stdout=None, stderr=subprocess.STDOUT)