Running shell commands from Python and printing the output in real time

Question

I want to write a function that will execute multiple shell commands one at a time and print what the shell returns in real time.

I currently have the following code

Answer 1

It is possible to handle stdin and stdout in different threads. That way one thread can be handling printing the output from stdout and another one writing new commands on stdin. However, since stdin and stdout are independent streams, I do not think this can guarantee the order between the streams. For the current example it seems to work as intended, though.

import subprocess
import threading

def stdout_printer(p):
    for line in p.stdout:
        print(line.rstrip())

commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", stdin=subprocess.PIPE, 
                     stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                     universal_newlines=True)

t = threading.Thread(target=stdout_printer, args=(p,))
t.start()

for command in commands:
    p.stdin.write((command + "\n"))
    p.stdin.flush()

p.stdin.close()
t.join()

Also, note that I am writing stdout line by line, which is normally OK, since it tends to be buffered and being generated a line (or more) at a time. I guess it is possible to handle an unbuffered stdout stream (or e.g. stderr) character-by-character instead, if that is preferable.

Answer 2

I believe you need something like this

commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
                 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
    p.stdin.write((command + "\n").encode("utf-8"))
out, err = p.communicate()
print("{}".format(out))
print("{}".format(err))

Answer 3

Assuming you want control of the output in your python code you might need to do something like this

import subprocess

def run_process(exe):
    'Define a function for running commands and capturing stdout line by line'
    p = subprocess.Popen(exe.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    return iter(p.stdout.readline, b'')


if __name__ == '__main__':
    commands = ["foo", "foofoo"]
    for command in commands:
        for line in run_process(command):
            print(line)