Python subprocess module, how do I give input to the first of series of piped commands?

Question

I am trying to use Python\'s subprocess module. What I require is to send input to the first process whose output becomes the input of the second process. The situation is b

Answer 1

I assume that cat, grep are just example commands otherwise you could use a pure Python solution without subprocesses e.g.:

for line in simple.splitlines():
    if "line" in line:
       print(line)

Or if you want to use grep:

from subprocess import Popen, PIPE

output = Popen(['grep', 'line'], stdin=PIPE, stdout=PIPE).communicate(simple)[0]
print output,

You can pass the output of the first command to the second one without storing it in a string first:

from subprocess import Popen, PIPE
from threading import Thread

# start commands in parallel
first = Popen(first_command, stdin=PIPE, stdout=PIPE)
second = Popen(second_command, stdin=first.stdout, stdout=PIPE)
first.stdout.close() # notify `first` if `second` exits 
first.stdout = None # avoid I/O on it in `.communicate()`

# feed input to the first command
Thread(target=first.communicate, args=[simple]).start() # avoid blocking

# get output from the second command at the same time
output = second.communicate()[0]
print output,

If you don't want to store all input/output in memory; you might need threads (to read/write in chunks without blocking) or a select loop (works on POSIX).

If there are multiple commands, it might be more readable just to use the shell directly as suggested by @Troels Folke or use a library such as plumbum that hides all the gory details of emulating the shell by hand.