Python subprocess module, how do I give input to the first of series of piped commands?

Question

I am trying to use Python\'s subprocess module. What I require is to send input to the first process whose output becomes the input of the second process. The situation is b

Answer 1

If you do

from subprocess import Popen, PIPE
p1 = Popen(["cat"], stdout=PIPE, stdin=PIPE)

You should do p1.communicate("Your Input to the p1") and that will flow through the PIPE. The stdin is the process's input and you should communicate to that only.

The program which have given is absolutely fine, there seems no problem with that.

Answer 2

I assume that cat, grep are just example commands otherwise you could use a pure Python solution without subprocesses e.g.:

for line in simple.splitlines():
    if "line" in line:
       print(line)

Or if you want to use grep:

from subprocess import Popen, PIPE

output = Popen(['grep', 'line'], stdin=PIPE, stdout=PIPE).communicate(simple)[0]
print output,

You can pass the output of the first command to the second one without storing it in a string first:

from subprocess import Popen, PIPE
from threading import Thread

# start commands in parallel
first = Popen(first_command, stdin=PIPE, stdout=PIPE)
second = Popen(second_command, stdin=first.stdout, stdout=PIPE)
first.stdout.close() # notify `first` if `second` exits 
first.stdout = None # avoid I/O on it in `.communicate()`

# feed input to the first command
Thread(target=first.communicate, args=[simple]).start() # avoid blocking

# get output from the second command at the same time
output = second.communicate()[0]
print output,

If you don't want to store all input/output in memory; you might need threads (to read/write in chunks without blocking) or a select loop (works on POSIX).

If there are multiple commands, it might be more readable just to use the shell directly as suggested by @Troels Folke or use a library such as plumbum that hides all the gory details of emulating the shell by hand.

Answer 3

Hmm, why not mix in a bit of (ba)sh? :-)

from subprocess import Popen, PIPE
cproc = Popen('cat | grep line', stdin=PIPE, stdout=PIPE, stderr=PIPE, shell=True)
out, err = cproc.communicate("this line has the word line in it")

BEWARE though:

• This only works on systems that use a Bourne Shell compatible shell (like most *nix'es)

• Usign shell=True and putting user input in the command string is a bad idea, unless you escape the user input first. Read the subprocess docs -> "Frequently Used Arguments" for details.

• This is ugly, non portable, non pythonic and so on...

EDIT: There is no need to use cat though, if all you want to do is grep. Just feed the input directly to grep, or even better, use python regular expressions.