Is there any way to pass 'stdin' as an argument to another process in python?

Question

I'm trying to create a script which is using multiprocessing module with python. The script (lets call it myscript.py) will get the input from another script with pipe.

Assume that I call the scripts like this;

$ python writer.py | python myscript.py 

And here is the codes;

// writer.py
import time, sys

def main():
    while True:
        print "test"
        sys.stdout.flush()
        time.sleep(1)

main()

//myscript.py
def get_input():
    while True:
        text = sys.stdin.readline()
        print "hello " + text
        time.sleep(3)

if __name__ == '__main__':        
    p1 = Process(target=get_input, args=())
    p1.start()

this is clearly not working, since the sys.stdin objects are different for main process and p1. So I have tried this to solve it,

//myscript.py
def get_input(temp):
    while True:
        text = temp.readline()
        print "hello " + text
        time.sleep(3)

if __name__ == '__main__':        
    p1 = Process(target=get_input, args=(sys.stdin,))
    p1.start()

but I come across with this error;

Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "in.py", line 12, in get_input
    text = temp.readline()
ValueError: I/O operation on closed file

So, I guess that main's stdin file closed and I can't read from it. At this conjunction, how can I pass main's stdin file to another process? If passing stdin is not possible, how can I use main's stdin from another process?

update: Okay, I need to clarify my question since people think using multiprocessing is not really necessary. consider myscript.py like this;

//myscript.py
def get_input():
    while True:
        text = sys.stdin.readline()
        print "hello " + text
        time.sleep(3)

def do_more_things():
    while True:
        #// some code here
        time.sleep(60*5)

if __name__ == '__main__':        
    p1 = Process(target=get_input, args=())
    p1.start()

    do_more_things()

so, I really need to run get_input() function parallelly with main function (or other sub processes). Sorry for the conflicts, I have a decent English, and I guess I couldn't be clear on this question. I would appreciate if you guys can tell me if i can use the main processes STDIN object in another process.

thanks in advance.

Answer 1

The simplest thing is to swap get_input() and do_more_things() i.e., read sys.stdin in the parent process:

def get_input(stdin):
    for line in iter(stdin.readline, ''):
        print("hello", line, end='')
    stdin.close()

if __name__ == '__main__':
    p1 = mp.Process(target=do_more_things)
    p1.start()
    get_input(sys.stdin)

The next best thing is to use a Thread() instead of a Process() for get_input():

if __name__ == '__main__':
    t = Thread(target=get_input, args=(sys.stdin,))
    t.start()
    do_more_things()

If the above doesn't help you could try os.dup():

newstdin = os.fdopen(os.dup(sys.stdin.fileno()))
try: 
   p = Process(target=get_input, args=(newstdin,))
   p.start()    
finally:
   newstdin.close() # close in the parent
do_more_things()

Answer 2

Each new process created with the multiprocessing module gets its own PID, and therefore it's own standard input device and output devices, even if they're both writing to the same terminal, hence the need for locks.

You're already creating two processes by separating the content into two scripts, and creating a third process with get_input(). get_input could read the standard input if it was a thread instead of a process. Then, no need to have a sleep function in the reader.

## reader.py
from threading import Thread
import sys

def get_input():
    text = sys.stdin.readline()
    while len(text) != 0:
        print 'hello ' + text
        text = sys.stdin.readline()

if __name__ == '__main__':
    thread = Thread(target=get_input)
    thread.start()
    thread.join()

Answer 3

This will only be a partial answer - as I'm unclear about subsequent parts of the question.

You start by saying that you anticipate calling your scripts:

$ python writer.py | python myscript.py 

If you're going to do that, writer needs to write to standard out and myscript needs to read from standard input. The second script would look like this:

def get_input():
    while True:
        text = sys.stdin.readline()
        print "hello " + text
        time.sleep(3)
if __name__ == '__main__':    
    get_input()

There's no need for the multiprocessing.Process object... you're firing off two processes from the command line already - and you're using the shell to connect them with an (anonymous) pipe (the "|" character) that connects standard output from the first script to standard input from the second script.

The point of the Process object is to manage launch of a second process from the first. You'd need to define a process; then start it - then you'd probably want to wait until it has terminated before exiting the first process... (calling p1.join() after p1.start() would suffice for this).

If you want to communicate between a pair of processes under python control, you'll probably want to use the multiprocess.Pipe object to do so. You can then easily communicate between the inital and the subordinate spawned process by reading and writing to/from the Pipe object rather than standard input and standard output. If you really want to re-direct standard input and standard output, this is probably possible by messing with low-level file-descriptors and/or by overriding/replacing the sys.stdin and sys.stdout objects... but, I suspect, you probably don't want (or need) to do this.

Answer 4

To read the piped in input use fileinput:

myscript.py

import fileinput

if __name__ == '__main__':
    for line in fileinput.input():
        #do stuff here
        process_line(line)