How to create non-blocking continuous reading from `stdin`?

Question

I have a single process, which has been created like this:

p = subprocess.Popen(args   = \'./myapp\',
                     stdin  = subprocess.PIPE,
                     


        

Answer 1

p = subprocess.Popen(args   = './myapp',
                     stdin  = subprocess.PIPE,
                     stdout = subprocess.PIPE,
                     universal_newlines=True)

while p.poll() is None:
    data = p.stdout.readline()

This will create a non-blocking read of your process until the process exits. However, there are some cautions to be aware of here. For instance, if you would pipe stderr as well, but not read from it.. Then you will most likely fill a buffer or two and you will hang the program anyway. So always make sure you clear out any buffer I/O's when doing things manually.

A better alternative would be to use select.epoll() if possible, this is only available on unix systems but gives you a hell of a lot better performance and error handling :)

epoll = select.epoll()
epoll.register(p.stdout.fileno(), select.EPOLLHUP) # Use select.EPOLLIN for stdin.

for fileno, event in epoll.poll(1):
    if fileno == p.stdout.fileno():
        # ... Do something ...

NOTE: Remember that whenever a process expects input, it usually indicates this via stdout, so you'll still register STDOUT with select.epoll in order to check for "waiting for input". You can register select.EPOLLIN to check if input was given, but I hardly see the point because remember, that would what you choose to input to the process which you should already be aware is "happening".

Checking if the process expects input

You can use select.epoll to check if the process is awaiting input or not without blocking your application execution with the above example. But there are better alternatives.

Pexpect is one library that does it really well and works with SSH for instance.

It works a little bit different from subprocess but might be a good alternative.

Getting subprocess.popen to work with SSH

I'll redirect to another question+answer if this is what you're after (because SSH will spawn a stdin in a protected manner.

Python + SSH Password auth (no external libraries or public/private keys)?