How do you read from stdin in python from a pipe which has no ending

Question

I've problem to read from Standard input or pipe in python when the pipe is from a "open" (do not know right name) file.

I have as example pipetest.py:

import sys
import time
k = 0
try:
   for line in sys.stdin:
      k = k + 1
      print line
except KeyboardInterrupt:
   sys.stdout.flush()
   pass
print k

I run a program that have continues output and Ctrl+c after a while

$ ping 127.0.0.1 | python pipetest.py
^C0

I get no output. But if I go via an ordinary file it works.

$ ping 127.0.0.1 > testfile.txt

this is ended by Ctrl+c after a short while

$ cat testfile.txt |  python pipetest.py

PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.017 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.015 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.014 ms
64 bytes from 127.0.0.1: icmp_seq=4 ttl=64 time=0.013 ms
64 bytes from 127.0.0.1: icmp_seq=5 ttl=64 time=0.012 ms

--- 127.0.0.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3998ms
rtt min/avg/max/mdev = 0.012/0.014/0.017/0.003 ms
10

How do I do to get any output before the program ends, in this case ping has ended?

Answer 1

Try the next:

import sys
import time
k = 0
try:
    buff = ''
    while True:
        buff += sys.stdin.read(1)
        if buff.endswith('\n'):
            print buff[:-1]
            buff = ''
            k = k + 1
except KeyboardInterrupt:
   sys.stdout.flush()
   pass
print k

Answer 2

For this to work without waiting until the stdin stream ends, you can iter on the readline. I think this is the simplest solution.

import sys
k = 0
try:
   for line in iter(sys.stdin.readline, b''):
      k = k + 1
      print line
except KeyboardInterrupt:
   sys.stdout.flush()
   pass
print k

Answer 3

k = 0
try:
    while True:
        print sys.stdin.readline()
        k += 1
except KeyboardInterrupt:
    sys.stdout.flush()
    pass
print k

Answer 4

while sys.stdin is a file-like object, meaning you can iterate over its lines, it will block until a EOF is inserted.

The behaviour can be described with the following pseudo-code:

while True:
    input = ""
    c = stdin.read(1)
    while c is not EOF:
        input += c
        c = stdin.read(1)
    for line in input.split('\n'):
        yield line

this means that, while you can iterate over sys.stdin's lines, you cannot use this approach for the task at hand and you must explicitly call read() or readline()

Answer 5

This is how I ended up doing this. I didn't really like any of the other solutions, they didn't seem very pythonic.

This will make a container for any open file input to iterate over all the lines. This will also take care of closing the file at the end of the context manager.

I feel like this is probably how the for line in sys.stdin: block should operate by default.

class FileInput(object):                                                        
    def __init__(self, file):                                                   
        self.file = file                                                       

    def __enter__(self):                                                        
        return self                                                             

    def __exit__(self, *args, **kwargs):                                        
        self.file.close()                                                       

    def __iter__(self):                                                         
        return self                                                             

    def next(self):                                                             
        line = self.file.readline()                                             

        if line == None or line == "":                                          
            raise StopIteration                                                 

        return line  

with FileInput(sys.stdin) as f:
    for line in f:
        print f

with FileInput(open('tmpfile') as f:
    for line in f:
        print f

From the command line both of the following should work:

tail -f /var/log/debug.log | python fileinput.py
cat /var/log/debug.log | python fileinput.py