Turn off buffering

Question

Where is the buffer in this following ... and how do I turn it off?

I am writing out to stdout in a python program like so:

for line in sys.stdin:
           


        

Answer 1

The problem is in your for loop. It will wait for EOF before continuing on. You can fix it with a code like this.

while True:
    try:
        line = sys.stdin.readline()
    except KeyboardInterrupt:
        break 

    if not line:
        break

    print line,

Try this out.

Answer 2

sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0) and make sure PYTHONUNBUFFERED is set in your environment.

Answer 3

The problem, I believe is in grep buffering its output. It is doing that when you pipe tail -f | grep ... | some_other_prog. To get grep to flush once per line, use the --line-buffered option:

% tail -f data.txt | grep -e APL --line-buffered | test.py
APL

APL

APL

where test.py is:

import sys
for line in sys.stdin:
    print(line)

(Tested on linux, gnome-terminal.)

Answer 4

file.readlines() and for line in file have internal buffering which is not affected by -u option (see -u option note). Use

while True:
   l=sys.stdin.readline()
   sys.stdout.write(l)

instead.

By the way, sys.stdout is line-buffered by default if it points to terminal and sys.stderr is unbuffered (see stdio buffering).