can not read correctly from STDIN

问题

I have a weird problem to read from STDIN in a python script.

Here is my use case. I have rsyslog configured with an output module so rsyslog can pipe log messages to my Python script.

My Python script is really trivial :

#! /usr/bin/env python
# -*- coding: utf-8 -*-
import sys

fd = open('/tmp/testrsyslogomoutput.txt', 'a')
fd.write("Receiving log message : \n%s\n" % ('-'.join(sys.stdin.readlines())))
fd.close()

If I run echo "foo" | mypythonscript.py I can get "foo" in the target file /tmp/testrsyslogomoutput.txt. However when I run it within rsyslog, messages seems to be sent only when I stop/restart rsyslog (I believe some buffer is flushed at some point).

I first thought it was a problem with Rsyslog. So I replaced my python program with a shell one, without changing anything to the rsyslog configuration. The shell script works perfectly with rsyslog and as you can see in the code below, the script is really trivial:

#! /bin/sh
cat /dev/stdin >> /tmp/testrsyslogomoutput.txt

Since my shell script works but my Python one does not, I believe I made a mistake somewhere in my Python code but I can not find where. If you could point me to my mistake(s) that would be great.

Thanks in advance :)

回答1:

I'd also suspect the reason is that rsyslog does not terminate. readlines() should not return until it reaches a real EOF. But why would the shell script act differently? Perhaps the use of /dev/stdin is the reason. Try this version and see if it still runs without hanging:

#!/bin/sh
cat >> /tmp/testrsyslogomoutput.txt

If this makes a difference, you'll also have a fix: open and read /dev/stdin from python, instead of sys.stdin.

Edit: So cat somehow reads whatever is waiting at stdin and returns, but python blocks and waits until stdin is exhausted. Strange. You can also try replacing readlines() with a single read() followed by split("\n"), but at this point I doubt that will help.

So, forget the diagnosis and let's try a work-around: Force stdin to do non-blocking i/o. The following is supposed to do that:

import fcntl, os

# Add O_NONBLOCK to the stdin descriptor flags 
flags = fcntl.fcntl(0, fcntl.F_GETFL)
fcntl.fcntl(0, fcntl.F_SETFL, fl | os.O_NONBLOCK)

message = sys.stdin.read().split("\n")  # Read what's waiting, in one go
fd = open('/tmp/testrsyslogomoutput.txt', 'a')
fd.write("Receiving log message : \n%s\n" % ('-'.join(message)))
fd.close()

You probably want to use that in combination with python -u. Hope it works!

回答2:

readlines will not return until it has finished reading the file. Since the pipe feeding stdin never finishes, readlines never finishes either. Stopping rsyslog closes the pipe and lets it finish.

回答3:

If you use readline() instead, it will return on \n, though this will only write one line then quit.

If you want to keep writing lines as long they are there, you can use a simple for:

for line in fd:
  fd.write("Receiving log message : \n%s\n" % (line)
fd.close()

来源：https://stackoverflow.com/questions/9438200/can-not-read-correctly-from-stdin

标签

python

rsyslog