Python parsing log file to extract events in real time

Question

I\'ve a process that is logging messages to a file.

I want to implement another process (in Python) that parses these logs (as they are written to the file), filter

Answer 1

C programs usually seek to the current position to clear any “end of file” flags. But as @9000 correctly pointed out, python apparently takes care of this, so you can read from the same file repeatedly even if it has reached end of file.

You might have to take care of incomplete lines, though. If your application writes its log in pieces, then you want to make sure that you handle whole lines, and not those pieces. The following code will accomplish that:

f = open('some.log', 'r')
while True:
    line = ''
    while len(line) == 0 or line[-1] != '\n':
        tail = f.readline()
        if tail == '':
            time.sleep(0.1)          # avoid busy waiting
            # f.seek(0, io.SEEK_CUR) # appears to be unneccessary
            continue
        line += tail
    process(line)