tail -f in python with no time.sleep

Question

I need to emulate \"tail -f\" in python, but I don\'t want to use time.sleep in the reading loop. I want something more elegant like some kind of blocking read, or select.se

Answer 1

To minimize the sleep issues I modified Tzury Bar Yochay's solution and now it polls quickly if there is activity and after a few seconds of no activity it only polls every second.

import time

def follow(thefile):
    thefile.seek(0,2)      # Go to the end of the file
    sleep = 0.00001
    while True:
        line = thefile.readline()
        if not line:
            time.sleep(sleep)    # Sleep briefly
            if sleep < 1.0:
                sleep += 0.00001
            continue
        sleep = 0.00001
        yield line

logfile = open("/var/log/system.log")
loglines = follow(logfile)
for line in loglines:
    print line,

Answer 2

Most implementations I've seen use readlines() / sleep(). A solution based on inotify or similar might be faster but consider this:

• once libinotify tells you a file has changed you would end up using readlines() anyway

• calling readlines() against a file which hasn't changed, which is what you would end up doing without libinotify, is already a pretty fast operation:

  giampaolo@ubuntu:~$ python -m timeit -s "f = open('foo.py', 'r'); f.read()" -c "f.readlines()" 1000000 loops, best of 3: 0.41 usec per loop

Having said this, considering that any solution similar to libinotify has portability issues, I might reconsider using readlines() / sleep(). See: http://code.activestate.com/recipes/577968-log-watcher-tail-f-log/

Answer 3

If you can use GLib on all platforms, you should use glib.io_add_watch; then you can use a normal GLib mainloop and process events as they happen, without any polling behavior.

http://library.gnome.org/devel/pygobject/stable/glib-functions.html#function-glib--io-add-watch

Answer 4

When reading from a file, your only choice is sleep (see the source code). If you read from a pipe, you can simply read since the read will block until there is data ready.

The reason for this is that the OS doesn't support the notion "wait for someone to write to a file". Only recently, some filesystems added an API where you can listen for changes made to a file but tail is too old to use this API and it's also not available everywhere.

Answer 5

IMO you should use sleep, it works on all platform and code will be simple

Otherwise you can use platform specific APIs which can tell you when file change e.g. on window use FindFirstChangeNotification on folder and watch for FILE_NOTIFY_CHANGE_LAST_WRITE events

On linux i think you can use i-notify

On Mac OSX use FSEvents

Answer 6

You can see here how to do a "tail -f" like using inotify:

This is an exemple[sic] to show how to use the inotify module, it could be very usefull unchanged though.

A Watcher instance let you define callbacks for any event that occur on any file or directory and subdirectories.

The inotify module is from Recipe 576375