Reading log files as they're updated in Go

Question

I\'m trying to parse some log files as they\'re being written in Go but I\'m not sure how I would accomplish this without rereading the file again and again while checking f

Answer 1

A simple example:

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
    "time"
)

func tail(filename string, out io.Writer) {
    f, err := os.Open(filename)
    if err != nil {
        panic(err)
    }
    defer f.Close()
    r := bufio.NewReader(f)
    info, err := f.Stat()
    if err != nil {
        panic(err)
    }
    oldSize := info.Size()
    for {
        for line, prefix, err := r.ReadLine(); err != io.EOF; line, prefix, err = r.ReadLine() {
            if prefix {
                fmt.Fprint(out, string(line))
            } else {
                fmt.Fprintln(out, string(line))
            }
        }
        pos, err := f.Seek(0, io.SeekCurrent)
        if err != nil {
            panic(err)
        }
        for {
            time.Sleep(time.Second)
            newinfo, err := f.Stat()
            if err != nil {
                panic(err)
            }
            newSize := newinfo.Size()
            if newSize != oldSize {
                if newSize < oldSize {
                    f.Seek(0, 0)
                } else {
                    f.Seek(pos, io.SeekStart)
                }
                r = bufio.NewReader(f)
                oldSize = newSize
                break
            }
        }
    }
}

func main() {
    tail("x.txt", os.Stdout)
}

Answer 2

I have written a Go package -- github.com/hpcloud/tail -- to do exactly this.

t, err := tail.TailFile("/var/log/nginx.log", tail.Config{Follow: true})
for line := range t.Lines {
    fmt.Println(line.Text)
}

...

Quoting kostix's answer:

in real life files might be truncated, replaced or renamed (because that's what tools like logrotate are supposed to do).

If a file gets truncated, it will automatically be re-opened. To support re-opening renamed files (due to logrotate, etc.), you can set Config.ReOpen, viz.:

t, err := tail.TailFile("/var/log/nginx.log", tail.Config{
    Follow: true,
    ReOpen: true})
for line := range t.Lines {
    fmt.Println(line.Text)
}

Config.ReOpen is analogous to tail -F (capital F):

 -F      The -F option implies the -f option, but tail will also check to see if the file being followed has been
         renamed or rotated.  The file is closed and reopened when tail detects that the filename being read from
         has a new inode number.  The -F option is ignored if reading from standard input rather than a file.

Answer 3

There are many ways to do this. In modern POSIX based Operating Systems, one can use the inotify interface to do this.

One can use this package: https://github.com/fsnotify/fsnotify

Sample code:

watcher, err := fsnotify.NewWatcher()
if err != nil {
    log.Fatal(err)
}

done := make(chan bool)

err = watcher.Add(fileName)
if err != nil {
    log.Fatal(err)
}
for {
    select {
    case event := <-watcher.Events:
        if event.Op&fsnotify.Write == fsnotify.Write {
            log.Println("modified file:", event.Name)

        }
}

Hope this helps!

Answer 4

You have to either watch the file for changes (using an OS-specific subsystem to accomplish this) or poll it periodically to see whether its modification time (and size) changed. In either case, after reading another chunk of data you remember the file offset and restore it before reading another chunk after detecting the change.

But note that this seems to be easy only on paper: in real life files might be truncated, replaced or renamed (because that's what tools like logrotate are supposed to do).

See this question for more discussion of this problem.

Answer 5

I'm also interested in doing this, but haven't (yet) had the time to tackle it. One approach that occurred to me is to let "tail" do the heavy lifting. It would likely make your tool platform-specific, but that may be ok. The basic idea would be to use Cmd from the "os/exec" package to follow the file. You could fork a process that was the equivalent of "tail --retry --follow=name prog.log", and then listen to it's Stdout using the Stdout reader on the the Cmd object.

Sorry I know it's just a sketch, but maybe it's helpful.