How to filter logs easily with awk?

Question

Suppose I have a log file mylog like this:

[01/Oct/2015:16:12:56 +0200] error number 1
[01/Oct/2015:17:12:56 +0200] error number 2
[01/Oct/2015:         


        

Answer 1

Use ISO 8601 time format!

However, this seems to be quite a bit of work for something that should be more straight forward.

Yes, this should be straightforward, and the reason why it is not, is because the logs do not use ISO 8601. Application logs should use ISO format and UTC to display times, other settings should be considered broken and fixed.

Your request should be split in two parts. The first part canonise the logs, converting dates to the ISO format, the second performs a research:

awk '
match($0, /([0-9]+)\/([A-Z][a-z]{2})\/([0-9]{4}):([0-9]{1,2}):([0-9]{1,2}):([0-9]{1,2}) ([+-][0-9]{4})/, a) {
  day=a[1]
  month=a[2];
  year=a[3]
  hour=a[4]
  min=a[5]
  sec=a[6]
  utc=a[7];
  month=sprintf("%02d", (match("JanFebMarAprMayJunJulAugSepOctNovDec",month)+2)/3);
  myisodate=sprintf("%4d-%2d-%2dT%2d:%2d:%2d%6s", year,month,day,hour,min,sec,utc);
 $1 = myisodate
 print
}' mylog

The nice thing about ISO 8601 dates – besides them being a standard – is that the chronological order coincide with lexicographic order, therefore, you can use the /…/,/…/ operator to extract the dates you are interested in. For instance to find what happened between 1 Oct 2015 18:00 +0200 and 1 Nov 2015 01:00 +0200, append the following filter to the previous, standardising filter:

awk '/2015-10-01:18:00:00+0200/,/2015-11-01:01:00:00+0200/'