Parsing apache log files

Question

I just started learning Python and would like to read an Apache log file and put parts of each line into different lists.

line from the file

1

Answer 1

Add this in httpd.conf to convert the apache logs to json.

LogFormat "{\"time\":\"%t\", \"remoteIP\" :\"%a\", \"host\": \"%V\", \"request_id\": \"%L\", \"request\":\"%U\", \"query\" : \"%q\", \"method\":\"%m\", \"status\":\"%>s\", \"userAgent\":\"%{User-agent}i\", \"referer\":\"%{Referer}i\" }" json_log

CustomLog /var/log/apache_access_log json_log
CustomLog "|/usr/bin/python -u apacheLogHandler.py" json_log

Now you see you access_logs in json format. Use the below python code to parse the json logs that are constantly getting updated.

apacheLogHandler.py

import time
f = open('apache_access_log.log', 'r')
for line in f: # read all lines already in the file
  print line.strip()

# keep waiting forever for more lines.
while True:
  line = f.readline() # just read more
  if line: # if you got something...
    print 'got data:', line.strip()
  time.sleep(1)