Rotating S3 Logging using log4j with Elastic Beanstalk

Question

I'm trying to transfer each log file to s3. There is an automatic script that picks up tail_catina.log and overwrites anything there.

I want each log file: tail_catalina.log1, tail_catalina.log2, tail_catalina.log3 etc. saved to my s3.

I want everything!!

Answer 1

I use a basic combination of logrotate, s3cmd, and cron to achieve this quite simply.

I've done a detailed writeup and explanation on my blog. It should work for anyone running an Apache server on a linux environment. I hope folks find it helpful as it took me a few hours to get the details hammered out.

The basic script is below, see the blog posting for a line by line breakdown:

# rotate the logs!
# common settings
compress
compresscmd /bin/gzip
compressoptions -9
compressext .gz

dateext
dateformat -%Y-%m-%d-%s

rotate 3
nomail
missingok
daily
size 5k
create 640 username username

/var/logs/www.runpartner.com/*.log {
sharedscripts
postrotate
sudo /usr/sbin/apache2ctl graceful

/usr/bin/s3cmd sync /var/logs/www.runpartner.com/*.gz s3://bucket-logs/www.runpartner.com/
endscript
}

Answer 2

Early Beanstalk AMIs were not rotating logs properly. You can fix it by using the latest AMI in your deployment. Go to EC2 Console, AMIs. Filter the list by choosing Amazon Images, "elasticbeanstalk" then sort by "Source" to see the latest AMIs.

Alternatively, you can edit the file /etc/logrotate.conf.elasticbeanstalk on the Beanstalk server to fix log rotation. The following config appends the timestamp after file names. It produces logs like tail_catalina.log-1322236861.gz, tail_catalina.log-1322240461.gz, etc.

/var/log/tomcat6/catalina.out /var/log/tomcat6/monitor_catalina.log /var/log/tomcat6/tail_catalina.log {
    size 1M
    missingok
    rotate 2
    compress
    notifempty
    copytruncate
    dateext
    dateformat -%s
    lastaction
        /bin/chown tomcat:elasticbeanstalk /var/log/tomcat6/*gz; /bin/chmod 664 /var/log/tomcat6/*gz
    endscript
}