SLF4J logging to file vs. DB vs. Solr

Question

I need some suggestions in terms of SLF4J logging.

Currently, we are using SLF4J logging (log4j binding) for a our Java web app, which uses the simple ConsoleAppender. Our next step is researching for places where we can save the logs.

Our app processes about 100,000 messages per day. Each message generates about 60 -100 lines of logs. Our goal is to be able to quickly search and find failed messages (using an messageId) and identify causes for the failure.

My question is: which of the following is a good place to store our logs:

• File(s)
• DB
• Solr

Thanks.

Answer 1

Consider switching away from log4j and using the logback implementation of the slf4j API Logback has an extensive list of appenders available.

I think perhaps your questions is more concerning making your logs searchable. The answer depends on what you're search for.

• For simple applications I just use a rolling file appender and grep this for the messages I'm interested in.
• More complicated applications will additionally log messages to the database.
• There is currently no Solr appender available for log4j and logback. It should however be easy to write using the solrj API
• For monitoring log messages there is a lilith which is a remote GUI for log messages. Don't know how well it scales, but it's certainly interesting for demos and simple monitoring.

Update

As suggested by Sebastien there is also a Graylog2 appender for logback. Now available in Maven Central

<dependency>
    <groupId>me.moocar</groupId>
    <artifactId>logback-gelf</artifactId>
    <version>0.9.6p2</version>
</dependency>

Of course this will depend on having a graylog2 server installed.

Answer 2

There is no facility in the servlet specification providing you with a file system location for putting logs.

Hence the most robust, long term solution is to simply use java.util.logging (with the slf4j binding) and let the web container handle the logs generated.

You have about 10 million log entries pr day. This mean that you need to be careful with resource usage. Database communication is much more expensive than file access. I would suggest you profile the approaches to see if you can get the performance you need to consider anything else but flat files backed up on a nightly basis.