Logging from multiple apps/processes to a single log file

Question

Our app servers (weblogic) all use log4j to log to the same file on a network share. On top of this we have all web apps in a managed server logging errors to a common erro

Answer 1

Best case I would imagine you have a potential performance problem with sync access to the log file resource. Worst case would be some of the scenarios you mention.

I would ask two Qs: a) what was the desired goal with this config and b) can you find a better technical solution to achieve the goal.

My guess is that you have a sysadmin who wants to get a "single log file for the system." The file is thrown on the network share to make it easy to get to. The better answer for the goal might be multiple files, local to each system and something like http://www.splunk.com/ for a nice monitor.

Answer 2

This looks like a really bad idea (corrupt logs, uncertainty of the source of a given log entry are two reasons that come to mind). If you are using Log4j in Weblogic like this, I suggest doing it by-the-book. This will allow you to use one file for the whole app server without any issues.

The suggestion of syncronizing the log writing makes no sense to me, as you would be basically blocking all applications in the app server when they write a log. If logging is frequent, that will slow everything down significantly.

As for multiple app servers, you need to use something other than file based logging if you want them all consolidated. There are a few ways to do this, one is to log to different files and have a different process combine them, but the better option is probably to use a network based logging repository, using Log4j's SocketAppender or some other method (nathan mentions SyslogAppender which is great if you want a Syslog) to ensure that the file access does not get corrupted.

Answer 3

We use org.apache.log4j.net.SyslogAppender to log to a single machine using syslog, and it has worked well for us. I would recommend looking into that as an alternative.

Answer 4

In prudent mode logback will safely handle multiple JVMs possibly on different hosts writing to the same network-shared file. It can even deal with temporary network failures. Performance should be quite acceptable for few nodes, say 4 or less. For 5 or more nodes all logging heavily you may notice a performance hit.

Answer 5

We have a requirement where we need to produce a single file from all the managed server running the same application . we developed a java logging server which opens ups a port and listen for log event. we used the log4j socket appender to write log event to the same port and created a single file.

Answer 6

If at all possible use a different file for each instance. This will give the best result with the least effort.

The logback alternative to log4j has a prudent mode to its log writer which explicitly jump through hoops to ensure that new things are written at the end of file. I don't think that will work on network shares then.

If you MUST have a central logging location, then consider setting up a server accepting log events and writing them to the appropriate files. This will ensure it is only a single process actually accessing the file system, and will let the JVM help all it can in terms of synchronization etc.