12Factor App: Capturing stdout/stderr logs with Fluentd

Question

By reading the following post from 12factor I have come up with a question I'd like to check how you guys handle this.

Basically, an app should write directly to stdout/stderr. Is there anyway to redirect these streams directly to fluentd (not bound to rsyslog/syslog)? As I become more aware of fluentd, I believe it would be a great tool for log aggregation from multiple apps/platforms.

The main reasoning for this is, if the app is cross-platform, rsyslog/syslog may not be available, and as I understand, using logging frameworks (which need the required configuration for them to work) would be a violation of the 12factor.

Thanks!

Answer 1

You need to configure your process manager to use fluentd.

"Twelve-factor app processes should [...] rely on the operating system’s process manager (such as Upstart, a distributed process manager on a cloud platform, or a tool like Foreman in development) to manage output streams [...]."

Basically, the idea is that log redirection is a concern of the process manager. Upstart, for example, usually relies on logger, which has an option (-u) to write to a Unix Domain Socket. In turn, you can configure fluentd to use that same socket as an input stream.

Fluentd supports a lot of input streams (they call them data sources), which should provide a solution for just about any environment & process manager you might be using (which we need to know in order to provide a more complete solution).

Answer 2

This is still an experimental program, but you can take a look at stdout-hook.

• https://github.com/treasure-data/stdout-hook

This program reads STDOUT output from child process, and post the records to Fluentd.