Multiple patterns in one log

Question

So I wrote now several patterns for logs which are working. The thing is now, that I have these multiple logs, with multiple patterns, in one single file. How does logstash know what kind of pattern it has to use for which line in the log? ( I am using grok for my filtering ) And if you guys would be super kind, could you give me the link to the docs, because I weren't able to find anything regarding this :/

Answer 1

You could use multiple patterns for your grok filter,

grok {
  match => ["fieldname", "pattern1", "pattern2", ..., "patternN"]
}

and they will be applied in order but a) it's not the best option performance-wise and b) you probably want to treat different types of logs differently anyway, so I suggest you use conditionals based on the type or tags of a message:

if [type] == "syslog" {
  grok {
    match => ["message", "your syslog pattern"]
  }
}

Set the type in the input plugin.

The documentation for the currently released version of Logstash is at http://logstash.net/docs/1.4.2/. It probably doesn't address your question specifically but it can be inferred.

Answer 2

Write the most specific grok first and use this syntax:

grok {
    match => {
      "message" => [
      #Most specific grok:
        "%{TIMESTAMP_ISO8601:temp_date}%{SPACE}%{LOGLEVEL:log_level}%{UUID:user_id}",
      #Less specific:
        "%{TIMESTAMP_ISO8601:temp_date}%{SPACE}%{GREEDYDATA:log_message}"
     ]
  }
}