How to handle non-matching Logstash grok filters

Question

I am wondering what the best approach to take with my Logstash Grok filters. I have some filters that are for specific log entries, and won\'t apply to all entries. The on

Answer 1

You can also do this

remove_tag => [ "_grokparsefailure" ]

whenever you have a match.

Answer 2

When possible, I'd go with a conditional wrapper just like the one you're using. Feel free to post that as an answer!

If your application produces only a few different line formats, you can use multiple match patterns with the grok filter. By default, the filter will process up to the first successful match:

grok {
    patterns_dir => "./patterns"
    match => {
        "message" => [ 
              "%{BASE_PATTERN} %{EXTRA_PATTERN}",
              "%{BASE_PATTERN}",
              "%{SOME_OTHER_PATTERN}"
        ]
    }
}

If your logic is less straightforward (maybe you need to check the same condition more than once), the grep filter can be useful to add a tag. Something like this:

grep {
    drop => false #grep normally drops non-matching events
    match => ["message", "/took\s\d+/"]
    add_tag => "has_traceback"
}


...

if "has_traceback" in [tags] {
    ...
}

Answer 3

This is the most efficient way of doing this. Ignore the filter

filter {

        grok {
            match => [ "message", "something"]
    }

    if "_grokparsefailure" in [tags] {
            drop { }
        }
}

Answer 4

You can also add tag_on_failure => [] to your grok stanza like so:

grok {
    match => ["context", "\"tags\":\[%{DATA:apptags}\]"]
    tag_on_failure => [ ]
}

grok will still fail, but will do so without adding to the tags array.