Remove Field from event by pattern

Question

So I\'m using a standard ELK stack to analyse Apache access logs, which is working well, but I\'m looking to break out URL parameters as fields, using the KV filter, in orde

Answer 1

I know this is dated and has been answered, but for anyone looking into it as of 2017. There's a plugin named prune that allows you to trim based on difference criteria including patterns.

prune {
    blacklist_names => ["[0-9]+", "unknown_fields", "tags"]
}

Answer 2

If the set of fields that you are interested in is known and well-defined you could set target for the kv filter, move the interesting fields to the top level of the message with a mutate filter and delete the field with the nested key/value pairs. I think this is pretty much what you suggested at the end.

Alternatively you could use a ruby filter:

filter {
  ruby {
    code => "
      event.to_hash.keys.each { |k|
        if k.start_with?('rand')
          event.remove(k)
        end
      }
    "
  }
}