logstash grok filter for logs with arbitrary attribute-value pairs

Question

(This is related to my other question logstash grok filter for custom logs )

I have a logfile whose lines look something like:

14:46:16.603 [http-nio         


        

Answer 1

You can change your grok pattern like this to have all the key value pairs in one field (kvpairs):

%{TIME:timestamp} %{NOTSPACE:http} %{WORD:loglevel}%{SPACE}%{WORD:logtype} - %{GREEDYDATA:kvpairs}

Afterwards you can use the kv filter to parse the key value pairs.

kv {
    source => "kvpairs"
    remove_field => [ "kvpairs" ] # Delete the field afterwards
}

Unfortunately, you have some simple values inside your kv pairs (e.g. CREATE_JOB). You could parse them with grok and use one kv filter for the values before and another kv filter for the values after those simple values.