How to make Logstash multiline filter merge lines based on some dynamic field value?

Question

I am new to logstash and desparate to setup ELK for one of the usecase. I have found this question relevent to mine Why won't Logstash multiline merge lines based on grok

Answer 1

You need to use a multiline filter with stream_identity set. The documentation here isn't clear on what it's used for, but your basic strategy would be something like this:

if (!"multiline" in [tags]) {
  grok { // parse out your identity field }
  multiline { 
    stream_identity => "%{id}"
    pattern => "." // match anything because we're gathering by id field
    what => "previous"
    periodic_flush => true
    max_age => 5 // however many seconds it takes to get all of your lines together
    add_tags => ["multiline" ]
  }
} else {
  // process multiline event that's been flushed
}

I haven't tried anything like this since 1.5 came out, but the docs say it should work (in 1.4.2 and prior, the flushing mechanism didn't work, so you could lose events).