How to remove trailing newline from message field

Question

I am shipping Glassfish 4 logfiles with Logstash to an ElasticSearch sink. How can I remove with Logstash the trailing newline from a message field?

My event looks like

Answer 1

You have to use the multiline filter with the correct pattern, to tell logstash, that every line with precending whitespace belongs to the line before. Add this lines to your conf file.

filter{
  ...
  multiline {
    type => "gflogs"
    pattern => "\[\#\|\d{4}"
    negate => true
    what => "previous"
  }
  ...
}

You can also include grok plugin to handle timestamp and filter irregular lines from beeing indexed.

See complete stack with single logstash instance on same machine

input {
  stdin {
    type => "stdin-type"
  }

  file {
    path => "/path/to/glassfish/logs/*.log"
    type => "gflogs"
  }
}

filter{
  multiline {
    type => "gflogs"
    pattern => "\[\#\|\d{4}"
    negate => true
    what => "previous"
  }

  grok {
    type => "gflogs"
    pattern => "(?m)\[\#\|%{TIMESTAMP_ISO8601:timestamp}\|%{LOGLEVEL:loglevel}\|%{DATA:server_version}\|%{JAVACLASS:category}\|%{DATA:kv}\|%{DATA:message}\|\#\]"
    named_captures_only => true
    singles => true
  }

  date {
    type => "gflogs"
    match => [ "timestamp", "ISO8601" ]
  }

  kv {
    type => "gflogs"
    exclude_tags => "_grokparsefailure"
    source => "kv"
    field_split => ";"
    value_split => "="
  }
}

output {
  stdout { codec => rubydebug }
  elasticsearch { embedded => true }
}

This worked for me. Pleas look also this post on logstash-usergroup. I can also advice the great and up to date logstash book. Its also a good way to support the work of the logstash author.

Hope to see you on any JUG-Berlin Event!