Use grok to add the log filename as a field in logstash

问题

I'm using Grok & Logstash to send access logs from Nginx to Elastic search. I'm giving Logstash all my access logs (with a wildcard, works well) and I would like to get the filename (some part of it, to be exact) and use it as a field.

My config is as follows :

input {
  file {
    path => "/var/log/nginx/*.access.log"
    type => "nginx_access"
  }
}

filter {
  if [type] == "nginx_access" {
    grok { 
      match => { "message" => "%{COMBINEDAPACHELOG}" }
      match => { "path" => "%{GREEDYDATA}/%{GREEDYDATA:app}.access.log" }
      add_field => { "app" => "%{app}" }
    }
  }
}
output{
   # whatever
}

But it doesn't seem to work : the app field is added, but has a value of %{app} (not replaced).

I tried different things but to no avail. I may be missing something ... Any ideas ?

Thanks a lot

回答1:

Ok, found it. grok breaks on match by default. So the first match being good, it skips the second one.

I solved it like that :

filter {
  if [type] == "nginx_access" {
    grok { 
      match => { "message" => "%{COMBINEDAPACHELOG}" }
      match => { "path" => "%{GREEDYDATA}/%{GREEDYDATA:app}.access.log" }
      break_on_match => false
    }
  }
}

回答2:

I found it more desirable to use 2 grok blocks if there will be unmatching lines in the log files.

filter {
  if [type] == "nginx_access" {
    grok { 
      match => { "path" => "%{GREEDYDATA}/%{GREEDYDATA:app}.access.log" }
    }
    grok { 
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
  }
}

来源：https://stackoverflow.com/questions/23780000/use-grok-to-add-the-log-filename-as-a-field-in-logstash