Logstash: Renaming nested fields based on some condition

问题

I am trying to rename the nested fields from Elasticsearch while migrating to Amazonelasticsearch

In the document, I want to change the

1.If the value field has JSON type. Change the value field to value-keyword and remove "value-whitespace" and "value-standard" if present

2.If the value field has a size of more than 15. Change the value field to value-standard

 "_source": {
          "applicationid" : "appid",
          "interactionId": "716bf006-7280-44ea-a52f-c79da36af1c5",
          "interactionInfo": [
            {
              "value": """{"edited":false}""",
              "value-standard": """{"edited":false}""",
              "value-whitespace" :  """{"edited":false}"""
              "title": "msgMeta"
            },
            {
              "title": "msg",
              "value": "hello testing",
            },
            {
              "title": "testing",
              "value": "I have a text that can be done and changed only the size exist more than 20 so we applied value-standard ",
            }
          ],
          "uniqueIdentifier": "a21ed89c-b634-4c7f-ca2c-8be6f31ae7b3",
        }
      }

the end result should be

 "_source": {
          "applicationid" : "appid",
          "interactionId": "716bf006-7280-44ea-a52f-c79da36af1c5",
          "interactionInfo": [
            {
              "value-keyword": """{"edited":false}""",
              "title": "msgMeta"
            },
            {
              "title": "msg",
              "value": "hello testing",
            },
            {
              "title": "testing",
              "value-standard": "I have a text that can be done and changed only the size exist more than 20 and so we applied value-standard  ",
            }
          ],
          "uniqueIdentifier": "a21ed89c-b634-4c7f-ca2c-8be6f31ae7b3",
        }
      }

回答1:

For 2), you can do it like this:

filter {
    if [_source][interactionInfo][2][value] =~ /.{15,15}/ {

        mutate {
            rename => ["[_source][interactionInfo][2][value]","[_source][interactionInfo][2][value-standard]"]
        }
    }
}

The regex .{15,15} matches any string 15 characters long. If the field is shorter than 15 characters long, the regex doesn't match and the mutate#rename isn't applied.

For 1), one possible solution would be trying to parse the field with the json filter and if there's no _jsonparsefailure tag, rename the field.

回答2:

Founded the solution for this one. I have used a ruby filter in Logstash to check each and every document as well as nested document Here is the ruby code

require 'json'

def register(param)
end

def filter(event)
  infoarray = event.get("interactionInfo")
  infoarray.each {  |x|
      if x.include?"value"
         value = x["value"]
         if value.length > 15
           apply_only_keyword(x)
         end
       end
      if x.include?"value"
        value = x["value"]
         if validate_json(value)
           apply_only_keyword(x)
         end
       end
  }
event.set("interactionInfo",infoarray)
return [event]
end


def validate_json(value)
  if value.nil?
    return false
  end
  JSON.parse(value)
  return true
rescue JSON::ParserError => e
  return false
end

def apply_only_keyword(x)
  x["value-keyword"] = x["value"]
  x.delete("value")
  if x.include?"value-standard"
    x.delete("value-standard")
  end
  if x.include?"value-whitespace"
    x.delete("value-whitespace")
  end
end

来源：https://stackoverflow.com/questions/58538903/logstash-renaming-nested-fields-based-on-some-condition

标签

logstash