Elastic search load csv data with context

问题

I have 3m records. Headers are value, type, other_fields..

Here I need to load the data as in this

I need to specify type as context for that value in the record. Is there any way to do this with log stash? or any other options?

val,val_type,id
Sunnyvale it labs, seller, 10223667

回答1:

For this, I'd use the new CSV ingest processor

First create the ingest pipeline to parse your CSV data

PUT _ingest/pipeline/csv-parser
{
  "processors": [
    {
      "csv": {
        "field": "message",
        "target_fields": [
          "val",
          "val_type",
          "id"
        ]
      }
    },
    {
      "script": {
        "source": """
          def val = ctx.val;
          ctx.val = [
            'input': val,
            'contexts': [
              'type': [ctx.val_type]
            ]
          ]
          """
      }
    },
    {
      "remove": {
        "field": "message"
      }
    }
  ]
}

Then, you can index your documents as follow:

PUT index/_doc/1?pipeline=csv-parser
{
  "message": "Sunnyvale it labs,seller,10223667"
}

After ingestion, the document will look like this:

{
  "val_type": "seller",
  "id": "10223667",
  "val": {
    "input": "Sunnyvale it labs",
    "contexts": {
      "type": [
        "seller"
      ]
    }
  }
}

UPDATE: Logstash solution

Using Logstash, it's also feasible. The configuration file would look something like this:

input {
    file {
        path => "/path/to/your/file.csv"
        sincedb_path => "/dev/null"
        start_position => "beginning"
    }
}
filter {
    csv {
        skip_header => true
        separator => ","
        columns => ["val", "val_type", "id"]
    }
    mutate {
        rename => { "val" => "value" }
        add_field => { 
            "[val][input]" => "%{value}" 
            "[val][contexts][type]" => "%{val_type}" 
        }
        remove_field => [ "value" ]
    }
}
output {
    elasticsearch {
        hosts => "http://localhost:9200"
        index => "your-index"
    }    
}

来源：https://stackoverflow.com/questions/62210701/elastic-search-load-csv-data-with-context

标签

ElasticSearch

logstash