Removing old indices in elasticsearch

Question

I have the many of my logs indexed in logstash-Year-Week format. That is if i want to delete indices older than a few weeks, how can I achieve that in elasticsearch. Is ther

Answer 1

As of elasticsearch 6.6, Index Lifecycle Management comes included with basic (free) versions elasticsearch, and accomplishes what Curator used to, but in a more graceful way.

The steps below are reproduced without permission from Martin Ehrnhöfer's excellent and concise blog post.

Assumptions (heads up to the copy-pasters):

• Your elasticsearch server is accessible at http://elasticsearch:9200
• You want your indices to be purged after thirty days (30d)
• Your policy name will be created as cleanup_policy
• Your filebeat index names begin with filebeat-
• Your logstash index names begin with logstash-

1. Create a policy that deletes indices after one month

curl -X PUT "http://elasticsearch:9200/_ilm/policy/cleanup_policy?pretty" \
     -H 'Content-Type: application/json' \
     -d '{
      "policy": {                       
        "phases": {
          "hot": {                      
            "actions": {}
          },
          "delete": {
            "min_age": "30d",           
            "actions": { "delete": {} }
          }
        }
      }
    }'

2. Apply this policy to all existing filebeat and logstash indices

curl -X PUT "http://elasticsearch:9200/logstash-*/_settings?pretty" \
     -H 'Content-Type: application/json' \
     -d '{ "lifecycle.name": "cleanup_policy" }'
curl -X PUT "http://elasticsearch:9200/filebeat-*/_settings?pretty" \
     -H 'Content-Type: application/json' \
     -d '{ "lifecycle.name": "cleanup_policy" }'

3. Create a template to apply this policy to new filebeat and logstash indices

curl -X PUT "http://elasticsearch:9200/_template/logging_policy_template?pretty" \
     -H 'Content-Type: application/json' \
     -d '{
      "index_patterns": ["filebeat-*", "logstash-*"],                 
      "settings": { "index.lifecycle.name": "cleanup_policy" }
    }'