Index Cleanup and Retention Elastic Cloud

Question

I am having a hard time finding a clear cut answer: What is the default retention time before deletion for an index in Elastic Cloud (ELK Stack) and can it be easily modified? We have recently migrated from a on-prem ELK stack to a cloud solution. Previously we did this with Curator.

Last post I found was from 2017 and said that this was not supported in the Cloud: https://discuss.elastic.co/t/configure-retention-period-for-different-index/106491 Has this changed?

Wobbley Wobbley · Accepted Answer · 2019-11-14T07:33:28

I solved this by creating rollovers through the use of Index Lifecycle Managment (ILM), and Index Templates. Most information can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html

However the gist of it is:

Create a ILM

Create an Index Template with a rollover alias:

PUT _template/example-template-name
{
  "index_patterns": [
    "example-index-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "name-of-lifecycle-created-earlier",
        "rollover_alias": "example-index-name"
      }
    }
  }
}

Then create the first index that has the rollover alias alias. Personally I use the rollover date system, with URL encoding:

PUT /%3Cexample-index-name-%7Bnow%2Fd%7D-1%3E
{
  "aliases": {
    "example-index-name": {
      "is_write_index" : true
    }
  }
}

After this simply write to the alias, not the index itself. It will automatically assign the correct index and rollover when the ILM limits are met.

Index Cleanup and Retention Elastic Cloud

1 Answers