How properly rollover indices in elasticsearch

Question

I harvest logs with filebeat from all docker containers, sending them to logstash and from logstash are forwarded to elasticsearch. I would like to rollover my indices, that are automatically created, if they are too big or too old. There are few variants I tested.

Use Elasticsearch ILM. I created policy my_policy1 that should rollover when 5M are exceeded and template template_charizard:

{ "index_patterns" : ["printmessage_charizard*"], "settings" : { "number_of_shards" : 1, "number_of_replicas": 1, "index.lifecycle.name": "my_policy1", "index.lifecycle.rollover_alias": "charizard-actual" }, "aliases": { "charizard-actual": { "is_write_index": true } } }

This does not work, I have to create index with alias , in this case charizard-actual manualy. If I not, there will be error:

Rollover alias [charizard-actual] can point to multiple indices, found duplicated alias [[charizard-actual]] in index template [template_charizard]

There will be more container that will be consumed, so I cant manually create index every time when some new index should be created. I would expect that it will be create automatically.

UPDATE:

I tested this option (exactly by tutorial, with the same names), and it does not work. It just create new empty index, with increment number, but all logs went to the bootstrap index.
I can configure logstash ILM. This works fine, but I cant use variables in alias name, so I would have to create elasticsearch output for each index and make a lot of ifs.
I can configure filebeat ILM. Similar problem like with logstash. I can use variable in rollover alias, but it is hardcoded for all indices.

My expectation is, that I will configure some policies, some templates, and based on regexp of index name will be template applied to the index. All the options are basically usefull, everytime there must be manual change (edit logstash config) or request (create index manually first).

Questions:
What is the best solution for proper rollover?
Why everything must be hardcoded, it is very hard to maintain then.
According to me, curator would be the best solution, but then tehere is neccessitiy to maintain another application.
Could anybody explain me usage of all ILM options? Cant imagine to change configuration for every new index, even create manually index first or use hardcoded alias for all logs like in filebeat example. Maybe I am missing something.

alonana alonana · Accepted Answer · 2021-03-15T18:38:11

You can use ILM configuration by filebeat itself. There is no need in logstash as a middleman. Notice that filebeat automatically updates the alias in rollover, so you are not required to do any manual updates.

The filebeat configuration is:

setup.ilm:
  enabled: true
  policy_name: "filebeat"
  rollover_alias: "filebeat"
  pattern: "{now/d}-000001"
  policy_file: /etc/ilm.json
  overwrite: true

And you need to add the ilm.json file, for example:

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "1h"
          }
        }
      },
      "delete": {
        "min_age": "4h",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

See full description of this in the following post: http://runkiss.blogspot.com/2020/08/using-ilm-for-filebeat-indices-retention.html

How properly rollover indices in elasticsearch

1 Answers