Using Elasticsearch ILM without Bootstraping the First Index

Question

I am trying to set an ILM policy for an Elasticsearch cluster of 3 nodes (hot, warm and cold) whose logs are coming from Logstash. I need the indices to be rolled over after reaching to a specific size. Also title of each log's data source should be referred in the name of indices (e.g. event-nxlog-2022.01.01-000001). The problem is that since plenty of data sources exist in the project, for each data source, a lot of indices should be bootstrapped as the first index and many logstash config files should be modified.

With regard to the provided description, I have 2 questions:

1- Is it neccessary for ILM policy to be referred in the Logstash output plugin using ilm_enabled, ilm_policy, ilm_rollover_alias and ilm_pattern settings? Is there any way to use only index name instead and let elasticsearch to manage ILM itself?

2- Should the first indices of each data source bootstrapped? Is there any way to create first indices and to roll them over automattically? or any way to bootstrap all the indices with a single query using index patterns?

Answer 1

If all indices can have the same ILM policy, it is possible to replace the default logstash policy like below:

  logstash:
    image: logstash:7.17.7
    environment:
      - STDOUT=true
    volumes:
      - ./logstash/logstash.yaml:/etc/logstash.yaml:ro
      - ./logstash/ilm-policy.json:/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/default-ilm-policy.json:ro
    command: 'logstash -f /etc/logstash.yaml'

This will work if the elastic data is created from the scratch, like after a cluster data bootstrap. I have opened a feature request for the logstash elastic plugin to add an output option that would allow to configure a custom ILM policy file.