feat: add opt-in dynamic ILM resolution for sprintf-based alias and policy

[jithsungh]

Overview

This PR introduces opt-in dynamic ILM resolution when ilm_rollover_alias or ilm_policy contains sprintf patterns.

In this mode, alias and policy are resolved per event instead of at startup.

---

Motivation

In multi-service environments, users often need ILM configurations (alias + policy) to vary per event (e.g. based on service or container name).

Currently, this requires:

• manual routing logic
• multiple pipeline conditions
• static ILM configuration per output

This change allows ILM resolution to be driven directly from event fields.

---

Behavior

• Activated only when sprintf is used in:

  • ilm_rollover_alias
  • ilm_policy

• Per-event:

  • resolve alias and policy
  • validate resolved values
  • verify policy and alias exist in Elasticsearch

• Startup ILM bootstrap is skipped in dynamic mode

---

Constraints (Intentional)

• No automatic creation of:

  • ILM policies
  • templates
  • rollover aliases

• All ILM resources must be pre-created

• Missing resources result in EventMappingError

---

Backward Compatibility

• No change to existing behavior
• Static ILM configurations remain unaffected
• Fully opt-in via sprintf usage

---

Performance Considerations

• Validation is cached per resolved alias/policy combination
• Avoids repeated Elasticsearch API calls for identical values

---

Testing

• Added unit tests covering:

  • alias/policy resolution
  • validation behavior
  • caching behavior
  • failure scenarios
  • startup behavior bypass

---

Additional Context

This approach has been used in production environments with strict pre-provisioned ILM setups to support multi-service indexing patterns.
Faced the linear search bottleneck at logstash_output_elasticsearch config containing hundreds of if-else blocks each for one microservice manually routing event to their alias and policies.

visit additional documentation
If there is interest, additional features (e.g. controlled auto-provisioning) can be proposed separately.

[cla-checker-service]

💚 CLA has been signed

[Copilot]

Pull request overview

This PR adds an opt-in “dynamic ILM” mode for the Elasticsearch output plugin that resolves ilm_rollover_alias and ilm_policy per-event when either setting contains sprintf patterns, and skips startup ILM bootstrap in that mode.

Changes:

• Skip setup_ilm bootstrap when ILM settings contain sprintf patterns, and add per-event alias/policy resolution + existence validation with caching.
• Route event _index to the dynamically-resolved rollover alias when dynamic ILM mode is active.
• Add unit tests for dynamic resolution, caching behavior, and failure scenarios.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
lib/logstash/outputs/elasticsearch/ilm.rb	Introduces dynamic ILM detection, per-event resolution/validation, and skips startup ILM bootstrap in dynamic mode.
lib/logstash/outputs/elasticsearch.rb	Hooks dynamic ILM validation into per-event mapping and uses resolved rollover alias as the target index.
spec/unit/outputs/elasticsearch_spec.rb	Adds unit tests covering the new dynamic ILM behavior and startup bootstrap bypass.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.