Log the estimate of batch metrics memory consumption#18916

Merged

andsel merged 19 commits intoelastic:mainfrom

andsel:feature/estimate_batch_metrics_memory_consumption_and_log

Apr 15, 2026

Commits on Apr 13, 2026

Exposed HdrHistogram memory occupation and aggregated at pipeline level
andsel
committed
[Test] added test coverage of the computation when batch metrics is enabled and when don't
andsel
committed
Added logging of estimated space for each pipeline
andsel
committed
Minor, expanded * import
andsel
committed
Fix logger format string
andsel
committed
Commented lifetime part to be enabled after #18911
andsel
committed
[Test] Spread the disabling of batch sampling mode in Pipeline's Actions classes. This is needed to avoid start collecting such data on not yet fully initialized pipelines.
andsel
committed
[Test] Spread the disabling of batch sampling mode for pipeline and pipeline report tests
andsel
committed
Disable batch structure monitoring for internal '.monitoring-logstash' pipeline
andsel
committed
Adds a check on the collector being enabled before logging batch size metrics. This simplifies the configuration. Instaed of setting also 'pipeline.batch.metrics.sampling_mode' to disabled when 'me…
andsel
committed
[Minor] cleaned commented test code
andsel
committed
[Test] Fixed tests after addition of lifecycle policy
andsel
committed
Changed count of lifetime policy datapoints to count also for the staged capture histogram instance
andsel
committed
Fixed datapoints sumup for policies considering also the staging snapshots
andsel
committed
Fixed naming, avoid 'occupation' and use 'memory consumption'
andsel
committed
Fixed typo
andsel
committed
Better wording to comments

andsel
and
estolfo
committed
[Docs] Updates Logstash tuning guide to describe that more memory is consumed when pipeline.batch.metrics.sampling_mode is enabled
andsel
committed

Commits on Apr 14, 2026

[Doc] reworded the note
andsel
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Log the estimate of batch metrics memory consumption#18916

Log the estimate of batch metrics memory consumption#18916
andsel merged 19 commits intoelastic:mainfrom
andsel:feature/estimate_batch_metrics_memory_consumption_and_log

Commits on Apr 13, 2026

Exposed HdrHistogram memory occupation and aggregated at pipeline level

[Test] added test coverage of the computation when batch metrics is enabled and when don't

Added logging of estimated space for each pipeline

Minor, expanded * import

Fix logger format string

Commented lifetime part to be enabled after #18911

[Test] Spread the disabling of batch sampling mode in Pipeline's Actions classes. This is needed to avoid start collecting such data on not yet fully initialized pipelines.

[Test] Spread the disabling of batch sampling mode for pipeline and pipeline report tests

Disable batch structure monitoring for internal '.monitoring-logstash' pipeline

Adds a check on the collector being enabled before logging batch size metrics. This simplifies the configuration. Instaed of setting also 'pipeline.batch.metrics.sampling_mode' to disabled when 'me…

[Minor] cleaned commented test code

[Test] Fixed tests after addition of lifecycle policy

Changed count of lifetime policy datapoints to count also for the staged capture histogram instance

Fixed datapoints sumup for policies considering also the staging snapshots

Fixed naming, avoid 'occupation' and use 'memory consumption'

Fixed typo

Better wording to comments

[Docs] Updates Logstash tuning guide to describe that more memory is consumed when pipeline.batch.metrics.sampling_mode is enabled

Commits on Apr 14, 2026

[Doc] reworded the note