[vahle-at-psu]

Summary

Fixes #4108. Async instruments no longer export attribute sets absent from the current reader callback. Cumulative state is retained so values remain correct when a tracked observation reappears.

Root Cause

TemporalMetricStorage::buildMetrics() carries cumulative state across collections. That state is needed for cumulative aggregation, but stale async attribute sets must not be included in every export.

Async observations are also reader-scoped: observations from one reader's callback must not be queued as observations for other readers.

Changes

• Preserve cumulative state while filtering exports by the current reader's callback observations.
• Keep async observations in the collecting reader's queue for both delta and cumulative temporality.
• Preserve measurements routed to the cardinality overflow point.
• Bound per-reader monotonic counter baselines to the configured cardinality limit. Baselines are retained without eviction; after the limit is reached, new source baselines are not admitted and follow the existing cardinality-limit behavior.
• Cover ObservableCounter, ObservableGauge, and ObservableUpDownCounter behavior.

Behavior

Scenario	Result
Async attribute absent from the current callback	Its point is omitted from that reader's export
Tracked async cumulative attribute reappears	Its cumulative baseline is retained
Async delta observation from another reader's callback	It is not attributed to the collecting reader
Current observation is routed to overflow	Its contribution is exported
Sync cumulative collection	Existing carry-forward behavior is unchanged

The baseline cap does not evict previously tracked series. Behavior after the cap is reached follows the existing cardinality-limit semantics; correctness for additional source series beyond the limit is not expanded by this PR.

Tests

The following parameterized storage tests run with both delta and cumulative temporality:

• StaleAttributeSetDropped
• AttributeReappearanceAfterGap
• NonMonotonicInstrumentsDropStaleObservations
• StaleAttributeSetDroppedMultiCollector
• MultiCollectorLaggingCollector
• MultiCollector
• CardinalityLimitCapsCounterBaselines

Additional tests:

• WritableMetricStorageCardinalityLimit.AsyncCumulativeCurrentRoundTakesPrecedenceForLaggingReader
• WritableMetricStorageCardinalityLimit.AsyncCumulativeOverflowIsExportedForLaggingReader
• WritableMetricStorageCardinalityLimit.AsyncCumulativeCurrentRoundTakesPrecedenceOverStaleSeries
• CardinalityLimit.SyncCumulativeKeepsExistingSeriesBeforeOverflow
• MeterTest.AsyncCumulativeObservationsAreReaderScoped

Validated with the async metric storage, cardinality-limit, and SDK reader-scoping tests.

---

Co-authored-by: pranitaurlam 227409059+pranitaurlam@users.noreply.github.com

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: pranitaurlam / name: pranitaurlam (e9cc6f3)
• ✅ login: vahle-at-psu / name: Patrick Summerer (e9cc6f3)

[codecov]

Codecov Report

❌ Patch coverage is 95.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.82%. Comparing base (adebdf0) to head (5781472).
⚠️ Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
...dk/include/opentelemetry/sdk/metrics/instruments.h	75.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4484      +/-   ##
==========================================
+ Coverage   86.79%   86.82%   +0.03%     
==========================================
  Files         525      525              
  Lines       20729    20753      +24     
==========================================
+ Hits        17989    18016      +27     
+ Misses       2740     2737       -3     

Files with missing lines	Coverage Δ
sdk/src/metrics/state/temporal_metric_storage.cc	100.00% <100.00%> (ø)
...dk/include/opentelemetry/sdk/metrics/instruments.h	75.56% <75.00%> (-0.05%)	⬇️

... and 12 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[mateenali66]

Built this on 731099b, the reported case is fixed. One problem with suppressing at export time.

merged_metrics is both the running cumulative state and the exported set (temporal_metric_storage.cc:179-190), so the new else if (!is_async_) at :163 does not just omit the point, it drops the baseline. On reappearance Record supplies delta = new - last_seen with nothing left to merge onto, so the cumulative point is the increment. A=10, absent, A=30, one collector:

cycle	main	this PR	expected
A=10	10	10	10
absent	10 (stale, #4108)	nothing	nothing
A=30	30	20	30

AttributeReappearanceAfterGapDeltaTemporality misses it because one collector under delta returns on the fast path at :67, before the changed branch. Happy to post the cumulative version of that test.

Would you rather keep the carry-forward in last_reported_metrics_ and filter when building the MetricData, so the baseline survives the gap?

[vahle-at-psu]

Built this on 731099b, the reported case is fixed. One problem with suppressing at export time.

merged_metrics is both the running cumulative state and the exported set (temporal_metric_storage.cc:179-190), so the new else if (!is_async_) at :163 does not just omit the point, it drops the baseline. On reappearance Record supplies delta = new - last_seen with nothing left to merge onto, so the cumulative point is the increment. A=10, absent, A=30, one collector:
cycle main this PR expected
A=10 10 10 10
absent 10 (stale, #4108) nothing nothing
A=30 30 20 30

AttributeReappearanceAfterGapDeltaTemporality misses it because one collector under delta returns on the fast path at :67, before the changed branch. Happy to post the cumulative version of that test.

Would you rather keep the carry-forward in last_reported_metrics_ and filter when building the MetricData, so the baseline survives the gap?

good catch!

I moved the is_async guard logic to the export callback, so the baseline state gets kept.
2 tests have been added:

• AttributeReappearanceAfterGapCumulativeTemporality
• MultiCollectorDeltaTemporality to test the slow path

Hope everything is addressed now :-)

[mateenali66]

rebuilt on 70a6832e. the baseline survives now. A=10, absent, A=30 on one collector gives 10, nothing, 30, and AttributeReappearanceAfterGapCumulativeTemporality pins exactly that. 226 metric tests pass.

also ran the two-collector cumulative case, since the new multi-collector test is delta and the is_async_ filter only fires on cumulative. second collector's interval spanning a cycle it did not itself collect in: c2 reports 10, exports zero points at the gap, then reports 30 on reappearance rather than 20 or 50. the carry-forward holds on the slow path too.

one note, not a blocker. the filter tests delta_metrics, which is the current call's map rather than the union of that collector's unreported deltas. with two collectors, c2 can skip an export for an attribute that another collector's cycle observed inside c2's own interval. the cumulative value is still right when it reappears, so this is a cadence difference rather than lost data, but worth confirming it is deliberate.

looks right to me otherwise.

[vahle-at-psu]

rebuilt on 70a6832e. the baseline survives now. A=10, absent, A=30 on one collector gives 10, nothing, 30, and AttributeReappearanceAfterGapCumulativeTemporality pins exactly that. 226 metric tests pass.

also ran the two-collector cumulative case, since the new multi-collector test is delta and the is_async_ filter only fires on cumulative. second collector's interval spanning a cycle it did not itself collect in: c2 reports 10, exports zero points at the gap, then reports 30 on reappearance rather than 20 or 50. the carry-forward holds on the slow path too.

one note, not a blocker. the filter tests delta_metrics, which is the current call's map rather than the union of that collector's unreported deltas. with two collectors, c2 can skip an export for an attribute that another collector's cycle observed inside c2's own interval. the cumulative value is still right when it reappears, so this is a cadence difference rather than lost data, but worth confirming it is deliberate.

looks right to me otherwise.

Again well analyzed - thank you!
I thought it would be better if I again change the introduced tests to run as fixture with both Delta and Cumulative temporality like the existing tests. Now they cover more cases inherently.
The cadence in the multi-collector case could be mitigated by remembering the individual collections and then filter on this set.

[lalitb]

In PR desc, Behavior section:

Sync cumulative, attribute not measured this cycle	Carried forward (correct)	Unchanged

Please update the description's delta-reappearance row to say "unchanged." Main already preserves the baseline across gaps, and this PR does not change that delta behavior.

[vahle-at-psu]

641ce25 fixed format and clang-tidy issues

[mateenali66]

re-checked at 641ce251. the cumulative baseline point is fixed, and lalitb's two-reader case now exports {GET=12} for counter, gauge and up-down.

one regression at the cardinality limit. two readers, default limit, no view. reader 1's round observes 2000 sets, then reader 2 (cumulative) observes only x=7. reader 2 exports 0 points at head, main exports OVERFLOW=7. x goes to the overflow key when reader 2's queue is merged, and the filter at temporal_metric_storage.cc:206 looks the overflow key up in delta_metrics, where it never is. the spec says measurements MUST NOT be dropped during an overflow.

smaller thing, every new test uses ObservableCounter, but #4108 reports gauge and up-down, which go through BuildDeltaFromRound since #4635.

[vahle-at-psu]

re-checked at 641ce251. the cumulative baseline point is fixed, and lalitb's two-reader case now exports {GET=12} for counter, gauge and up-down.

one regression at the cardinality limit. two readers, default limit, no view. reader 1's round observes 2000 sets, then reader 2 (cumulative) observes only x=7. reader 2 exports 0 points at head, main exports OVERFLOW=7. x goes to the overflow key when reader 2's queue is merged, and the filter at temporal_metric_storage.cc:206 looks the overflow key up in delta_metrics, where it never is. the spec says measurements MUST NOT be dropped during an overflow.

smaller thing, every new test uses ObservableCounter, but #4108 reports gauge and up-down, which go through BuildDeltaFromRound since #4635.

Thx! 5781472 incorporates your feedback
It further adds a test for non-monotonic instruments ObservableGauge and ObservableUpDownCounter

[mateenali66]

re-checked at d6f95afc. the overflow drop is fixed at both limits, and the new test covers gauge and up-down.

one regression from the new merge order. temporal_metric_storage.cc:144-157 admits the previous cumulative state before the current round for async instruments too, and stale async sets stay there, so once a reader has held limit sets every later set exports only as overflow. one cumulative reader, gauge, 2000 sets once, then only x=7. head exports {OVERFLOW=7}, main and 641ce251 export x=7. the next round, {x=8, y=3}, gives {OVERFLOW=8}. the spec has async instruments prefer the callback's first-observed attributes.

keeping that order for sync and merging the reader's own round first for async cumulative fixed both locally. reader 2 then exports x=7, and only AsyncCumulativeOverflowIsExportedForLaggingReader changes.

the sync side changed too, limit 2 now keeps a and b plus OVERFLOW=12, but the description says sync is unchanged and no test pins it.

[vahle-at-psu]

re-checked at d6f95afc. the overflow drop is fixed at both limits, and the new test covers gauge and up-down.

one regression from the new merge order. temporal_metric_storage.cc:144-157 admits the previous cumulative state before the current round for async instruments too, and stale async sets stay there, so once a reader has held limit sets every later set exports only as overflow. one cumulative reader, gauge, 2000 sets once, then only x=7. head exports {OVERFLOW=7}, main and 641ce251 export x=7. the next round, {x=8, y=3}, gives {OVERFLOW=8}. the spec has async instruments prefer the callback's first-observed attributes.

keeping that order for sync and merging the reader's own round first for async cumulative fixed both locally. reader 2 then exports x=7, and only AsyncCumulativeOverflowIsExportedForLaggingReader changes.

the sync side changed too, limit 2 now keeps a and b plus OVERFLOW=12, but the description says sync is unchanged and no test pins it.

sorry for always reintroducing regressions - I am missing the big picture and dependencies within the code base.
The topic is getting quite complex and the changes are growing...
Hopefully continuously adding tests improves the situation and show up regressions faster in future.

With cdf2aff I instructed Copilot to fulfill all relevant metric specs, address feedback of all reviewers in this PR while keeping the changes minimal (except in tests)

Do you agree with continuing this change/review iteration in this way or does it make sense to revisit the whole topic again from beginning?

[mateenali66]

worth continuing here on a narrower scope, not restarting. re-checked cdf2aff4. the churn case now exports x=7, then x=8, y=3, N1 keeps x under its own key, all 248 metrics tests pass and TSan is clean on the touched tests. every earlier regression is now pinned by a test, which a restart would lose.

the new issue is memory. last_observed_monotonic_hash_maps_ is never capped or pruned, so an ObservableCounter with churning attributes grows for good. one reader, 200k new sets: +43 MiB at head, +0.3 MiB on main. up-down still double counts at the limit as on main (x observed 7, 9, 9 exports 7, 16, 25), so the uncapped map only fixes counters. capped per reader again, only CardinalityLimitPreservesCounterBaseline fails locally, and the double count can be its own issue.

@lalitb reader scoping now changes multi-reader delta output too. keep it in this PR?

[vahle-at-psu]

I want to summarize the remaining design trade-off before expanding this PR further.

The current change addresses stale async exports, reader-scoped observations, and overflow handling. However, last_observed_monotonic_hash_maps_ still retains per-reader counter baselines for every observed attribute set, so sustained attribute churn can grow that state without bound.

A simple cap on this map is not safe by itself: it can evict a source baseline while TemporalMetricStorage::last_reported_metrics_ still retains the corresponding cumulative value. If that attribute reappears, the SDK may treat its absolute observation as a new delta and merge it into the old cumulative state. The OTel SDK spec also requires an evicted cumulative series to be treated as a new series when it reappears, including a new start timestamp.

Sibling SDKs illustrate different approaches:

• Java keeps cardinality-limited async aggregator handles per reader and retires handles with no observations. Its reappearance test checks the new series start time.
• .NET uses a fixed-size MetricPoint pool: delta temporality reclaims unused slots, while cumulative temporality keeps allocated slots and routes new attribute sets to overflow. Its memory-management docs describe this model.
• Go uses cardinality-limited aggregation maps, but its cumulative sum still notes stale-series cleanup as TODO. Its experimental FinishSum provides explicit series retirement.

The current C++ MetricData model has one start_ts shared by all points, which makes independent per-series eviction and restart difficult. These sibling implementations therefore provide useful patterns, but none appears directly portable without broader changes.

Could maintainers advise whether to:

1. Keep this PR focused and address bounded baseline lifecycle in a follow-up;
2. Expand this PR to coordinate per-reader baseline eviction with cumulative state and per-series start times; or
3. Split the reader-scoped delta behavior into a separate change if that is outside this PR's scope?

I have not included a standalone baseline cap because it can break cumulative correctness unless eviction and series reset are coordinated.

[mateenali66]

the hazard is real, but only for a cap that evicts. main's AttributesHashMap never evicts, it stops storing baselines at the limit, so a cap like main's is no worse than main.

built at cdf2aff4. ObservableCounter, view keeping only k, limit 2, one cumulative reader. pid 1 at 7, then pids 2 and 3 at 1, then pid 1 at 8, then all three twice. head gives x = 7, 9, 10, 10, 10.

an LRU cap that evicts gives 7, 9, 17, 18, 19, your case. main gives 7, 9, 10, 11, 12, as does head with the counter baseline capped like up-down's at async_metric_storage.h:226-230. that is main's double count past the limit, which up-down already has at head.

with that cap, 200k churned sets cost +0.2 MiB, not +43 MiB, and only CardinalityLimitPreservesCounterBaseline fails. option 1 with that cap looks right to me. the choice is a maintainer's.

[vahle-at-psu]

Thanks, agreed. I changed the counter-baseline tracking to a non-evicting per-reader cap, matching the existing main behavior: retain baselines for the first-observed attribute sets up to the configured cardinality limit, then stop admitting new ones. This bounds memory under attribute churn without evicting and later re-seeding previously tracked baselines.

CardinalityLimitCapsCounterBaselines checks the bound and pins the behavior once the limit is reached. I agree that the remaining counter/UpDownCounter behavior beyond the limit is a separate cardinality issue and can be tracked separately.

I recommend keeping reader-scoped delta output in this PR because callback observations are scoped to the collecting reader by the spec, and the multi-reader tests cover both delta and cumulative temporality. If you think that scope should be split, I’m happy to do that.