Span batch size exceeds the queue size, so spans are dropped since the opentelemetry-cpp bump in #143

Since #143, esp_opentelemetry_tracing_setup() drops most spans on any app that ends more than a few spans a second. It also stalled the camera on dust-mite's car.

Symptom

dust-mite moved from bb5c90c to dd73720 (#143) and changed nothing else. Its car then streamed Camera frames over HTTP at about 18 spans a second (one camera.frame.send span per frame).

Every build (base, all-features, profiling) logs BatchSpanProcessor queue is full - dropping span. repeatedly, from sdk/src/trace/batch_span_processor.cc:93.
After 15–30 s the camera stops delivering frames (cam_hal: Failed to get frame: timeout), and dust-mite's smoke test fails on a 7 s gap with no frames.

The same car code on bb5c90c passes the smoke test, with no queue-full lines and no camera timeouts.

Cause

src/integration/esp_tracing.cpp sets max_queue_size (CONFIG_ESP_OPENTELEMETRY_BATCH_MAX_QUEUE_SIZE, default 64) and schedule_delay_millis. It leaves max_export_batch_size at the SDK default of 512. esp_logs.cpp does the same with CONFIG_ESP_OPENTELEMETRY_LOGS_BATCH_MAX_QUEUE_SIZE (128).

BatchSpanProcessorOptions documents that max_export_batch_size "must be smaller or equal to max_queue_size", and the OpenTelemetry SDK specification says the same. Our configuration has broken that rule all along, but it did no harm until #143 bumped opentelemetry-cpp to 7f93577b. That version includes open-telemetry/opentelemetry-cpp#4466 (a2739943, "BatchSpanProcessor: wait for a full batch instead of draining partial batches"):

The worker's wait condition changed from !buffer_.empty() to buffer_.size() >= max_export_batch_size_.
With a queue of 64 and a batch of 512, that condition can never hold. The worker therefore exports only when schedule_delay_millis (5000 ms) expires.
At 18 spans a second the queue fills in about 3.5 s. Spans are dropped, with a warning each, until the next scheduled export.

BatchLogRecordProcessor still drains partial batches, so logs are not affected yet. The same mismatch applies to them, though.

Verified fix

Adding only batch_options.max_export_batch_size = CONFIG_ESP_OPENTELEMETRY_BATCH_MAX_QUEUE_SIZE / 2; to esp_tracing.cpp on dd73720 makes dust-mite's smoke test pass on the car, with no queue-full lines and no camera timeouts.

Proposed change

Add CONFIG_ESP_OPENTELEMETRY_BATCH_MAX_EXPORT_BATCH_SIZE and CONFIG_ESP_OPENTELEMETRY_LOGS_BATCH_MAX_EXPORT_BATCH_SIZE, defaulting to half of each queue size. Pass them to the span and log processor options.
Reject a batch size larger than its queue at build time, with a static_assert or a Kconfig range, rather than leaving it to the SDK.
Cover the case with a test that ends spans faster than schedule_delay_millis allows and checks that none are dropped.

Upstream observations (not filed)

opentelemetry-cpp does not validate or clamp max_export_batch_size against max_queue_size, so a bad configuration fails silently. The Java SDK throws and the Python SDK raises ValueError.
After #4466, OnEnd still wakes the worker once the queue is half full (batch_span_processor.cc:100), but the new wait condition ignores that wake-up unless a full batch is ready. With a batch size equal to the queue size, which is allowed, nothing exports until the queue is completely full, and spans that arrive during the export are dropped.

Related: stale CMake cache after the bump

#143 renames opentelemetry-cpp's options to OTELCPP_*. An existing build directory still holds WITH_ETW:BOOL=OFF in CMakeCache.txt, and the reconfigure then fails with WITH_ETW is unavailable.OTELCPP_WITH_ETW is only supported on Windows (cmake/tools.cmake:450). Deleting CMakeCache.txt fixes it, and the compiled objects are still reused. Consider noting this in the release notes or README.

Spec

This spec refines the issue above. Where it differs from the issue's "Proposed change" section, this spec takes precedence.

Problem Statement

An application that ends more spans per second than the queue can hold in one schedule delay loses most of its spans. The BatchSpanProcessor exports only when a full batch is ready. The export batch size is left at the SDK default of 512, but the queue holds only 64, so a full batch never forms. Exports happen only when the 5 s schedule delay expires, and in the meantime the queue fills and every further span is dropped with a warning. On dust-mite's car, the dropping and warning at about 18 spans a second also stalled the camera.

The log record processor has the same batch-larger-than-queue mismatch (512 against 128). It works today only because the logs SDK still drains partial batches.

Solution

The export batch size becomes a build-time setting for each batched signal, configured the same way as its queue size. It defaults to half of that signal's default queue size. Spans and log records then export as soon as a batch is ready, without waiting for the schedule delay, and nothing is dropped at rates the queue can absorb.

User Stories

As a firmware developer, I want spans to be exported as soon as a full batch is ready, so that spans aren't dropped while the processor waits for the schedule delay.
As a firmware developer, I want to end tens of spans a second without BatchSpanProcessor queue is full warnings, so that my console isn't flooded and tracing doesn't slow my application down.
As a firmware developer, I want the default configuration to keep the export batch size no larger than the queue size, so that the out-of-the-box setup follows the OpenTelemetry SDK specification.
As a firmware developer, I want to set the span export batch size in menuconfig next to the span queue size, so that I can tune both together for my RAM budget and span rate.
As a firmware developer, I want to set the log record export batch size in menuconfig next to the log queue size, so that logs are tuned the same way as spans.
As a firmware developer, I want the span batch-size option to appear only when tracing is enabled, so that menuconfig doesn't show settings for signals I don't use.
As a firmware developer, I want the log batch-size option to appear only when logs are enabled, so that menuconfig stays relevant.
As a firmware developer who has not changed the queue size, I want the default batch size to be half of the default queue, so that each export leaves room for spans that arrive while it is in progress.
As a firmware developer, I want log records to keep exporting correctly after a future opentelemetry-cpp bump that changes the log processor to wait for a full batch, so that logs don't break the way spans did.
As a firmware developer, I want the camera (or any other time-critical task) to keep running while tracing is on, so that observability doesn't change the behaviour I'm observing.
As an esp-opentelemetry-cpp maintainer, I want CI to fail if a future SDK bump makes the span processor wait for the schedule delay again, so that this regression can't come back unnoticed.
As an esp-opentelemetry-cpp maintainer, I want CI to fail if log records stop being exported before the schedule delay, so that the logs side is guarded by the same test.
As an esp-opentelemetry-cpp maintainer, I want the regression test to go through the public setup functions with an injected exporter, so that it tests behaviour users rely on rather than internals.
As dust-mite's car, I want to stream Camera frames with one span per frame and pass the smoke test, so that I can move to the current esp-opentelemetry-cpp without a local workaround.

Implementation Decisions

Two new integer build-time options, one per batched signal:
- span export batch size: default 32, available only when tracing is enabled, placed next to the span queue size;
- log record export batch size: default 64, available only when logs are enabled, placed next to the log queue size.
Each option has a prompt string only: no help text and no range.
The existing queue-size options don't change: no range, no help text, and the same defaults (64 and 128).
The tracing setup passes the span batch-size option to the BatchSpanProcessor options as max_export_batch_size. The logs setup does the same for the BatchLogRecordProcessor. The schedule delay settings don't change.
The rule that the batch size must not exceed the queue size is not enforced in this component, with neither a static_assert nor a Kconfig range. Enforcement belongs in opentelemetry-cpp, which is separate work. A queue or batch size of 0 counts as a configuration error and is not guarded.
The public setup API doesn't change. Applications that inject their own SpanProcessor or LogRecordProcessor are unaffected.
Metrics and profiles are left alone. Metrics use a periodic reader that aggregates in place, with no queue or batch. Profiles use their own sample ring and export interval, which is a separate mechanism with its own drop rules.

Testing Decisions

A good test here checks only external behaviour: every item the application ends or emits reaches the exporter, and it does so before the schedule delay expires. It must not inspect the processor's buffer or worker.
Seam: the public tracing and logs setup functions, which already accept an injected exporter. The test passes a counting exporter and uses the real batch processors with the component's default configuration (queue 64 / batch 32 for spans, queue 128 / batch 64 for logs, 5 s schedule delay). No new seam is needed.
A new QEMU test app with two test cases, one assertion each:
- Span case: end twice the span queue size in short bursts. Assert that the exporter received every span within about 2 s, well before the 5 s schedule delay. Before the fix this fails: about half the spans are dropped, and the rest arrive only after the delay.
- Log record case: the same scenario for log records. It passes before the fix, because the log processor still drains partial batches. It guards against a future SDK bump that ports #4466-style behaviour to logs.
The new app joins the CI matrix as a QEMU entry.
Prior art: the existing QEMU test apps (signals_disabled, profiling_only, profiles_export_memory) and their test_qemu.sh runners in the CI matrix.
Delivery order: a test commit first (the span case fails), then the fix commit, which closes this issue.

Out of Scope

Enforcing or clamping batch ≤ queue in opentelemetry-cpp, and the issue's other upstream observation about the half-full wake-up being ignored. Both are separate work.
The stale CMakeCache.txt / WITH_ETW reconfigure failure after the bump in Remove the instrument metadata validator workaround #143.
Ranges or help text on the queue-size options.
Drop or rate tests for metrics and profiles.
Bumping the submodule in dust-mite, which is a follow-up once the fix has merged.

Further Notes

The fix was verified on dust-mite's car by setting the span batch size to half the queue size on dd73720: the smoke test passed, with no queue-full lines and no camera timeouts.
Half the queue is a deliberate default. It matches the SDK's own wake-up when the queue is half full, so every wake-up has a full batch to export, and half the queue stays free for spans that arrive during an HTTP export. A batch size equal to the queue size is allowed, but spans that arrive during an export are then dropped.

Spec: upstream opentelemetry-cpp clamps `max_export_batch_size` to `max_queue_size`

This spec covers the upstream follow-up to the first item in "Upstream observations" above. It is separate from the fork fix that this issue proposes (Kconfig options, build-time check). The fork still needs that fix.

Problem Statement

In opentelemetry-cpp, a BatchSpanProcessor configured with a max_export_batch_size larger than its max_queue_size fails silently. The OpenTelemetry SDK specification states that the batch size must be less than or equal to the queue size, and the options' doc comments say the same, but the SDK neither validates nor corrects the value.

Since open-telemetry/opentelemetry-cpp#4466, the worker waits for a full batch. If the batch is larger than the queue, a full batch can never be ready, so the worker exports only when schedule_delay_millis expires. Any application that ends spans faster than the queue size per schedule delay then drops spans, logging a "queue is full" warning for each. Nothing points at the configuration as the cause. On dust-mite's car, this dropped most camera.frame.send spans and stalled the camera.

Other SDKs do not accept this configuration:

Java clamps the batch size to the queue size and logs a warning: BatchSpanProcessorBuilder.build() (L191-L197), and the same in BatchLogRecordProcessorBuilder.build() (L184-L189).
Python raises ValueError("max_export_batch_size must be less than or equal to max_queue_size."): BatchSpanProcessor._validate_arguments (L287-L288), called from __init__ (L183), and the same in BatchLogRecordProcessor._validate_arguments (L392-L393).

opentelemetry-cpp must also work with exceptions disabled, so raising an error the way Python does is not an option.

Solution

When a BatchSpanProcessor is constructed with max_export_batch_size larger than max_queue_size, it uses max_queue_size as the batch size. It also logs one internal warning that names both configured values and the value it uses instead. A valid configuration (batch size less than or equal to the queue size) is left unchanged and logs nothing.

This matches the Java SDK, which clamps the batch size to the queue size and logs a warning (links above). It also follows a precedent within opentelemetry-cpp: PeriodicExportingMetricReader corrects an invalid timeout/interval pair at construction and logs a warning instead of failing.

The value is clamped to max_queue_size, not to some smaller value, because a batch size equal to the queue size is a valid configuration under the specification.

User Stories

As an SDK user, I want a batch size larger than the queue size to be corrected automatically, so that my spans are exported instead of silently dropped.
As an SDK user, I want a warning when my batch size is corrected, so that I can see and fix the misconfiguration.
As an SDK user, I want the warning to name both configured values and the value used, so that I can fix my configuration without reading SDK source code.
As an SDK user, I want the warning logged once at construction, not once per span, so that it doesn't flood my logs.
As an SDK user with a valid configuration, I want my batch size left exactly as configured, so that the fix doesn't change behaviour I rely on.
As an SDK user who sets a batch size equal to the queue size, I want that value kept, because the specification allows it.
As an SDK user who configures the processor through OTEL_BSP_* environment variables, I want the same correction, so that a bad environment cannot silently drop spans.
As an SDK user who configures the processor through declarative (YAML) configuration, I want the same correction, so that a bad configuration file cannot silently drop spans.
As an SDK user who builds the options struct in code, I want the same correction, so that every configuration path behaves the same.
As an SDK user who passes runtime options (thread instrumentation), I want the same correction, so that the choice of constructor doesn't matter.
As an SDK user building with exceptions disabled, I want an invalid configuration handled without throwing, so that my build and runtime keep working.
As an embedded SDK user with a small, memory-budgeted queue, I want my queue size kept as configured, so that the fix doesn't raise my memory usage.
As an SDK user upgrading past #4466, I want a configuration that used to work by accident to keep exporting spans, so that the upgrade doesn't quietly lose telemetry.
As an SDK maintainer, I want the correction applied in the one place every configuration path goes through, so that there is exactly one warning and one rule.
As an SDK maintainer, I want no change to public API headers apart from documentation, so that the fix stays within the ABI policy.
As an SDK maintainer, I want the options' documentation to say that larger values are clamped, so that the documented and the actual behaviour match.
As an SDK maintainer, I want a regression test that fails on the current main, so that the bug cannot come back unnoticed.
As a reviewer, I want the change limited to the span processor, so that the PR is small and doesn't touch the log processor's separate open issue.

Implementation Decisions

Scope: only the span processor (BatchSpanProcessor). The log processor is out of scope (see below).
Where the check lives: in both BatchSpanProcessor constructors (options only, and options plus runtime options). Environment variables, the options struct and declarative configuration all go through these constructors, so no other place needs changing. The declarative-configuration parser and builders stay as they are.
What happens on a violation: the effective batch size becomes max_queue_size, and an internal warning is logged at construction. The SDK does not throw, abort or reset values to defaults. Resetting to defaults would raise the queue size and the memory it uses.
Valid configurations (batch size ≤ queue size) are not changed and log nothing.
Mechanism: the batch-size member is const, so the clamp runs in each constructor's initializer list through a file-local helper. The helper returns the effective value and logs the warning. There is no new shared or public helper and no build-system change.
Warning content: names the processor and states the configured batch size, the configured queue size and the value used, in this format: [BatchSpanProcessor] max_export_batch_size (512) is larger than max_queue_size (64), using 64.
Public API: no signature, member or ABI changes. The only header change is one sentence added to the max_export_batch_size doc comment on the span processor options, saying that larger values are clamped to max_queue_size with a warning.
Changelog: one entry under Unreleased.
Commit and PR mechanics: one local commit on a branch in the local opentelemetry-cpp checkout, based on the local main with no fetch. The PR description is drafted locally. Nothing is pushed or opened upstream until the maintainer decides to.

Testing Decisions

What a good test does: it checks only externally observable behaviour, here what the exporter receives and when. It does not read private or protected state, does not add a test-only accessor or friend declaration, and does not check the warning text.
Seam: the existing span exporter mock that the span processor tests already use. It records how many spans it receives and the size of each batch. No new seam is needed.
Tests (span processor test suite, existing fixture, one assertion each):
1. Batch size larger than the queue is clamped. Use a queue of 8, a batch size of 512 and a long schedule delay, end 8 spans, and wait briefly. Expect all 8 spans to have been exported. On the current main this test fails, because nothing exports until the schedule delay expires.
2. A batch size within the queue size is kept. Use a queue of 8 and a batch size of 4, end 8 spans, and force a flush. Expect the first exported batch to hold 4 spans. This catches a clamp that wrongly changes valid values.
No warning tests. The warning is still logged, but a test for it would need the internal log-handler mock, and that mock isn't shared between test suites.
Stability: repeat the new tests many times to confirm the short wait is reliable.
Prior art: the span processor tests that drive a blocking exporter mock, check recorded batch sizes against the configured maximum, and wait briefly before checking. The SDK's existing check for an invalid metric reader timeout/interval pair is the behavioural precedent, though it has no test.

Out of Scope

Log processor. BatchLogRecordProcessor has the same mismatch, but it still exports partial batches ([SDK] BatchLogRecordProcessor still drains the queue in a tight loop, inconsistent with BatchSpanProcessor fix open-telemetry/opentelemetry-cpp#4498), so a clamp there can't be observed through its exporter yet. Follow up once #4498 lands.
Zero values. A batch size of 0 makes the post-#4466 worker spin in a tight loop, and a queue size of 0 drops every span. Possible follow-ups.
Half-full wake-up after #4466. OnEnd still wakes the worker once the queue is half full, but the worker ignores that wake-up unless a full batch is ready. With a batch size equal to the queue size (valid, and the result of this clamp), the worker exports only when the queue is completely full. That leaves no headroom: spans that arrive while the worker is waking up are dropped. This is a behaviour change to #4466, so it needs its own issue and discussion.
Declarative-configuration parse-time validation. No second warning at parse time.
The fork fix in this issue (Kconfig batch-size options, build-time check, fork-level drop test). That fix is tracked here separately and is still needed, because the clamp alone leaves the fork at batch size = queue size.
Any remote action upstream (issue, push, PR). These stay the maintainer's call.

Further Notes

Java comparison: the clamp has the same shape as this change, so the PR can cite it directly. Note that the issue body above says Java throws. That is incorrect: Java clamps and warns.
Specification: the batch span processor's maxExportBatchSize "must be less than or equal to maxQueueSize".
Related upstream: #4466 (merged; added the full-batch wait condition that exposes the bug) and #4498 (open; the log processor still drains in a tight loop).
The PR description should list the three follow-ups above and include the dust-mite reproduction from this issue: about 18 spans per second with a queue of 64 and the default batch size of 512.

Bump third_party/opentelemetry-cpp to v1.30.0

Keep each export batch within its queue

Navigation Menu

Repository navigation

Span batch size exceeds the queue size, so spans are dropped since the opentelemetry-cpp bump in #143 #145

Symptom

Cause

Verified fix

Proposed change

Upstream observations (not filed)

Related: stale CMake cache after the bump

Spec

Problem Statement

Solution

User Stories

Implementation Decisions

Testing Decisions

Out of Scope

Further Notes

Spec: upstream opentelemetry-cpp clamps `max_export_batch_size` to `max_queue_size`

Problem Statement

Solution

User Stories

Implementation Decisions

Testing Decisions

Out of Scope

Further Notes

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Span batch size exceeds the queue size, so spans are dropped since the opentelemetry-cpp bump in #143 #145

Description

Symptom

Cause

Verified fix

Proposed change

Upstream observations (not filed)

Related: stale CMake cache after the bump

Activity

ltowarek commented on Oct 21, 2026

Spec

Problem Statement

Solution

User Stories

Implementation Decisions

Testing Decisions

Out of Scope

Further Notes

ltowarek commented on Oct 21, 2026

Spec: upstream opentelemetry-cpp clamps max_export_batch_size to max_queue_size

Problem Statement

Solution

User Stories

Implementation Decisions

Testing Decisions

Out of Scope

Further Notes

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Spec: upstream opentelemetry-cpp clamps `max_export_batch_size` to `max_queue_size`