Token-based JSON serialization interface for protobuf-free OTLP/JSON #102

Problem Statement

The ESP32 firmware links the full protobuf C++ runtime and Abseil for one reason: to serialize OTLP payloads. Nothing else in the firmware needs them. They dominate the binary, slow every build, and are a recurring source of toolchain breakage — this repo already carries a set of workarounds solely to keep protobuf and Abseil compiling and linking for an embedded target.

OpenTelemetry defines two OTLP/HTTP encodings, binary protobuf and JSON. Switching to http/json looks like the obvious escape, but it is not one: opentelemetry-cpp implements OTLP/JSON on top of protobuf. Its JSON converter walks messages through protobuf reflection (GetReflection, ListFields, json_name), which exists only in the full runtime. The OTLP HTTP client is typed on google::protobuf::Message, the trace recordable embeds a generated Span by value, and the response path parses replies with protobuf's JsonStringToMessage. Selecting JSON therefore changes the bytes on the wire and removes nothing from the binary.

The consequence is that a device emitting human-readable JSON still pays the entire cost of a binary serialization framework it never invokes. There is no build configuration, in this repo or upstream, that avoids it.

Solution

Introduce a token-based JSON serialization interface in opentelemetry-cpp, then build OTLP/JSON exporters on top of it that never touch protobuf.

This follows a proposal an opentelemetry-cpp maintainer already made in upstream issue #2541: an abstract interface for JSON serialization with a default nlohmann-json implementation, letting users supply their own, modelled on the existing HttpClientFactory/curl split. The refinement here is that the interface speaks tokens (StartObject, Key, Value, EndArray) rather than documents. Abstracting a document object means abstracting node identity, ownership and allocators across libraries whose shapes do not agree; abstracting a stream of calls requires no shared data structure at all. It is the smaller abstraction, and it is the only one that permits a non-DOM backend later.

The work splits into two upstream changes plus an embedded backend:

• PR1 adds the interface and retrofits the two existing reflection-based converters onto it. No behavior change; it also deduplicates two near-identical copies of the same mapping code.
• The cJSON backend in this repo implements the interface for ESP-IDF and is proved by porting the existing profiles exporter onto it — validating the interface on hardware before PR2 begins.
• PR2 adds OTLP/JSON exporters for traces, metrics and logs that map from SDK recordables directly, emit through the interface, and parse responses without protobuf.

After PR2, the firmware links neither protobuf nor Abseil. The profiles exporter is already protobuf-free and needs no upstream counterpart, because opentelemetry-cpp has no profiles SDK.

User Stories

1. As a firmware developer, I want the car binary to link neither protobuf nor Abseil, so that I recover the flash they currently occupy for application code.
2. As a firmware developer, I want OTLP export to work without a binary serialization framework, so that my build stops breaking whenever protobuf or Abseil release a new version.
3. As a firmware developer, I want to delete the protobuf and Abseil workarounds this repo carries, so that the component has less code that exists only to appease a dependency.
4. As a firmware developer, I want builds to complete faster, so that my iteration loop on hardware is shorter.
5. As a firmware developer, I want the JSON backend to use cJSON, so that I depend on the JSON library ESP-IDF already ships rather than maintaining my own.
6. As a firmware developer, I want the JSON backend to allocate no more than the request body demands, so that export does not spike heap on a memory-constrained device.
7. As a firmware developer, I want serialization to report failure without exceptions, so that it works in a build with exceptions and RTTI disabled.
8. As a firmware developer, I want serialization failures logged through the SDK's internal log handler, so that I can diagnose them over the serial console like any other SDK error.
9. As a firmware developer, I want the existing profiles exporter ported onto the new interface, so that the component has one JSON emission path instead of two.
10. As a firmware developer, I want traces, metrics and logs converted together, so that the binary actually sheds protobuf rather than shedding it from one signal while another still links it.
11. As a firmware developer, I want partial-success responses parsed and logged, so that backend rejections are visible on a device I cannot otherwise debug.
12. As a firmware developer, I want the OTLP/JSON deviations — hex-encoded trace and span IDs, integer-valued enums, 64-bit integers as decimal strings — handled correctly, so that collectors accept my payloads.
13. As a firmware developer, I want 64-bit nanosecond timestamps emitted as strings rather than JSON numbers, so that they are not silently truncated by double-precision rounding.
14. As an upstream maintainer, I want the JSON interface to match the shape I already proposed in issue #2541, so that reviewing it is a matter of checking execution rather than relitigating design.
15. As an upstream maintainer, I want the first change to alter no observable behavior, so that I can approve it on the strength of tests rather than on trust.
16. As an upstream maintainer, I want the two duplicated JSON converters unified, so that the codebase carries one copy of the mapping instead of two that can drift.
17. As an upstream maintainer, I want the interface to add no new third-party dependency, so that adopting it costs downstream consumers nothing.
18. As an upstream maintainer, I want the interface to follow the existing factory-injection pattern, so that it is consistent with how the HTTP client is already pluggable.
19. As an upstream maintainer, I want a test proving the protobuf and protobuf-free paths emit identical JSON, so that I am not accepting two mappings that will silently diverge.
20. As an upstream maintainer, I want the change to build under both CMake and Bazel, so that neither build system regresses.
21. As an upstream maintainer, I want the interface to remain a generic JSON emitter with no OTLP knowledge, so that it is justifiable as a general-purpose component.
22. As an opentelemetry-cpp consumer, I want to substitute rapidjson or Boost.JSON for nlohmann, so that I can address the serialization performance gap reported in issue #2541.
23. As an opentelemetry-cpp consumer on a constrained platform, I want to supply a backend that streams rather than builds a document, so that peak memory is bounded by output size.
24. As an opentelemetry-cpp consumer, I want to exclude nlohmann from my build entirely, so that I do not compile a library I have replaced.
25. As an opentelemetry-cpp consumer, I want the default build to behave exactly as before, so that upgrading requires no action from me.
26. As a dust-mite operator, I want traces, metrics, logs and profiles to keep arriving in Tempo, Mimir, Loki and Pyroscope, so that observability is unaffected by the serialization change.
27. As a dust-mite operator, I want the existing end-to-end tests to pass unmodified, so that correctness is judged on what reaches the backends rather than on what the device claims to send.
28. As a dust-mite operator, I want the profiles pipeline undisturbed, so that a change aimed at traces does not regress a working signal.
29. As a maintainer of this component, I want the upstream branch based on upstream main, so that the change stays mergeable without a painful rebase.
30. As a maintainer of this component, I want the second change branched off the first, so that I can develop it while the first is still under review.
31. As a maintainer of this component, I want the interface exercised by a second, independent backend before it is proposed upstream, so that it is not accidentally shaped around a single caller.
32. As a maintainer of this component, I want the CMake gating from issue Add Kconfig option to disable OTLP/HTTP exporter and eliminate the protobuf+Abseil dependency #38 completed first, so that excluding protobuf is actually possible once the exporters no longer need it.
33. As a maintainer of this component, I want to confirm by inspection that protobuf is no longer compiled, so that I know the exclusion took effect.
34. As a maintainer of this component, I want to decide myself when anything is submitted upstream, so that public contributions happen on my timing.

Implementation Decisions

Two seams, not one. The maintainer's proposal in #2541, read literally, keeps protobuf reflection and swaps only where JSON lands — it resolves the performance complaint but removes no dependency. Reaching a protobuf-free build needs a second, larger change where mapping code reads SDK recordables directly. These are sequenced as PR1 and PR2 rather than conflated.

Interface shape. A JsonWriter abstract class plus a JsonWriterFactory, both placed alongside the HTTP client abstraction in the ext tree, mirroring HttpClientFactory. Token vocabulary: begin/end object, begin/end array, key, and typed value overloads covering string, 32- and 64-bit signed and unsigned integers, double, bool and bytes.

Naming. JsonWriter, not JsonSerializer — "serializer" implies ownership of the mapping, which this type must not have. "Writer" also matches the term rapidjson and Boost.JSON use for the same shape.

Error contract. Every method returns void and is marked noexcept. Failure is recorded in a sticky flag exposed by a single ok() query, with the SDK's internal error log emitted at the first failure. This follows the codebase's existing convention of logging plus status returns rather than exceptions, and it is required because the firmware builds with exceptions and RTTI disabled — a configuration opentelemetry-cpp already contemplates through its OPENTELEMETRY_HAVE_EXCEPTIONS macro. There is no existing Status type in the SDK to reuse.

Output model. The writer accumulates and hands the caller a finished string. The token vocabulary is chosen so a streaming backend remains possible later, but incremental flush is not built now — it would require the HTTP client to consume a stream, which is a separate concern.

Injection. No global setter. Each affected exporter factory entry point gains an overload accepting a shared pointer to a JsonWriterFactory, defaulting to a function returning the nlohmann implementation. This mirrors exactly how a custom HttpClientFactory is supplied today. Most of PR1's line count lives in these parallel overloads.

Ownership of OTLP deviations. The mapping layer applies hex-encoded IDs, integer enums and 64-bit-as-string; the writer stays a generic JSON emitter with no OTLP awareness. Upstream has already drawn this boundary — the bytes-mapping policy is an option on the HTTP client, not on the JSON layer.

PR1 scope. Both existing reflection-based converters — the one in the OTLP HTTP client and the near-identical copy in the OTLP file client — retrofit onto the interface. Roughly 470 lines of duplicated mapping code across the two. Two converters rather than one is what makes this a seam rather than a local refactor. Writer only; no reader.

Build options. A WITH_JSON_WRITER_NLOHMANN option shaped as a cmake_dependent_option, defaulting on under the same conditions the curl HTTP client uses, with a matching Bazel select(). The existing coupling that forces nlohmann on whenever the OTLP HTTP exporter is enabled must be relaxed, or the backend can never be excluded. Option naming follows current unprefixed convention; the OTELCPP_ prefix migration is a separate upstream effort.

cJSON backend. Implemented in this repo, not upstream. cJSON is a document library, so the backend maintains a node stack to translate begin/end tokens into node construction — roughly 120 lines. This does not realize the streaming memory benefit; it buys the option at no extra cost, and cJSON is chosen deliberately to minimize hand-maintained code and to reuse the correct escaping and number formatting it already provides. cJSON is already a declared dependency of this component.

PR1 validation. The existing profiles exporter, which currently hand-rolls cJSON calls, is ported onto the new interface. This exercises the interface through a second independent consumer and is verifiable end to end on hardware without any part of PR2.

PR2 scope. Traces, metrics and logs in a single change, matching how the custom HTTP client was landed. Mapping reads SDK recordables; protobuf is not referenced. Profiles are excluded because opentelemetry-cpp has no profiles SDK — the existing exporter stays as it is, and it already needs no protobuf, so PR2 alone brings the firmware to zero.

Reader. A minimal document-shaped JsonReader of roughly five accessors, added in PR2, replacing the protobuf-based response parse. The asymmetry with the token-based writer is deliberate and must be stated explicitly in the change: the writer handles unbounded telemetry batches where streaming matters, while the reader handles a two-field partial-success response where a pull parser would be pure cost.

Response handling. Partial-success fields — rejected counts and error message — are parsed and logged. Discarding the response body entirely would hide genuine backend rejections on a device with no other diagnostic channel.

Prerequisite. The CMake conditionals tracked by issue #38 must land before PR2. Without them nothing is excluded from the build regardless of what the exporters need.

The HTTP client is retyped. OtlpHttpClient is typed on google::protobuf::Message — on the exported message, the arena owning the response, the response message, and the result callback — so no protobuf-free payload can be handed to it. Its transport core is therefore extracted into a protobuf-free OtlpHttpTransport that takes a finished request body and a callback receiving the raw response bytes, with OtlpHttpClient keeping its exact signature and delegating. Session pooling, retry, compression, flush and shutdown move across unchanged; the protobuf response parse and the arena move into the callback the protobuf client supplies. The transport lives in its own build target that does not link protobuf, which is what lets the new exporters link a send path without the message runtime. This is a shared prerequisite of all three signal exporters rather than the business of any one of them, and it is tracked as #117, which #110, #111 and #112 are blocked by. It was not anticipated when this spec was first written.

Branching. json-writer-seam for PR1 and otlp-json-no-protobuf for PR2, the same names across repos, with PR2 branched off PR1 so review feedback on PR1 does not force repeated rebases of PR2's mapping code. The cJSON backend work is a third independent line named json-writer-cjson, tracking PR1's interface rather than PR2's exporters. Upstream branches are based on upstream main; a fork remote is added to the opentelemetry-cpp checkout, and this repo's submodule URL is retargeted at the fork so the branches are consumable downstream.

Testing Decisions

A good test here asserts on what the exporter actually transmits, never on how it was constructed. The observable behavior is the HTTP request body; the token sequence that produced it is implementation detail and must not be asserted on.

Primary seam, reused not invented. opentelemetry-cpp already ships a no-send HTTP client target used by the OTLP HTTP exporter tests for all three signals. Those tests already run the exporter with JSON content type and assert on the captured request body. This is the highest available seam inside the library and it serves both changes:

• PR1 captures request bodies before the retrofit and asserts they are byte-identical afterwards, across traces, metrics and logs. This is what substantiates the no-behavior-change claim.
• PR2 runs identical input through the protobuf and protobuf-free paths and compares bodies. This equivalence test is the centerpiece, because "how do two mappings not drift" is the objection most likely to sink the proposal upstream.

Prior art. The existing custom-client tests for the trace, metric and log OTLP HTTP exporters exist specifically to verify injecting a custom HttpClientFactory. They are the pattern to copy for JsonWriterFactory injection.

No fake writer. A recording writer capturing token sequences was considered and rejected: with the request body available at a higher seam, a token-level fake would test implementation detail.

Hardware and end-to-end. This repo's existing device-gated end-to-end tests for traces, metrics, logs and profiles run unchanged against the headless observability stack. Their value is that they are encoding-agnostic — they assert on what arrived in Tempo, Mimir, Loki and Pyroscope, so any JSON mapping error surfaces without new test code. The profiles test doubles as a regression check on the ported exporter.

Binary inspection. Confirming protobuf is no longer compiled is a sanity check on the build configuration, not an acceptance criterion or a target to optimize against.

Out of Scope

• Opening pull requests against any upstream repository. Branches and descriptions are prepared; submission is the maintainer's decision alone.
• Commenting on upstream issue #2541 ahead of the change.
• Profiles support upstream. opentelemetry-cpp has no profiles SDK, so the existing exporter has no upstream counterpart and stays in this repo.
• Streaming the HTTP request body. The writer accumulates; incremental flush would require the HTTP client to consume a stream and is deliberately deferred. This excludes streaming specifically, not every change to the HTTP client — see the retyping decision above, which extracts the transport but still hands it a complete body.
• A raw string-appender backend with no JSON library. Considered and rejected in favor of cJSON to minimize maintained code.
• Binary protobuf via nanopb or a flattened schema. A viable fallback if payload size ever becomes the binding constraint, but it reintroduces a code generation step, which is what this work removes.
• Alternative wire protocols such as Zipkin or Elasticsearch. Both have protobuf-free HTTP exporters upstream, but neither speaks OTLP, and adopting one would change the collector configuration and lose OTLP semantics.
• Renaming CMake options to the OTELCPP_ prefix.
• A CONTEXT.md glossary or an architecture decision record.
• gzip compression of the request body. It would offset JSON's size disadvantage, but it costs a compression buffer in RAM and is independent of this work.

Further Notes

The design follows a proposal an opentelemetry-cpp maintainer made in upstream issue #2541 in February 2024: an abstract JSON serialization interface with a default nlohmann implementation, explicitly modelled on the HttpClientFactory/curl split. That issue went stale two months later and remains unclaimed, so the seam has maintainer endorsement without anyone having acted on it.

That same issue reports rapidjson serializing 40–50% faster than nlohmann. The interface therefore serves two constituencies at once — constrained devices and throughput-sensitive users — which is a stronger position than an embedded-only request.

The protobuf and Abseil dependency is a recurring source of upstream user pain, visible in several open issues covering cross-compilation symbol errors, breakage against newer protobuf releases, Abseil duplicate-symbol link failures, and Abseil compile errors on newer GCC. These support the argument that OTLP/JSON should not require the protobuf runtime.

The curl precedent from this component does not transfer directly to protobuf: curl already sat behind a pure-virtual interface with an unchanged payload, whereas protobuf types are structural in the OTLP exporter. It does transfer to the factory-injection mechanism, which is why that part is copied closely.

OTLP permits exactly two encodings, binary protobuf and JSON. There is no third format to evaluate.