Protobuf-free OTLP/HTTP send path #117

Parent

Part of #102

Problem Statement

The protobuf-free mapping layer for all three signals is complete and proven: SDK spans, metrics and log records convert to OTLP/JSON through the JsonWriter token interface, and byte-identity equivalence tests assert the output matches what the protobuf path emits. Partial-success responses parse without protobuf too.

None of it can be reached. OtlpHttpClient is the only send path in the OTLP HTTP exporter, and it is typed on google::protobuf::Message — on the exported message, on the arena that owns the response, on the response message itself, and on the result callback. A protobuf-free payload cannot be handed to it. The response path compounds this by parsing replies with protobuf's JSON-to-message conversion.

The consequence is that #110, #111 and #112 each carry an acceptance criterion — "a trace / metric / log exporter produces OTLP/JSON without referencing protobuf" — that none of them can satisfy, for a reason that has nothing to do with the signal each one covers. It is a single shared blocker sitting underneath all three, and it is invisible in the issue graph: #111 is nominally blocked by #110, but implementing #111 would require solving the very thing that leaves #110 open.

The same gap explains why the equivalence seam named in #102's Testing Decisions — the no-send HTTP client's captured request body — is unreachable for the protobuf-free path. That seam is only reachable through an exporter, so it is not a second problem, it is this one.

Solution

Extract the transport core of OtlpHttpClient into a protobuf-free type, OtlpHttpTransport, and have OtlpHttpClient delegate to it.

Protobuf enters the transport in three narrow places: producing the request body, parsing the response into a caller-supplied message, and the arena plus response pointer carried alongside each in-flight session. Everything else — the session pool, session garbage collection, the concurrency counter and its condition variable, the cached parsed URI, retry policy, compression, ForceFlush, Shutdown and session release — is already protobuf-agnostic and moves across unchanged.

The extraction turns those three couplings into one: the caller hands the transport a finished request body and a completion callback that receives the raw response bytes. The protobuf client's own response parsing moves into the callback it supplies, and the arena and response message move into that callback's closure. The per-session bookkeeping struct loses both protobuf fields and becomes protobuf-free along with the rest.

OtlpHttpClient keeps its exact public signature and delegates. No call site changes, and no observable behavior changes — which the existing golden-body tests prove.

The new type lives in a build target that does not link protobuf, so the protobuf-free exporters added by #110, #111 and #112 can link the transport without the message runtime. That target boundary is what makes the protobuf-free claim checkable by reading the build file rather than by inspecting a binary.

User Stories

1. As a firmware developer, I want an OTLP/HTTP send path that accepts a serialized body rather than a protobuf message, so that the protobuf-free mapping I already have can actually reach the network.
2. As a firmware developer, I want the send path to link no protobuf symbols, so that the exporters built on it genuinely shed the runtime rather than merely avoiding it in my own source.
3. As a firmware developer, I want the transport to keep the session lifetime handling the existing client has, so that an export that outlives its caller does not leave a dangling handler.
4. As a firmware developer, I want ForceFlush and Shutdown to behave identically on the protobuf-free path, so that the SDK's exporter contract is honored whichever exporter I select.
5. As a firmware developer, I want response bodies handed back raw, so that partial-success parsing can be done by the JSON reader without protobuf.
6. As a firmware developer, I want the transport to work with a synchronous HTTP client implementation, so that a platform whose transport cannot spawn threads is still supported.
7. As a firmware developer, I want the transport to work with an asynchronous HTTP client implementation, so that the extraction does not quietly narrow what the library supports.
8. As a firmware developer, I want request compression handled by the transport rather than by each exporter, so that enabling it does not require touching mapping code.
9. As a firmware developer, I want the URL parse failure path to report the same result as before, so that a misconfigured endpoint fails the same way regardless of encoding.
10. As an upstream maintainer, I want OtlpHttpClient's public signature unchanged, so that this is additive and no downstream consumer needs to react.
11. As an upstream maintainer, I want the existing golden-body tests to pass byte-for-byte and unmodified, so that I can approve the extraction on evidence rather than on trust.
12. As an upstream maintainer, I want one transport implementation rather than two parallel ones, so that the protobuf and protobuf-free paths cannot drift in session handling or retry behavior.
13. As an upstream maintainer, I want the new type kept out of the public API surface, so that accepting this change does not commit the project to a new interface it must support indefinitely.
14. As an upstream maintainer, I want the result callback to follow the convention already used by the HTTP and gRPC clients, so that the new signature is recognizable rather than novel.
15. As an upstream maintainer, I want the request body type to be the one the HTTP client abstraction already consumes, so that no conversion is introduced at the seam.
16. As an upstream maintainer, I want the change to build under both CMake and Bazel, so that neither build system regresses.
17. As an upstream maintainer, I want a test that links the transport without protobuf, so that the dependency claim is enforced by the build rather than asserted in a description.
18. As an opentelemetry-cpp consumer, I want my existing OTLP HTTP exporter code to compile untouched, so that upgrading costs me nothing.
19. As an opentelemetry-cpp consumer using binary protobuf encoding, I want the wire bytes unchanged, so that my collector configuration keeps working.
20. As an opentelemetry-cpp consumer using JSON encoding, I want the wire bytes unchanged, so that my collector configuration keeps working.
21. As an opentelemetry-cpp consumer, I want partial-success responses still parsed into my protobuf response message, so that the read path behaves as it did.
22. As a maintainer of this component, I want this blocker tracked as its own issue, so that the three signal issues close on their own merits instead of one of them absorbing shared work.
23. As a maintainer of this component, I want the send path landed before the signal exporters, so that Protobuf-free OTLP/JSON trace exporter #110, Protobuf-free OTLP/JSON metric exporter #111 and Protobuf-free OTLP/JSON log record exporter #112 are unblocked simultaneously rather than in sequence.
24. As a maintainer of this component, I want the transport in the same repository as the mapping code it serves, so that an exporter and its mapping are not split across two projects.
25. As a maintainer of this component, I want the parent spec amended to record that the HTTP client is retyped, so that its Out of Scope section stops contradicting the work.
26. As a maintainer of this component, I want to decide myself when anything is submitted upstream, so that public contributions happen on my timing.

Implementation Decisions

Extract and delegate, not a parallel client. The transport core moves into a new type and the existing OtlpHttpClient delegates to it. A second, independently written client was rejected: it would leave two send paths free to drift in session handling, retry and shutdown semantics, which is the objection most likely to sink the change in review. Extraction also makes the no-behavior-change claim provable rather than argued, because the existing client keeps its own code.

Where it lands. Upstream, in the opentelemetry-cpp fork on branch otlp-json-no-protobuf, not in the downstream embedded component. All the mapping work this serves already lives there, so a downstream transport would split each exporter from its own mapping across two repositories. #102 already specifies the exporters as landing upstream, and its consumer-facing stories about substituting or excluding a JSON backend only pay off there. A downstream sender was considered and rejected on those grounds.

Name. OtlpHttpTransport. The *Client family — OtlpHttpClient, OtlpGrpcClient, OtlpFileClient — now reads as "the type that is typed on a message"; the extracted type is what remains after the message is removed, and "transport" names that. "Sender" was rejected as understating the session, flush and shutdown responsibilities the type retains.

Visibility. Internal, under the exporter's detail directory, alongside the existing default writer and reader factory headers. Only in-repo exporters need it, so keeping it out of the public API minimizes what upstream must commit to supporting and leaves the signature free to change without a breaking release. It is consumed across target boundaries, which is a slight departure from how detail headers are usually scoped; detail here signals "not API", not "not linkable". Promoting it later is easy, unpublishing it is not.

Export interface. The transport's export entry point takes the request body, the content type, a completion callback and the maximum number of running requests, and returns an export result. The callback receives an export result and the raw response body, and returns a bool. This follows the convention already established by OtlpHttpClient::Export and OtlpGrpcClient::DelegateAsyncExport, both of which carry results through a moved-in std::function returning bool. A polymorphic response-parser interface was rejected: polymorphic interfaces in this codebase are used for injecting transports and writers, never for per-request result carriage, and a closure already carries the state a parser object would have to hold. Templating on response type was rejected for the same reason plus the header cost.

Request body type. The body type the HTTP client abstraction already consumes, paired with a content-type string. The existing session-creation code already converges on exactly that pair before sending, for both the binary and JSON branches, so this cuts at a seam that is already there. Taking a string instead would push the JSON path's conversion inside the transport and force the binary path to convert out of the body type, inverting the current flow for no gain. The JSON path still copies the writer's output into the body type, as it does today.

Options. The transport takes the existing OtlpHttpClientOptions unchanged. The header only forward-declares protobuf, so it already compiles in a translation unit that does not link the runtime; the coupling is the build target, not the header. A protobuf-free subset struct was rejected as churn that reads badly in review. The accepted cost is that the JSON-converter options — the json-name flag and the bytes-mapping kind — ride along unread on the protobuf-free path.

Response handling moves to the caller. The response handler stops holding a response message pointer and instead invokes the completion callback with the raw body. The protobuf client supplies a callback that performs its existing JSON-to-message or parse-from-string conversion and then invokes the caller's callback. The arena and the response message move into that closure, which is what allows the per-session bookkeeping struct to drop both protobuf fields.

Build target split. A new library target holds the transport, and the existing OTLP HTTP client target depends on it. The protobuf-free exporters link the transport target instead of the client target. Bazel gets the parallel library. Keeping a single target and gating by preprocessor was rejected: the target boundary is what makes "does this link protobuf?" answerable by reading the build definition, and it mirrors the facade/backend target split already used for the HTTP client and the JSON writer.

Sequencing. This lands before #110, #111 and #112, all three of which are re-pointed to be blocked by it. Their remaining acceptance criteria — mapping, hex identifiers, string-encoded 64-bit values, equivalence against the protobuf path, shared resource and scope mapping — are already satisfied by work on the branch; the exporter criterion is the only one outstanding, and it is the same criterion in all three.

Parent spec amendment. #102's Out of Scope currently defers changes to the HTTP client, in the context of streaming the request body. That boundary is being reopened for a different reason — retyping, not streaming — so #102's Implementation Decisions gain a statement that the transport is extracted, and its Out of Scope keeps the streaming exclusion but no longer reads as excluding this.

Nothing is submitted upstream. Branch and description are prepared; opening a pull request is the maintainer's decision alone.

Testing Decisions

A good test here asserts on what leaves the process: the HTTP request that reaches the client abstraction, and what the caller observes coming back. Session counts, garbage-collection timing, the internal callback chain and the transport's own field layout are implementation detail and must not be asserted on. The extraction moves a large amount of code without intending to change any behavior, so the tests that matter are the ones that already exist.

Seam 1, existing and primary: the no-send HTTP client's captured request body. The OTLP HTTP trace, metric and log exporter tests already assert exact serialized bodies through this seam, and their JSON and binary integration cases feed a response back through it. Those response cases exercise precisely the code the extraction moves: per-session lifetime and the response handler's parse into the caller's message. The acceptance evidence for this change is that every one of those tests passes byte-for-byte with no modification, in both synchronous and asynchronous modes and in both encodings. No new test code is written at this seam.

Seam 2, new and minimal: a transport-only test that does not link protobuf. Seam 1 can pass in full while the extraction fails its actual purpose, because every caller reaching it is protobuf-typed; nothing there proves the transport is usable without protobuf until #110 exists. One new test target links the transport library and the no-send HTTP client only, pushes a hand-built body through the transport, and asserts the captured request body, the content type on the request, and that the completion callback receives the raw response bytes. It fails to link, loudly, if the target split is wrong.

This is achievable as stated: the no-send HTTP client target depends only on the ext library, the shared test-common library and the test framework, so it introduces no protobuf of its own. If that ever changes, the body assertions stay where they are and the no-protobuf claim moves to a separate link-only target rather than being dropped.

This is deliberately at the same altitude as seam 1 — a captured request body — not at the level of sessions, handlers or token streams. Two seams, both at the request-body level, is the intended total; no third is introduced.

Prior art. The golden-body tests added for the OTLP/JSON exporters are the pattern for asserting on a full serialized body rather than selected fields. The existing custom HTTP client tests are the pattern for a test that constructs an exporter against an injected client and inspects what it produced.

Not tested. No test asserts the token sequence, the number of sessions created, or the order of internal callbacks. No binary inspection is used as an acceptance criterion; the build target boundary and the linking test cover that claim instead.

Out of Scope

• The protobuf-free exporters themselves. Traces, metrics and logs are Protobuf-free OTLP/JSON trace exporter #110, Protobuf-free OTLP/JSON metric exporter #111 and Protobuf-free OTLP/JSON log record exporter #112 respectively, and they consume this.
• Any change to the mapping layer, the JsonWriter interface, the JsonReader, or partial-success parsing. All are already implemented on the branch.
• Streaming the request body. The writer still accumulates and the transport still receives a complete body; incremental flush remains deferred as Token-based JSON serialization interface for protobuf-free OTLP/JSON #102 states.
• Changing the wire format of either encoding. Byte-identical output is the acceptance condition, not a side effect.
• Retyping the gRPC or file clients. Both remain protobuf-typed.
• Opening a pull request against any upstream repository.

Further Notes

Context from the downstream consumer, for risk assessment only — none of it is in this repository or reachable from the implementation branch. The ESP-IDF HTTP client implementation that consumer supplies performs its send synchronously and fires the session callbacks before returning, deliberately, because spawning a thread for the request overflows the task stack. It also stores the retry policy without acting on it, and the component configures compression off. So on the target device the client's asynchronous session pool, retry and compression are all nominal today, and every export is already a blocking single-shot request. The extraction preserves all three capabilities regardless, because the transport keeps the code that implements them — but it means the change cannot regress behavior the device never exercised, which lowers the risk of the extraction considerably.