Repository navigation
Make protobuf optional for OTLP/HTTP JSON trace export #4755
Description
Motivation
Some of our legacy C++ applications still use protobuf 2.x and cannot upgrade. We considered OTLP/HTTP JSON to avoid introducing an incompatible protobuf runtime, but found that the official HTTP exporter still requires protobuf when exporting JSON.
We also measured significant client CPU overhead: in our test cases, official HTTP/JSON used 3.7–4.0× the complete client CPU of official gRPC. Could JSON-only trace export avoid intermediate protobuf messages and the JSON DOM?
Where the cost comes from
Both official exporters were built unmodified from upstream main, pinned to 3ee38aaa (1.31.0-dev, fetched on October 30, 2026).
Official HTTP/JSON:
Protobuf-backed OtlpRecordable
→ Protobuf request
→ Reflection → JSON DOM
→ JSON string → HTTP
| Official HTTP/JSON phase | Small spans | Large spans |
|---|---|---|
| Populate protobuf-backed recordables | 8.4% | 8.1% |
| Assemble protobuf request, including span copies | 3.0% | 3.9% |
| Protobuf reflection → JSON DOM | 32.8% | 25.7% |
| JSON DOM → string | 21.9% | 30.6% |
| Release JSON DOM | 23.4% | 19.3% |
| Other cleanup and HTTP body copy | 10.5% | 12.4% |
DOM construction, serialization, and destruction account for 76–78% of the cost.
Results
We made actual exporter calls over loopback and measured complete client process CPU.
| Exporter | Small spans, µs/span | Large spans, µs/span |
|---|---|---|
| Official HTTP/JSON | 11.380 | 35.173 |
| Official gRPC | 2.875 | 9.410 |
| Our direct JSON implementation | 2.107 | 7.352 |
We built a custom HTTP/JSON exporter for our needs that writes JSON directly without protobuf, reducing intermediate processing overhead. It writes OTLP fields directly from SpanData, without protobuf messages or a JSON DOM, while preserving resource/scope grouping.
Compared with official HTTP/JSON, it reduced client CPU by 79–82%. The final HTTP body copy remains.
Proposal
Would maintainers consider a protobuf-runtime-free export path for applications using only OTLP/HTTP JSON, writing directly from SDK data into a JSON buffer while reusing the existing HTTP transport logic?
Activity
mateenali66 commentedon Oct 31, 2026
#4556 (the JsonWriter from #2541) does not get these apps off protobuf. at its head
ConvertGenericMessageToJsonstill walks the protobuf request by reflection into the writer (otlp_http_client.cc:739). a streaming writer would drop your DOM build, dump and free rows, but the recordables, the request copy and the reflection walk stay, and with the default nlohmann writer nothing changes.protobuf is also in the client itself on main.
OtlpHttpClient::Exporttakes agoogle::protobuf::Message(otlp_http_client.h:198), the JSON response is parsed withJsonStringToMessage(otlp_http_client.cc:142), and the OTLP types are proto3, which 2.x cannot compile (the minimum is 3.21.6). so a protobuf-free path needs its own recordable and a body-level entry into the client, which is closer to a second exporter than a change to this one.is traces enough for these apps, or would logs and metrics need the same path?
looks like
enhancement,area:exporter:otlp.yswdqz commentedon Oct 31, 2026
@mateenali66 Thanks for the detailed explanation. I agree that this would be closer to a new exporter.
Our main concern is the CPU cost of the current HTTP/JSON exporter. In our benchmarks, it used approximately 3.7–4.0× the total client process CPU of the official gRPC exporter for the same trace workloads.
For JSON-only export, it is not clear to us that using protobuf as an intermediate representation justifies this cost. Building protobuf messages, walking them through reflection to construct a JSON DOM, and then serializing and destroying that DOM introduces substantial overhead. Our direct JSON implementation shows that much of this processing can be avoided.
So far, we have only implemented a trace exporter for our internal services. Logs and metrics appear to have similar conversion overhead through the shared HTTP client, although we have not benchmarked them.
Would the community be open to a contribution adding a separate, opt-in OTLP/HTTP JSON trace exporter that writes directly from SDK data, avoiding intermediate protobuf messages and the JSON DOM?
marcalff commentedon Oct 31, 2026
@yswdqz
A custom
HTTP JSONexporter is likely to be more efficient compared to aOTLP/HTTP JSONexporter, and can be implemented directly, this sounds feasible.However, in this case, who / what defines the exact message format in JSON ?
In case of
OTLP/HTTP JSON, this comes from the definition of OTLP messages, which are defined in the https://github.com/open-telemetry/opentelemetry-proto repository.These messages are defined in protobuf format, so protobuf is to be involved at some point.
For example, https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto
Implementing a JSON export manually is a problem for maintenance, when the code drifts from the official OTLP specification.
Out of curiosity, have you profiled
OTLP/HTTPusing binary messages instead of JSON, to see how it compares with regards to performances ?yswdqz commentedon Nov 1, 2026
@marcalff Thanks for pointing this out. After reconsidering, we realize we underestimated the maintenance cost: the OTLP proto definitions evolve, and a handwritten encoder would need to keep up. We did not give this enough consideration in our original proposal.
We also considered generating direct-serialization C++ code from the .proto definitions to avoid a libprotobuf dependency and reduce intermediate conversion overhead. However, we have not found an existing tool that fits these requirements. Maintaining our own generator would introduce additional complexity, so we cannot yet present this as a well-established solution.