Repository navigation
[BUG] otel_traces GenAI enrichment doesn't propagate gen_ai.operation.name to root, so agent traces with an HTTP root are hidden in Agent Traces #7191
Description
Describe the bug
GenAI root enrichment in otel_traces (#6548) copies four string attributes to the root span, and gen_ai.operation.name isn't one of them. The OpenSearch Dashboards Agent Traces view only lists a trace if its root has gen_ai.operation.name, so any agent running behind an HTTP handler is missing from the Traces and Sessions tabs. Its spans still show up in the Spans tab.
Root cause
Propagated keys: PROPAGATED_STRING_KEYS is gen_ai.system, gen_ai.provider.name, gen_ai.agent.name and gen_ai.request.model (GenAiEnrichmentHelper.java#L32-L42). It's been that way since #6548. The README says gen_ai.operation.name is propagated too (README.md#L26), so docs and code disagree. The only operation.name assertions in GenAiEnrichmentTest cover vendor value normalization, which is why this wasn't caught.
What the UI filters on: the Traces tab uses prepareRootSpansQuery, which appends where parentSpanId = "" AND isnotnull(attributes.gen_ai.operation.name) (tab_queries.ts#L12-L18). Sessions reuses the same query (register_tabs.ts#L32-L45, L68-L76). With an HTTP root, the root has an empty parent but no operation name, and the spans that have an operation name all have a parent. Nothing matches.
To reproduce
A Strands agent served by FastAPI, instrumented with OTel, sent through otel_traces. One trace, trimmed:
POST /runs parentSpanId=""
└── invoke_agent observability gen_ai.operation.name=invoke_agent
├── execute_event_loop_cycle (5 cycles)
│ ├── chat gen_ai.operation.name=chat
│ └── execute_tool ... gen_ai.operation.name=execute_tool
└── ...
Root span as indexed:
"name": "POST /runs",
"parentSpanId": "",
"attributes": {
"gen_ai": {
"request": { "model": "us.anthropic.claude-sonnet-4-6" },
"agent": { "name": "observability" },
"system": "strands-agents",
"provider": { "name": "aws" }
}
}The root has no gen_ai.operation.name, while 18 of its descendants do.
Expected behavior
The root carries gen_ai.operation.name, so the trace lists in Agent Traces regardless of what the root span is.
Proposed fix
- Operation name: add
gen_ai.operation.nameto the propagated keys, taken from the outermost GenAI span (hereinvoke_agent). First-child-wins could land onchatorexecute_tooldepending on span order in the batch, which would be wrong for the trace as a whole. - Tests and docs: add a test with a non-GenAI root, and make the README match what's propagated.
Environment
- Data Prepper with feat: Add GenAI agent trace enrichment to otel_traces processor #6548 (2.14.0+). Code links are
mainat7bae96e. - OpenSearch Dashboards links are
mainat21711129. I didn't confirm the UI build the trace was viewed on, but the behavior matches that query. - Source: Strands Agents (Python), OTel Python SDK 1.45.1,
opentelemetry.instrumentation.fastapi0.66b1.
Additional context
Enrichment only sees children that land in the same batch as the root. A request that outlives trace_flush_interval (default 180s) gets its children flushed as orphans before the root arrives, so the root isn't enriched at all. That's separate from this bug, but long agent runs hit it, so it's worth knowing when reproducing.
Activity
ps48 commentedon Oct 15, 2026
Thanks for the detailed report. Confirmed on
main:PROPAGATED_STRING_KEYSdoesn't includegen_ai.operation.name, and the README says it does.Before fixing it as proposed, I looked at how the GenAI semantic conventions and other tools handle agent traces with a non-GenAI root, and I'd like to suggest a slightly different fix.
Why not copy
gen_ai.operation.nameto the rootgen_ai.operation.name, soPOST /runswould show up there as an extra Agent span.POST /runswith empty input and output, because the prompts and responses are on theinvoke_agentspan.trace_flush_intervalflushes the children first, so the root never gets enriched.What the spec and other tools do
invoke_agentspans (kindINTERNAL) are expected to sit under whatever span is active, such as an HTTPSERVERspan. The closest idea is the outermost agent or workflow: #385 (gen_ai.main_agent.name, open), which replaced #355 (gen_ai.root_operation.name, closed).AGENT"so it is kept as the trace root".So the common idea is that the outermost GenAI span is the agent trace, not the OTel root.
Proposal
otel_traces, mark the outermost GenAI span in each trace (a span withgen_ai.operation.nameand no GenAI ancestor) withderived.gen_ai.is_root = true, and aggregate token usage onto it.gen_ai.agent.name,gen_ai.request.modeland the other keys takes values from the outermost GenAI span instead of whichever child comes first.(parentSpanId = "" AND gen_ai.operation.name exists) OR derived.gen_ai.is_root = true. The row is then the agent span, with the right name, kind, input/output and latency. The Spans tab and Sessions are unchanged.mark_agent_root()helper for cases the structure can't settle, for example a service's agent entry point. Users without the SDK can set the attribute themselves.Known trade-offs
traceId, preferring the true root.Happy to put up the PRs if this direction sounds good. Feedback on the attribute name is especially welcome.
ps48 commentedon Oct 15, 2026
Draft PRs for the approach above:
derived.gen_ai.is_rootand aggregates its tokens onto it. Root propagation now uses the outer agent.mark_agent_root()/markAgentRoot().I tested them together end to end on observability-stack, using a FastAPI app with a real HTTP
SERVERroot:invoke_agentis listed with its input, output and tokens, and the tokens are not double counted on the root.trace_flush_interval, after its children were flushed as orphans, is still listed through the marked agent span.feat: add markAgentRoot() to mark a trace's agent root span (#88)
feat: add mark_agent_root() to mark a trace's agent root span (#48)
[Agent Traces] List agents behind an HTTP handler by their marked AI …
Mark the outermost GenAI span of traces with a non-GenAI root (opense…