Native OpenTelemetry tracing — foundation + ADR (Phase 1a)

[wbarnha]

Description

Groundwork for migrating Faust's built-in distributed tracing off the archived / EOL OpenTracing API to native OpenTelemetry, while keeping the full processing-pipeline span tree (agents, streams, table recovery, assignor, timers, Crontabs, rebalance).

This is Phase 1a — foundation only. It adds an architecture decision record and a self-contained, unit-tested native tracing core. No existing call sites are rewired yet, so the running OpenTracing layer is completely unaffected. Opening as a draft to get agreement on the approach before the larger Phase 1b rewire.

Why native OTel (and why not an instrumentor)

• OpenTracing is archived/EOL; the OpenTelemetry OpenTracing shim was formally deprecated on 2026-03-19 (removal ≥ Mar 2027). The shim bridge (Add optional faust[opentelemetry] extra + docs for OpenTelemetry via the OpenTracing shim #688) is a valid interim, not a destination.
• There is no opentelemetry-instrumentation-faust package — not on PyPI, not in opentelemetry-python-contrib, nothing community-maintained. The transport-level instrumentors (opentelemetry-instrumentation-aiokafka / -confluent-kafka, both beta) only patch the raw Kafka clients and cannot reproduce Faust's agent/stream/recovery/rebalance spans — they complement, they can't replace.
• A library should depend only on opentelemetry-api (no SDK); calls are cheap no-ops until the app registers a TracerProvider. opentelemetry.context is contextvars-based, so Faust's current-span propagation maps 1:1, including across await.

What's in this PR

• docs/adr/0001-native-opentelemetry-tracing.md — the decision record: options considered, semantic-convention coexistence, native re-expression of the two non-portable OpenTracing hacks, and the phased migration plan.
• faust/utils/otel_tracing.py — api-only tracing primitives that degrade to no-ops when OpenTelemetry is absent:
  • tracer resolution against the global provider;
  • contextvar current-span helpers;
  • deterministic_parent_context() — native replacement for the old span.context.trace_id = murmur2(...) mutation (OTel span contexts are immutable): a NonRecordingSpan parent carries the seeded trace id and the real rebalance span inherits it;
  • PendingSpan — native replacement for the _transform_span_lazy span.finish monkeypatch: defer span creation until the Kafka generation is known, or cancel;
  • semconv-coexistence helpers — every span carries both the legacy kafka-* tags (deprecated) and the OpenTelemetry messaging.* attributes + SpanKind;
  • W3C traceparent header inject/extract.
• tests/unit/utils/test_otel_tracing.py — 15 tests covering every primitive against an in-memory SDK exporter (both hacks validated end-to-end); the no-OTel degradation path is also exercised.
• requirements/extras/opentelemetry.txt + setup.py — new faust[opentelemetry] extra (opentelemetry-api/-sdk).

Semantic-convention coexistence

Adopted additively: span names stay legacy by default (consume-from-{topic}, job-{topic}, produce-to-{topic}) and are marked deprecated, while messaging.* attributes + SpanKind coexist on the same span. A future opt-in flips names to semconv ({topic} process, …).

Migration plan (see ADR)

• Phase 0 (done): opentracing optional (Make opentracing an optional dependency (faust[opentracing]) #686) + shim bridge (Add optional faust[opentelemetry] extra + docs for OpenTelemetry via the OpenTracing shim #688) — non-breaking.
• Phase 1a (this PR): native tracing core + ADR, nothing rewired.
• Phase 1b: convert the 12 modules; redefine TracerT in OTel terms; ship a legacy adapter wrapping existing OpenTracing app.tracer implementations via the shim for one major cycle. Targets a major version — app.tracer/TracerT is a breaking signature change.
• Phase 2: drop opentracing from core requirements and remove the adapter.

Notes for reviewers

• Nothing here changes runtime behavior — the new module isn't imported by Faust core yet.
• The breaking-change and release-timing decisions (Phase 1b) are intentionally left for maintainers before that work lands.

🤖 Generated with Claude Code

---

Generated by Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 94.33962% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.14%. Comparing base (f3019aa) to head (dd9eeda).

Files with missing lines	Patch %	Lines
faust/utils/otel_tracing.py	94.33%	2 Missing and 4 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master     #689    +/-   ##
========================================
  Coverage   94.14%   94.14%            
========================================
  Files         104      105     +1     
  Lines       11136    11242   +106     
  Branches     1201     1215    +14     
========================================
+ Hits        10484    10584   +100     
- Misses        551      553     +2     
- Partials      101      105     +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.