docs: Add extension system architecture document

[gouslu]

Change Summary

Adds the Phase 1 architecture document for the extension system (rust/otap-dataflow/docs/extension-system-architecture.md), describing how the extension system proposal is implemented.

A working PoC is available on the PoC branch.

What this document covers

The document maps each proposal requirement to a concrete Phase 1 implementation approach and introduces two additions beyond the proposal:

• Active/Passive lifecycle distinction — type-level enforcement via Active<E> / Passive<E> wrappers. Passive extensions provide capabilities without spawning a task or allocating control channels.
• Shared-to-local transparent fallback — a shared-only extension can serve local consumers automatically via a SharedAsLocal adapter. Extension authors who need lock-free local performance can still opt in with a dedicated local variant.

It also covers:

• Ownership model — Rc for local (no locks), Box<Clone+Send> with Arc-wrapped fields for shared (similar to tokio/axum patterns)
• #[capability] proc macro — generates local/shared trait variants, sealed impls, registration structs, and coercion functions from a single trait definition
• Type-safe capability resolution — require_local() / require_shared() with compile-time sealing and 4-step binding validation
• Core types — ExtensionWrapper, ExtensionFactory, CapabilityRegistry, builder pattern, and lifecycle management
• Configuration — YAML schema for extensions section and capabilities bindings on nodes
• Pipeline lifecycle — extension startup ordering, shutdown sequencing, unused variant cleanup
• Future vision — hierarchical scopes (Phase 2), WASM extensions (Phase 3)
• Phase 1 migration plan — 8 incremental PRs

Phase 1 — Migration Plan (PRs 1-8)

PRs 1-4 (Infrastructure): Config parsing (extensions section + capabilities bindings), core engine types (ExtensionFactory, ExtensionWrapper, Active/Passive wrappers), capability system (#[capability] proc macro, CapabilityRegistry, resolve_bindings()), and first capability definitions (BearerTokenProvider, KeyValueStore).

PRs 5-6 (Runtime wiring): Extension spawning + shutdown ordering in RuntimePipeline, and adding &Capabilities parameter to all node factories.

PRs 7-8 (First real usage): Azure Identity Auth extension (active, shared-only with local fallback, provides BearerTokenProvider) and Azure Monitor Exporter migration to consume it via require_local().

PoC Map

Engine (core extension/capability infrastructure):

• engine/src/extension.rs — ExtensionWrapper, builder, Active/Passive, dual control channels
• engine/src/capability/registry.rs — CapabilityRegistry, Capabilities, extension_capabilities! macro, resolve_bindings()
• engine/src/capability/bearer_token_provider.rs — BearerToken, Secret, #[capability] invocation
• engine/src/capability/key_value_store.rs — #[capability] invocation
• engine-macros/src/capability.rs — #[capability] proc macro implementation
• engine/src/runtime_pipeline.rs — Extension spawning + shutdown ordering

Extensions (sample implementations):

• azure_identity_auth_extension/ — Active, shared-only with local fallback, provides BearerTokenProvider
• sample_shared_key_value_store/ — Passive, shared-only, provides KeyValueStore
• sample_optimized_key_value_store/ — Passive, dual-type (local + shared)

Consumers (extensions used by nodes):

• azure_monitor_exporter/mod.rs — require_local::<BearerTokenProvider>() in factory
• azure_monitor_exporter/exporter.rs — token_rx.changed() event loop integration
• shared_kv_exporter/mod.rs — require_shared::<KeyValueStore>() in factory

What issue does this PR close?

• N/A — documentation only

How are these changes tested?

Documentation only — no code changes.

Are there any user-facing changes?

No. This is an internal architecture document for contributors.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.36%. Comparing base (a1fa843) to head (eebd07e).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2510   +/-   ##
=======================================
  Coverage   88.36%   88.36%           
=======================================
  Files         614      614           
  Lines      223275   223275           
=======================================
+ Hits       197293   197301    +8     
+ Misses      25458    25450    -8     
  Partials      524      524           

Components	Coverage Δ
otap-dataflow	90.26% <ø> (+<0.01%)	⬆️
query_abstraction	80.61% <ø> (ø)
query_engine	90.74% <ø> (ø)
syslog_cef_receivers	∅ <ø> (∅)
otel-arrow-go	52.45% <ø> (ø)
quiver	91.92% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[lquerel]

@gouslu To help reviewers of this document, should we:

1. Start by referencing the initial proposal to establish context and vision: https://github.com/open-telemetry/otel-arrow/pull/rust/otap-dataflow/docs/extensions.md

2. Position the current document in this PR relative to that proposal, clearly stating what this new document clarifies, adds, or changes, and highlighting any differences

3. Avoid referring to a “current architecture” in the document, since extensions are not yet implemented in main. As written, it gives the impression that extensions are already supported, which is not the case

[gouslu]

@gouslu To help reviewers of this document, should we:

1. Start by referencing the initial proposal to establish context and vision: https://github.com/open-telemetry/otel-arrow/pull/rust/otap-dataflow/docs/extensions.md
2. Position the current document in this PR relative to that proposal, clearly stating what this new document clarifies, adds, or changes, and highlighting any differences
3. Avoid referring to a “current architecture” in the document, since extensions are not yet implemented in main. As written, it gives the impression that extensions are already supported, which is not the case

Done, please let me know if these changes address all these comments.

[jmacd]

This is great! Thank you @gouslu.

I find the document matches my expectations and the sequence of PRs well described. Here is how I recommend we proceed. First, we'll wait for other reviewers to review and approve this. Then, I would be glad to see PR1 from your list, since it is well aligned with @lquerel's docs/extensions.md (which by the way, I recommend we rename to extensions-requirements.md).

[gouslu]

PR 1 - feat(config): extensions section, capabilities bindings, and config-level validation: #2538

[jmacd]

I will merge this PR if no one else comments. I requested #2538 as evidence that we are ready to move forward.

[lalitb]

LGTM. Nit comments - more to fix the document, don't think it would be blocker for implementation.