feat(otap-dataflow): expose startup helpers as a public library module

[lalitb]

Fixes: #2462, #2463

Summary

• Moves reusable startup functions (validate_engine_components, apply_cli_overrides, system_info) from src/main.rs into a new otap_df_controller::startup module. This lets custom distribution binaries share the same validation, override, and diagnostics logic without copying code that may drift across releases.

• Adds run_forever_with_observer and run_till_shutdown_with_observer to Controller, exposing the ObservedStateHandle via a callback so that in-process integrations can read pipeline liveness and health with zero HTTP overhead.

• src/main.rs becomes a thin wrapper over these library calls - no behavioral change for existing users. A new "Embedding in Custom Distributions" section is added to the README, and a runnable examples/custom_collector.rs demonstrates the full library embedding pattern end-to-end, including the observer API with an opt-in --poll-status flag for periodic health snapshots.

[codecov]

Codecov Report

❌ Patch coverage is 78.23344% with 69 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.35%. Comparing base (c66317f) to head (ebe25d1).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2505      +/-   ##
==========================================
- Coverage   88.36%   88.35%   -0.02%     
==========================================
  Files         614      615       +1     
  Lines      223280   223318      +38     
==========================================
+ Hits       197311   197314       +3     
- Misses      25445    25480      +35     
  Partials      524      524              

Components	Coverage Δ
otap-dataflow	90.24% <78.23%> (-0.02%)	⬇️
query_abstraction	80.61% <ø> (ø)
query_engine	90.74% <ø> (ø)
syslog_cef_receivers	∅ <ø> (∅)
otel-arrow-go	52.45% <ø> (ø)
quiver	91.92% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[lquerel]

LGTM. I just left a suggestion for a future improvement.

[Copilot]

Pull request overview

This PR makes the OTAP dataflow controller easier to embed as a library by moving common binary startup logic into a public otap_df_controller::startup module and by exposing an in-process ObservedStateHandle to callers via new Controller observer-based run methods.

Changes:

• Extracts reusable startup helpers (CLI overrides, component validation, system diagnostics) from src/main.rs into otap_df_controller::startup.
• Adds run_forever_with_observer / run_till_shutdown_with_observer to provide an ObservedStateHandle callback for zero-HTTP health/state access.
• Adds embedding documentation and a runnable examples/custom_collector.rs showing the full pattern end-to-end.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
rust/otap-dataflow/src/main.rs	Switches binary entrypoint to call the new startup module helpers.
rust/otap-dataflow/README.md	Documents embedding pattern and points to the runnable example.
rust/otap-dataflow/examples/custom_collector.rs	New example binary demonstrating embedding + observer handle polling.
rust/otap-dataflow/crates/controller/src/startup.rs	New public module implementing validation/override/system-info helpers (and tests).
rust/otap-dataflow/crates/controller/src/lib.rs	Exposes startup module and adds observer-enabled controller run methods.
rust/otap-dataflow/crates/controller/Cargo.toml	Adds sysinfo dependency for startup::system_info.
rust/otap-dataflow/Cargo.toml	Removes sysinfo dependency from the top-level binary crate.