Skip to content

[WIP] Add node-local delayed resume to ProcessorInbox#2471

Draft
lquerel wants to merge 31 commits intoopen-telemetry:mainfrom
lquerel:feature/local-delayed-resume
Draft

[WIP] Add node-local delayed resume to ProcessorInbox#2471
lquerel wants to merge 31 commits intoopen-telemetry:mainfrom
lquerel:feature/local-delayed-resume

Conversation

@lquerel
Copy link
Copy Markdown
Contributor

@lquerel lquerel commented Mar 31, 2026

Not ready for review. #2470 must be merged first to reduce the scope of this PR

Change Summary

Add node-local delayed pdata resume for processors via requeue_later(...) and integrate it into ProcessorInbox.

This PR extends the node-local scheduler work from the previous PR by adding a bounded local delayed-resume reservoir for pdata. Due delayed resumes are delivered back through ProcessorInbox, and retry_processor is migrated off the old global delayed-data path.

This is the next step in the redesign tracked in #2465. After PR3 separated wakeup-style local scheduling from delayed pdata handling, this PR addresses the delayed pdata side directly.

The goal is to move retry-style deferred work out of the global runtime path and into node-local inbox orchestration, while keeping the old global delayed-data plumbing in place temporarily for compatibility. That keeps this PR focused on local delayed resume and retry migration only; the cleanup and removal of the old global path happens in the next PR.

What issue does this PR close?

How are these changes tested?

  • Added/updated engine tests covering:
    • local delayed resume scheduling
    • bounded delayed-resume delivery through ProcessorInbox
    • shutdown behavior for pending delayed resumes
  • Added/updated retry processor tests covering:
    • retry rescheduling through requeue_later(...)
    • retry behavior after delayed local resume
  • cargo xtask check

Are there any user-facing changes?

No user-facing changes.

This PR is an internal runtime change only.

lquerel added 30 commits March 26, 2026 23:07
# Conflicts:
#	rust/otap-dataflow/crates/engine/src/message.rs
#	rust/otap-dataflow/crates/engine/src/processor.rs
# Conflicts:
#	rust/otap-dataflow/crates/engine/src/message.rs
#	rust/otap-dataflow/crates/engine/src/processor.rs
@github-actions github-actions bot added the rust Pull requests that update Rust code label Mar 31, 2026
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 31, 2026

Codecov Report

❌ Patch coverage is 77.94261% with 907 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.21%. Comparing base (5ef4ab5) to head (374af82).
⚠️ Report is 11 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2471      +/-   ##
==========================================
- Coverage   88.35%   88.21%   -0.14%     
==========================================
  Files         603      608       +5     
  Lines      212379   216476    +4097     
==========================================
+ Hits       187646   190971    +3325     
- Misses      24207    24979     +772     
  Partials      526      526              
Components Coverage Δ
otap-dataflow 90.14% <77.94%> (-0.24%) ⬇️
query_abstraction 80.61% <ø> (ø)
query_engine 90.74% <ø> (ø)
syslog_cef_receivers ∅ <ø> (∅)
otel-arrow-go 52.44% <ø> (ø)
quiver 91.94% <98.90%> (+0.03%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

rust Pull requests that update Rust code

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant