serial loop: adaptive receive hold + wedge fixes - #51
Merged
irvingouj@Devolutions (irvingoujAtDevolution) merged 7 commits intoJul 6, 2026
Conversation
Fake in-process WinRM server (honors per-Receive OperationTimeout) behind the HttpClient trait, driving the real serial loop via open_task_serial. Measures output latency (server-data-ready -> PipelineOutput event), input latency (Invoke -> Command on the wire), and kill-signal latency across REPL-roundtrip, steady-drip, quiet-burst, and Ctrl+C scenarios.
Replace the serial loop's client-side exponential backoff (which idled the connection during quiet periods) with an adaptive server-side Receive hold: the loop always keeps a Receive parked server-side so output is pushed back near-instantly. - Plumb an optional per-receive hold (OperationTimeout) through the fire_receive chain down to WsMan::invoke_with_operation_timeout. None keeps the config default (parallel loop and fire_active_receive unchanged). - Scheduler computes hold = 0.25s x 2^(empty polls), capped at 1.0s; progress resets the streak. Cancel caps the hold at 0.5s; recent user activity (within 3s) caps all holds at 0.25s. Empty polls no longer schedule a client-side sleep; finished targets are still never polled. - SessionCore records user activity on every user op and host response (idle and buffered paths) and passes the computed hold into each Receive.
WSMAN_CMDSHELL_OPTION_KEEPALIVE contains 'PT', so the fake server parsed every Receive hold as the 250ms fallback and never honored the adaptive holds under test. Baseline numbers are unaffected (baseline always requests PT0.250S). Also log the promoted hold_secs in the serial loop trace.
A Ctrl+C during Read-Host/Prompt could leave the browser host-call promise pending forever: the Rust bridge never got a result to submit, so the serial core kept host_call_state = Waiting, which gates all Receive promotion — the next command produced no output and the terminal wedged. Serial loop: bound the wait on an outstanding host call to 5s (mirrors the parallel loop's host-response timeout). On expiry, clear the waiting state and issue CancelHostCall so the server-side pipeline unblocks and Receive promotion resumes. The deadline only applies while a host call is outstanding. Web: on Ctrl+C during a host call, reject the handler's pending input so the Rust bridge submits a response instead of dropping it silently.
A single recoverable error used to tear down the whole serial session, leaving the web terminal dark with no explanation. Serial loop: a transport drop (e.g. gateway WS reset) on an in-flight Receive is now tolerated — the long-poll is idempotent, so polling is re-armed from the active streams. Drops on Sends stay fatal (server-side effect unknown), and a cap of 3 consecutive Receive failures still terminates a dead link. Runspace pool: a non-timeout SOAP fault answering a Receive for a pipeline that is already Stopping (post Ctrl+C teardown) is treated as that pipeline finishing instead of a fatal fault. Faults unrelated to a stopping pipeline stay fatal. Web: surface a mid-session error/close with a visible "Connection lost" line and reset the running flag before closing, so the user sees why input stopped.
…of cancelling A 5s timeout that cancelled the host call server-side broke legitimate slow interaction — a human answering Read-Host routinely takes longer. Past the grace window the loop now just resumes Receive polling while the call stays answerable. Orphaned calls (Ctrl+C during a prompt) are cleared when their pipeline finishes, and responses for calls no longer outstanding are dropped instead of being submitted against a dead server-side call.
irvingouj@Devolutions (irvingoujAtDevolution)
merged commit Jul 6, 2026
ea9b6c8
into
master
3 checks passed
irvingouj@Devolutions (irvingoujAtDevolution)
deleted the
bench/serial-latency
branch
July 6, 2026 23:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The serial (single-connection) loop used a 0.25s Receive poll with a client-side backoff that grows to 5s, so output from a quiet pipeline could sit on the server for up to ~5s before we fetched it. That's the "terminal feels dead" everyone hit in the web terminal.
This replaces the client-side sleep with an adaptive server-side hold: the Receive's OperationTimeout grows 0.25s → 1s on empty polls, so there's always a Receive parked on the server and output comes back the moment it exists. Holds cap at 0.25s for ~3s after any user input so typing stays snappy.
Measured with the new deterministic bench (
cargo run --release -p ironposh-async --example serial_latency_bench, fake WinRM server that honors per-request OperationTimeout): quiet-pipeline output latency went 3086ms → 7ms median (max 4.6s → 16ms), REPL roundtrips unchanged, Ctrl+C signal now waits out the in-flight hold (~420ms, bounded 1s).Also fixes two wedges found while investigating flakiness:
Tested: transport×auth×sealing matrix 10/10 against IT-HELP-DC, serial PTY e2e green, WebTerminal Playwright suite green with locally-linked wasm (the Read-Host and Ctrl+C specs specifically). Bumps ironposh-web to 0.6.0 for release.