Repository navigation
Conversation
Stream each finished result to a per-worker on-disk JSONL shard (flushing per write) for local popen topologies, so a worker killed mid-run keeps its already-finished results and the size cap applies per record. The controller recovers shards in pytest_testnodedown; workeroutput carries only a completion sentinel. Remote/proxied gateways fall back to the inline transport. Closes the durability (#3) and granular size-cap (microsoft#4) gaps from the microsoft#73 review. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2 of 3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Stacked on microsoft#113 (
dev/nina-msft/xdist-shard-primitives).This intra-fork draft targets the PR-A branch so the diff shows only the wiring
added on top of the shard primitives. Once microsoft#113 merges into
main, this branch will be rebased ontomainand reopened againstmicrosoft:main.This PR wires the durable shard-transport primitives added in
microsoft#113 into the pytest plugin, closing the two durability gaps
@bashirpartovi asked us to "call out clearly" in the review of
microsoft#73:
inline
workeroutputtransport only flushed at a cleanpytest_sessionfinish.exceeded
--rampart-xdist-max-bytes, not just the one oversized transcript.What changed
For local
popentopologies (plainpytest -n N, or--txwhere every gatewayis
popen), each worker now streams every finished result to its own on-diskJSONL shard, flushing per write. The controller reads the shards back in
pytest_testnodedown.workeroutputcarries only a small completion sentinel(schema, expected record count, trial specs).
-x) keeps everyresult it had already flushed; the controller recovers them from the shard and
marks the run incomplete rather than losing the worker's whole contribution.
is dropped as a truncation marker while its siblings survive.
popengateway (ssh,socket,viaproxy)transparently falls back to the existing inline transport, since the controller
cannot read a worker-local shard file. Eligibility is gated by
shard_eligible.Controller/worker wiring:
pytest_configureprovisions the shared shard dir on aneligible controller;
pytest_configure_nodepushes it into each worker's input;the worker opens a
ShardWriter; the_rampart_collectfixture appends perresult;
pytest_sessionfinishcloses the shard and writes the sentinel;pytest_testnodedownrecovers and merges. Shard dirs are removed at run endunless
--rampart-keep-shardsis passed. Single-process and remote runs areunchanged.
Breaking changes
None. Behavior is identical for single-process runs and for remote/proxied xdist
topologies. Local
-nruns gain durability transparently; the only observabledifference is more granular incomplete-reason strings (per-record rather than
whole-payload).
Checklist
pre-commit run --all-filespasses