Repository navigation
fix(tooling): replace the recursive cargo mutants alias with .cargo/mutants.toml - #1727
Merged
Merged
Conversation
…utants.toml The `mutants` alias in `.cargo/config.toml` expanded to a command of its own name, which cargo has rejected as an unresolvable recursive definition since 1.56, so the alias never worked. The same alias broke the scheduled mutation-test job, which reported success regardless: its `cargo mutants … | tee` step ran under GitHub's default `bash -e` with no `pipefail`, so the step took `tee`'s exit status. Two further defects sat behind that one. The issue-filing step read `target/mutants/missed.txt`, a path cargo-mutants never writes (it nests `mutants.out/` under the output directory), and `-f src/metrics/` selected no file at all, since a glob containing `/` is matched against the whole path. Remove the alias and move the shareable settings into a new `.cargo/mutants.toml`: the timeout floor, the output directory, and a `test_package` set so a checker or getter mutant is judged by the root crate's metric tests rather than only the ast crate's. The workflow sets `pipefail`, selects `src/metrics/**/*.rs`, reads the nested result path, drops `-j 2`, which conflicts with `--in-place`, and lowers `timeout-minutes` from 720 to the 360 that hosted runners enforce regardless. Rewrite the local-run and triage guidance in docs/development/mutation_testing.md and add lesson 104. The repaired glob selects 3812 mutants where the broken one selected 196, and a serial in-place run over them is not expected to fit six hours; a timed-out job is cancelled rather than failed and so files no issue. Sharding is #1726. Verified with cargo-mutants 27.1.0: `--list` with the workflow's exact arguments selects 3812 mutants within the three scopes, a single-file `-f` is not widened by the config, an unmatched `--examine-re` run confirms the `mutants.out/` layout, a harness running both `run:` bodies under plain `bash -e` with stub `cargo` and `gh` fails three of five cases when `pipefail` is removed, and a three-mutant smoke run through the config shows each mutant testing both packages and the tree clean afterwards. Evidence under ~/bca-evidence/1710/. Fixes #1710
…ard summaries The selection #1710 repaired is 3812 mutants, run serially because `--in-place` rules out `-j`, against the six hours at which GitHub cancels a hosted-runner job. A cancelled step is not a `failure()`, so a timed-out run would have filed nothing: the silent-pass shape #1710 removed, back through the timeout. Split the run into three jobs. `prepare` declares the shard list once as a JSON output. `mutants` runs one matrix job per shard with `--shard k/n --sharding round-robin` (slice would put every ast-crate mutant, which rebuilds both crates, in the last two shards) and a pinned cargo-mutants, since the partition is only disjoint if every shard generates the same list. The step captures cargo-mutants' exit status, has `utils/mutation-report.py summarize` join it to the result files into one small summary, uploaded first and separately, then exits with the status. A shard that is cancelled or crashes never reaches that line and uploads no summary. `report` runs after the matrix whatever its outcome and has `mutation-report.py report` aggregate the summaries that arrived, naming any shard without a well-formed one rather than counting it, so a timed-out shard cannot pass for a clean one. The body is uploaded as an artifact before anything talks to GitHub, then filed under a fixed title; an open issue by that title gets a comment, not a sibling. The summarize half refuses the one contradiction that would mean a masked exit status (exit 0 beside recorded escapes); the report half has two states per shard, usable or named-with-reason, so anything that stops a summary being written fails toward red by construction. Both are self-tested by `make mutation-report-test`, wired into `make pre-commit` and `make ci`, so the result handling is checked on every pull request rather than once a quarter. The step timeout sits below the job's so a long shard is stopped with time to upload, and rust-cache saves on failure, since every shard exits non-zero while escapes remain. Verified with cargo-mutants 27.1.0: the eight round-robin shards are pairwise disjoint and their union is the full 3812-line list; the shard step run locally with an unmatched `--examine-re` produces a well-formed summary; and perturbing either the masked-exit refusal or the absent-shard branch fails exactly the tests that guard them. Evidence under ~/bca-evidence/1726/. The per-language `checker/` and `getter/` modules (+1135 mutants) are deliberately not added here, so the first real run measures the plumbing alone; that widening is its own follow-up. Fixes #1726
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1710. Fixes #1726. Deferred scope widening: #1728.
Two commits. The first makes the quarterly mutation-testing gate able to fail; the second makes it able to finish.
6b147d13— the alias, and three defects behind it (#1710)The
cargo mutantsalias in.cargo/config.tomlexpanded to a command of its own name, which cargo has rejected as recursive since 1.56 — it never worked. Investigating it turned up three more defects in the same path, none of which had ever produced a red job:cargo mutants … | teeunder GitHub's defaultbash -e(nopipefail)target/mutants/missed.txtmutants.out/under--output; that path is never written, so no outcome could ever file an issue-f src/metrics/-fglob containing/is whole-path matched; it selected no file--in-place -j 2Fix: alias removed;
.cargo/mutants.tomlwithminimum_test_timeout,output, andtest_package = [root, ast]so a checker/getter mutant is judged by the root crate's metric tests; workflow setspipefail, selectssrc/metrics/**/*.rs, reads the nested path.96a45849— sharding (#1726)The repaired glob selects 3812 mutants where the broken one selected 196; serial in-place at the measured ~8 s/mutant is ~8.5 h against the 6-hour hosted-runner cap, and a cancelled step is not
failure(), so a timeout would have filed nothing.Three jobs:
preparedeclares the shard list once;mutantsruns an 8-shard round-robin matrix with a pinnedcargo-mutants@27.1.0(every shard must generate the same list for the partition to be disjoint);reportaggregates. The design point that survived three review rounds: each shard writes oneshard-summary-<k>.jsonafter cargo-mutants returns, joining the captured exit status to the result files at the source. A cancelled, crashed, or timed-out shard never reaches that line, so "no summary" reads downstream as "not complete" by construction. The summarizer refuses the combinations cargo-mutants cannot produce — exit 0 beside escapes (a masked status), exit 2 beside timeouts, and a complete exit with zero mutants tested (the #1710 empty-selection shape). The report names non-complete shards rather than counting them, uploads its body as an artifact before anyghcall, and comments on an open issue by a fixed title instead of filing siblings.Both halves are
utils/mutation-report.py, undermake mutation-report-test(40 cases) inmake pre-commit/make ci, with aci.ymlexplicit twin and a pre-commit hook — lesson 80 applied to the workflow that proved it.Verification (cargo-mutants 27.1.0; each item names its command; the workflow-shell harness is now in-tree as
utils/mutation-workflow-test.py, #1731)run:bodies executed underbash -ewith stubcargo/gh; step exit equals cargo-mutants' in every case; removingpipefailmakes the summarizer refuse.make pre-commit:BCA_GATE: passon both commits.Known limits
prepareif any shard's step passes ~200 of its 270 minutes.