Skip to content

fix(tooling): replace the recursive cargo mutants alias with .cargo/mutants.toml - #1727

Merged
dekobon merged 2 commits into
mainfrom
fix/1710-mutants-config
Oct 31, 2026
Merged

dekobon merged 2 commits into
mainfrom
fix/1710-mutants-config

Conversation

@dekobon

@dekobon dekobon commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

Fixes #1710. Fixes #1726. Deferred scope widening: #1728.

Two commits. The first makes the quarterly mutation-testing gate able to fail; the second makes it able to finish.

6b147d13 — the alias, and three defects behind it (#1710)

The cargo mutants alias in .cargo/config.toml expanded to a command of its own name, which cargo has rejected as recursive since 1.56 — it never worked. Investigating it turned up three more defects in the same path, none of which had ever produced a red job:

Defect Effect
cargo mutants … | tee under GitHub's default bash -e (no pipefail) The 2026-10-01 run died on the alias error after 68 s and was reported success (138-byte artifact)
Issue step read target/mutants/missed.txt cargo-mutants nests mutants.out/ under --output; that path is never written, so no outcome could ever file an issue
-f src/metrics/ A -f glob containing / is whole-path matched; it selected no file
--in-place -j 2 clap conflict; would have failed the instant the alias error was gone

Fix: alias removed; .cargo/mutants.toml with minimum_test_timeout, output, and test_package = [root, ast] so a checker/getter mutant is judged by the root crate's metric tests; workflow sets pipefail, selects src/metrics/**/*.rs, reads the nested path.

96a45849 — sharding (#1726)

The repaired glob selects 3812 mutants where the broken one selected 196; serial in-place at the measured ~8 s/mutant is ~8.5 h against the 6-hour hosted-runner cap, and a cancelled step is not failure(), so a timeout would have filed nothing.

Three jobs: prepare declares the shard list once; mutants runs an 8-shard round-robin matrix with a pinned cargo-mutants@27.1.0 (every shard must generate the same list for the partition to be disjoint); report aggregates. The design point that survived three review rounds: each shard writes one shard-summary-<k>.json after cargo-mutants returns, joining the captured exit status to the result files at the source. A cancelled, crashed, or timed-out shard never reaches that line, so "no summary" reads downstream as "not complete" by construction. The summarizer refuses the combinations cargo-mutants cannot produce — exit 0 beside escapes (a masked status), exit 2 beside timeouts, and a complete exit with zero mutants tested (the #1710 empty-selection shape). The report names non-complete shards rather than counting them, uploads its body as an artifact before any gh call, and comments on an open issue by a fixed title instead of filing siblings.

Both halves are utils/mutation-report.py, under make mutation-report-test (40 cases) in make pre-commit / make ci, with a ci.yml explicit twin and a pre-commit hook — lesson 80 applied to the workflow that proved it.

Verification (cargo-mutants 27.1.0; each item names its command; the workflow-shell harness is now in-tree as utils/mutation-workflow-test.py, #1731)

  • Eight round-robin shards: 477×4 + 476×4, pairwise disjoint, union = the 3812-line list, 24–25 ast-crate mutants each (slice would put all 196 in the last shard).
  • All four run: bodies executed under bash -e with stub cargo/gh; step exit equals cargo-mutants' in every case; removing pipefail makes the summarizer refuse.
  • Every refusal and report branch perturbed; each fails exactly its one guarding test.
  • make pre-commit: BCA_GATE: pass on both commits.
  • Reviews: three on the first commit, four parallel plus two astra verification passes on the second; all findings folded in.

Known limits

  • Per-shard wall time is unmeasured. Dispatch the workflow once after merge; the doc's rule is to raise the count in prepare if any shard's step passes ~200 of its 270 minutes.
  • A re-run shard that fails before uploading keeps its earlier attempt's summary, indistinguishable from a shard not re-run. The body names mixed attempts; closing this needs per-shard API calls from the report job.
  • Expect the first completed run to file a large issue: no run has ever filed one, so it reports the accumulated backlog, not a regression.

…utants.toml

The `mutants` alias in `.cargo/config.toml` expanded to a command of
its own name, which cargo has rejected as an unresolvable recursive
definition since 1.56, so the alias never worked. The same alias broke
the scheduled mutation-test job, which reported success regardless:
its `cargo mutants … | tee` step ran under GitHub's default `bash -e`
with no `pipefail`, so the step took `tee`'s exit status. Two further
defects sat behind that one. The issue-filing step read
`target/mutants/missed.txt`, a path cargo-mutants never writes (it
nests `mutants.out/` under the output directory), and `-f src/metrics/`
selected no file at all, since a glob containing `/` is matched
against the whole path.

Remove the alias and move the shareable settings into a new
`.cargo/mutants.toml`: the timeout floor, the output directory, and a
`test_package` set so a checker or getter mutant is judged by the
root crate's metric tests rather than only the ast crate's. The
workflow sets `pipefail`, selects `src/metrics/**/*.rs`, reads the
nested result path, drops `-j 2`, which conflicts with `--in-place`,
and lowers `timeout-minutes` from 720 to the 360 that hosted runners
enforce regardless. Rewrite the local-run and triage guidance in
docs/development/mutation_testing.md and add lesson 104.

The repaired glob selects 3812 mutants where the broken one selected
196, and a serial in-place run over them is not expected to fit six
hours; a timed-out job is cancelled rather than failed and so files
no issue. Sharding is #1726.

Verified with cargo-mutants 27.1.0: `--list` with the workflow's
exact arguments selects 3812 mutants within the three scopes, a
single-file `-f` is not widened by the config, an unmatched
`--examine-re` run confirms the `mutants.out/` layout, a harness
running both `run:` bodies under plain `bash -e` with stub `cargo`
and `gh` fails three of five cases when `pipefail` is removed, and a
three-mutant smoke run through the config shows each mutant testing
both packages and the tree clean afterwards. Evidence under
~/bca-evidence/1710/.

Fixes #1710
…ard summaries

The selection #1710 repaired is 3812 mutants, run serially because
`--in-place` rules out `-j`, against the six hours at which GitHub
cancels a hosted-runner job. A cancelled step is not a `failure()`,
so a timed-out run would have filed nothing: the silent-pass shape
#1710 removed, back through the timeout.

Split the run into three jobs. `prepare` declares the shard list once
as a JSON output. `mutants` runs one matrix job per shard with
`--shard k/n --sharding round-robin` (slice would put every ast-crate
mutant, which rebuilds both crates, in the last two shards) and a
pinned cargo-mutants, since the partition is only disjoint if every
shard generates the same list. The step captures cargo-mutants' exit
status, has `utils/mutation-report.py summarize` join it to the
result files into one small summary, uploaded first and separately,
then exits with the status. A shard that is cancelled or crashes
never reaches that line and uploads no summary. `report` runs after
the matrix whatever its outcome and has `mutation-report.py report`
aggregate the summaries that arrived, naming any shard without a
well-formed one rather than counting it, so a timed-out shard cannot
pass for a clean one. The body is uploaded as an artifact before
anything talks to GitHub, then filed under a fixed title; an open
issue by that title gets a comment, not a sibling.

The summarize half refuses the one contradiction that would mean a
masked exit status (exit 0 beside recorded escapes); the report half
has two states per shard, usable or named-with-reason, so anything
that stops a summary being written fails toward red by construction.
Both are self-tested by `make mutation-report-test`, wired into
`make pre-commit` and `make ci`, so the result handling is checked on
every pull request rather than once a quarter. The step timeout sits
below the job's so a long shard is stopped with time to upload, and
rust-cache saves on failure, since every shard exits non-zero while
escapes remain.

Verified with cargo-mutants 27.1.0: the eight round-robin shards are
pairwise disjoint and their union is the full 3812-line list; the
shard step run locally with an unmatched `--examine-re` produces a
well-formed summary; and perturbing either the masked-exit refusal or
the absent-shard branch fails exactly the tests that guard them.
Evidence under ~/bca-evidence/1726/. The per-language `checker/` and
`getter/` modules (+1135 mutants) are deliberately not added here, so
the first real run measures the plumbing alone; that widening is its
own follow-up.

Fixes #1726
@dekobon
dekobon merged commit 96a4584 into main Oct 31, 2026
49 checks passed
@dekobon
dekobon deleted the fix/1710-mutants-config branch October 31, 2026 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant