What is missing
crates/fkst-framework/src/test_runner.rs builds a per-test report carrying identity and status and
no duration, and the test subcommand accepts only --project-root, --package-root and
--report-json — no test filter.
Two capabilities are wanted, and they are independent:
- Per-test duration in the
--report-json payload. A monotonic elapsed time per test entry.
- A test filter on the
test subcommand — select a subset of tests by a stable identifier, so a
caller can execute a named subset rather than everything in a package root.
Why a package repository cannot supply either itself
Measured in fkst-packages (docs/dev/test-wall-clock-measurement.md), four independent lines of
attack on test wall-clock all terminate at these two capabilities:
- lowering the critical-path package needs a cost-weighted partition, which needs per-test duration;
- "run only the tests a change affects" at test-file granularity needs stable per-test IDs and a filter;
- knowing where time goes inside a unit needs per-test duration;
- an equal-count shard of the critical-path package was implemented and measured: span 640.7 s →
573.1 s, only −10.5% against a −54% projection, and the two shards summed to 680.4 s against
582.3 s unsharded — the split added ~98 s of work and distributed it 3.3:1. Cost-weighted
partitioning is the only shape that could work.
Six structurally different attempts to decompose per-test cost from outside the engine all failed,
each at a different boundary — stdout block-buffering, scripts not being an importable package,
guessed variable names, and hand-rolled package roots failing at manifest catalog is required: missing fkst.workspace.toml. Two of those failures reported an implausibly clean 0.0 s / tests=0,
i.e. a startup crash read as a measurement. The one measurement that worked cleanly sat inside
the machinery. That is the evidence for putting this in the engine rather than emulating it caller-side.
Why it matters, in numbers
From a dedicated CI runner (run 31442678543, 22 package units):
|
|
| package pool wall-span |
640.7 s |
| serial sum of all units |
1166.9 s |
github-devloop |
582.3 s = 90.9% of the span |
identical-machine makespan floor max(p_max, W/4) |
582.3 s |
The pool is at ~91% of its own capacity floor and one unit is the floor. No scheduling change in
the package repository can move it; only splitting that unit can, and splitting it correctly needs
per-test cost.
Separately, package granularity is also the floor for selection: scripts/test_affected.py
resolves a change to a set of packages, and a SCOPED verdict resolves to 10.4 packages on average
(20 of 134 sampled commits select all 22). Finer selection needs a filter.
Scope notes
- The two capabilities are separable and the first is much smaller; per-test duration alone unblocks
the cost-weighted-partition line and the "where does time go inside a unit" line.
- A stable test identifier is a prerequisite for the filter. Whether the current report's identity
field is already stable enough to serve as a filter key is ASSUMED-UNVERIFIED from the
package-repository side.
- No behaviour change to what tests run by default is requested. A filter that is absent must mean
"run everything", so the default gate stays comprehensive.
⟦AI:FKST⟧
What is missing
crates/fkst-framework/src/test_runner.rsbuilds a per-test report carrying identity and status andno duration, and the
testsubcommand accepts only--project-root,--package-rootand--report-json— no test filter.Two capabilities are wanted, and they are independent:
--report-jsonpayload. A monotonic elapsed time per test entry.testsubcommand — select a subset of tests by a stable identifier, so acaller can execute a named subset rather than everything in a package root.
Why a package repository cannot supply either itself
Measured in
fkst-packages(docs/dev/test-wall-clock-measurement.md), four independent lines ofattack on test wall-clock all terminate at these two capabilities:
573.1 s, only −10.5% against a −54% projection, and the two shards summed to 680.4 s against
582.3 s unsharded — the split added ~98 s of work and distributed it 3.3:1. Cost-weighted
partitioning is the only shape that could work.
Six structurally different attempts to decompose per-test cost from outside the engine all failed,
each at a different boundary — stdout block-buffering,
scriptsnot being an importable package,guessed variable names, and hand-rolled package roots failing at
manifest catalog is required: missing fkst.workspace.toml. Two of those failures reported an implausibly clean0.0 s/tests=0,i.e. a startup crash read as a measurement. The one measurement that worked cleanly sat inside
the machinery. That is the evidence for putting this in the engine rather than emulating it caller-side.
Why it matters, in numbers
From a dedicated CI runner (run 31442678543, 22 package units):
github-devloopmax(p_max, W/4)The pool is at ~91% of its own capacity floor and one unit is the floor. No scheduling change in
the package repository can move it; only splitting that unit can, and splitting it correctly needs
per-test cost.
Separately, package granularity is also the floor for selection:
scripts/test_affected.pyresolves a change to a set of packages, and a SCOPED verdict resolves to 10.4 packages on average
(20 of 134 sampled commits select all 22). Finer selection needs a filter.
Scope notes
the cost-weighted-partition line and the "where does time go inside a unit" line.
field is already stable enough to serve as a filter key is
ASSUMED-UNVERIFIEDfrom thepackage-repository side.
"run everything", so the default gate stays comprehensive.
⟦AI:FKST⟧