Skip to content

fkst-framework test reports test identity and status but no duration, and takes no filter — so a package repository cannot shard or select below package granularity #371

Description

@loning

What is missing

crates/fkst-framework/src/test_runner.rs builds a per-test report carrying identity and status and
no duration, and the test subcommand accepts only --project-root, --package-root and
--report-jsonno test filter.

Two capabilities are wanted, and they are independent:

  1. Per-test duration in the --report-json payload. A monotonic elapsed time per test entry.
  2. A test filter on the test subcommand — select a subset of tests by a stable identifier, so a
    caller can execute a named subset rather than everything in a package root.

Why a package repository cannot supply either itself

Measured in fkst-packages (docs/dev/test-wall-clock-measurement.md), four independent lines of
attack on test wall-clock all terminate at these two capabilities:

  • lowering the critical-path package needs a cost-weighted partition, which needs per-test duration;
  • "run only the tests a change affects" at test-file granularity needs stable per-test IDs and a filter;
  • knowing where time goes inside a unit needs per-test duration;
  • an equal-count shard of the critical-path package was implemented and measured: span 640.7 s →
    573.1 s, only −10.5% against a −54% projection, and the two shards summed to 680.4 s against
    582.3 s unsharded — the split added ~98 s of work and distributed it 3.3:1. Cost-weighted
    partitioning is the only shape that could work.

Six structurally different attempts to decompose per-test cost from outside the engine all failed,
each at a different boundary
— stdout block-buffering, scripts not being an importable package,
guessed variable names, and hand-rolled package roots failing at manifest catalog is required: missing fkst.workspace.toml. Two of those failures reported an implausibly clean 0.0 s / tests=0,
i.e. a startup crash read as a measurement. The one measurement that worked cleanly sat inside
the machinery. That is the evidence for putting this in the engine rather than emulating it caller-side.

Why it matters, in numbers

From a dedicated CI runner (run 31442678543, 22 package units):

package pool wall-span 640.7 s
serial sum of all units 1166.9 s
github-devloop 582.3 s = 90.9% of the span
identical-machine makespan floor max(p_max, W/4) 582.3 s

The pool is at ~91% of its own capacity floor and one unit is the floor. No scheduling change in
the package repository can move it; only splitting that unit can, and splitting it correctly needs
per-test cost.

Separately, package granularity is also the floor for selection: scripts/test_affected.py
resolves a change to a set of packages, and a SCOPED verdict resolves to 10.4 packages on average
(20 of 134 sampled commits select all 22). Finer selection needs a filter.

Scope notes

  • The two capabilities are separable and the first is much smaller; per-test duration alone unblocks
    the cost-weighted-partition line and the "where does time go inside a unit" line.
  • A stable test identifier is a prerequisite for the filter. Whether the current report's identity
    field is already stable enough to serve as a filter key is ASSUMED-UNVERIFIED from the
    package-repository side.
  • No behaviour change to what tests run by default is requested. A filter that is absent must mean
    "run everything", so the default gate stays comprehensive.

⟦AI:FKST⟧

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions