Skip to content

verify: the tracker runs of verify-reads load for real, and VERIFY_SHARD runs a slow suite in parts - #85

Merged
Criptic merged 2 commits into
mainfrom
fix/verify-reads-load
Nov 10, 2026
Merged

Criptic merged 2 commits into
mainfrom
fix/verify-reads-load

Conversation

@Criptic

@Criptic Criptic commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Summary

Two changes to the Prompt Builder's verify suites, both found while making them faster.

1. verify-reads: the tracker runs load for real, and a click waits 5 s instead of 30

The tracker checks never loaded anything. The checks "a tracker whose rows hold ... is drawn and loaded" clicked the run's Load button without opening Build & Test first.

  • On any other step that button is hidden, so the click waited out Playwright's 30 seconds.
  • Its error was swallowed, and the check passed without loading anything. This happened three times per run of the suite.
  • The checks now open Build & Test, and pass only when the click works and the run's system prompt is loaded.

The optimization-history check now gives up after 5 seconds.

  • The history entry with every optional field empty offers nothing to load, so its click also waited 30 seconds.
  • It now gives up after 5 seconds and says whether the entry loaded.
  • As before, loading is not required for that entry.

2. VERIFY_SHARD runs a slow suite in parts

When the suites run side by side, three of them set the total time: verify-new-item (255 s), verify-outcomes (225 s) and verify-reads. The first two spend most of their time waiting on the app's poll timers.

VERIFY_SHARD=i/n runs part i of n, so the parts can run in parallel, each against its own mock (MOCK_PORT, VERIFY_BASE).

  • page-waits.js: new outsideShard(name, { always, block }). Without a block, each part takes every n-th road. This is how verify-outcomes and verify-reads split: their slow roads sit together, so taking them in turn spreads the slow ones across the parts.
  • verify-new-item: splits by block (prompt and decision templates, RAG setups, optimizations), because its roads read what earlier roads of the same block left. The three reference roads run in every part.
  • verify-round-trip: stays whole, because its last check adds up what all of its roads wrote.
  • Unchanged without a shard: with VERIFY_SHARD unset, every road runs as before, so run-all.cjs is unchanged.
  • Docs: SKILL.md says how.

Test plan

  • verify-reads, whole: 141 checks pass. The three tracker checks report loaded: true; the all-empty optimization entry reports loaded: false. Time on Linux arm64 with Chromium: 249 s before, 134 s after.
  • Whole, the three changed suites pass as before (verify-new-item 89, verify-outcomes 77, verify-reads 141).
  • In parts, the three suites pass, and every distinct check of the whole run is in its parts (84, 75 and 134).
    • verify-new-item: 3 parts of 61–122 s.
    • verify-outcomes: 3 parts of 67–83 s.
    • verify-reads: 2 parts of 64–71 s.
  • All 13 suites as 18 jobs, 10 at once: 128 s, against 322 s with every suite whole and 33 min one after another on Windows.
  • Full CI on Python 3.11 and 3.14 passes.

🤖 Generated with Claude Code

Criptic and others added 2 commits November 10, 2026 15:13
The checks "a tracker whose rows hold ... is drawn and loaded" clicked the
Load button of the run without opening Build & Test first. There the
button is hidden, so the click waited out Playwright's 30 seconds, its
error was swallowed, and the check passed without loading anything - three
times per run of the suite. They now open Build & Test, and pass only when
the click works and the system prompt of the run is loaded.

The optimization history entry with every optional field empty offers
nothing to load; its click waited 30 seconds as well. It now gives up
after 5 and says whether the entry loaded, as before without requiring it.

verify-reads: 249 s -> 134 s on the DGX Spark (Chromium), 141 checks pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: David Weik <geekupyourlife@gmail.com>
verify-new-item (255 s), verify-outcomes (225 s) and verify-reads (134 s)
set the wall time of the verify suites when they run side by side, and
both of the first two spend most of it waiting on the app's poll timers.
VERIFY_SHARD=i/n runs part i of n, so the parts can run in parallel, each
against a mock of its own (MOCK_PORT, VERIFY_BASE).

- page-waits.js: outsideShard(name, { always, block }). Without a block a
  part takes every n-th road (verify-outcomes, verify-reads: their slow
  roads sit together, so round-robin spreads them).
- verify-new-item splits by block - prompt and decision templates, RAG
  setups, optimizations - because its roads read what earlier roads of the
  same block left; the three references run in every part.
- verify-round-trip stays whole: its last check adds up what all of its
  roads wrote.
- SKILL.md says how.

Measured on the DGX Spark (Chromium, one mock per part): every check of
the whole suites is in their parts (84, 77 and 134 distinct checks);
verify-new-item 3 parts of 61-122 s, verify-outcomes 3 of 67-83 s,
verify-reads 2 of 64-71 s; all 13 suites as 18 jobs in 128 s, against
322 s with the suites whole.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: David Weik <geekupyourlife@gmail.com>
@Criptic Criptic changed the title verify-reads: load the tracker runs for real, and wait 5 s instead of 30 verify: the tracker runs of verify-reads load for real, and VERIFY_SHARD runs a slow suite in parts Nov 10, 2026
@Criptic
Criptic merged commit fff1fd1 into main Nov 10, 2026
2 checks passed
@Criptic
Criptic deleted the fix/verify-reads-load branch November 10, 2026 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant