Repository navigation
verify: the tracker runs of verify-reads load for real, and VERIFY_SHARD runs a slow suite in parts - #85
Merged
Conversation
The checks "a tracker whose rows hold ... is drawn and loaded" clicked the Load button of the run without opening Build & Test first. There the button is hidden, so the click waited out Playwright's 30 seconds, its error was swallowed, and the check passed without loading anything - three times per run of the suite. They now open Build & Test, and pass only when the click works and the system prompt of the run is loaded. The optimization history entry with every optional field empty offers nothing to load; its click waited 30 seconds as well. It now gives up after 5 and says whether the entry loaded, as before without requiring it. verify-reads: 249 s -> 134 s on the DGX Spark (Chromium), 141 checks pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: David Weik <geekupyourlife@gmail.com>
verify-new-item (255 s), verify-outcomes (225 s) and verify-reads (134 s)
set the wall time of the verify suites when they run side by side, and
both of the first two spend most of it waiting on the app's poll timers.
VERIFY_SHARD=i/n runs part i of n, so the parts can run in parallel, each
against a mock of its own (MOCK_PORT, VERIFY_BASE).
- page-waits.js: outsideShard(name, { always, block }). Without a block a
part takes every n-th road (verify-outcomes, verify-reads: their slow
roads sit together, so round-robin spreads them).
- verify-new-item splits by block - prompt and decision templates, RAG
setups, optimizations - because its roads read what earlier roads of the
same block left; the three references run in every part.
- verify-round-trip stays whole: its last check adds up what all of its
roads wrote.
- SKILL.md says how.
Measured on the DGX Spark (Chromium, one mock per part): every check of
the whole suites is in their parts (84, 77 and 134 distinct checks);
verify-new-item 3 parts of 61-122 s, verify-outcomes 3 of 67-83 s,
verify-reads 2 of 64-71 s; all 13 suites as 18 jobs in 128 s, against
322 s with the suites whole.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: David Weik <geekupyourlife@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two changes to the Prompt Builder's verify suites, both found while making them faster.
1. verify-reads: the tracker runs load for real, and a click waits 5 s instead of 30
The tracker checks never loaded anything. The checks "a tracker whose rows hold ... is drawn and loaded" clicked the run's Load button without opening Build & Test first.
The optimization-history check now gives up after 5 seconds.
2.
VERIFY_SHARDruns a slow suite in partsWhen the suites run side by side, three of them set the total time:
verify-new-item(255 s),verify-outcomes(225 s) andverify-reads. The first two spend most of their time waiting on the app's poll timers.VERIFY_SHARD=i/nruns part i of n, so the parts can run in parallel, each against its own mock (MOCK_PORT,VERIFY_BASE).page-waits.js: newoutsideShard(name, { always, block }). Without a block, each part takes every n-th road. This is howverify-outcomesandverify-readssplit: their slow roads sit together, so taking them in turn spreads the slow ones across the parts.verify-new-item: splits by block (prompt and decision templates, RAG setups, optimizations), because its roads read what earlier roads of the same block left. The three reference roads run in every part.verify-round-trip: stays whole, because its last check adds up what all of its roads wrote.VERIFY_SHARDunset, every road runs as before, sorun-all.cjsis unchanged.Test plan
verify-reads, whole: 141 checks pass. The three tracker checks reportloaded: true; the all-empty optimization entry reportsloaded: false. Time on Linux arm64 with Chromium: 249 s before, 134 s after.verify-new-item89,verify-outcomes77,verify-reads141).verify-new-item: 3 parts of 61–122 s.verify-outcomes: 3 parts of 67–83 s.verify-reads: 2 parts of 64–71 s.🤖 Generated with Claude Code