Skip to content

perf(ci): add focused benchmark commands#216

Open
marcus-pousette wants to merge 1 commit into
mainfrom
perf/benchmark-focused-commands
Open

perf(ci): add focused benchmark commands#216
marcus-pousette wants to merge 1 commit into
mainfrom
perf/benchmark-focused-commands

Conversation

@marcus-pousette

@marcus-pousette marcus-pousette commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Why

The exhaustive benchmark workflow takes about 57 minutes. That is useful before a performance-sensitive merge, but too expensive for routine PR feedback.

Most of the time is real benchmark coverage rather than a second benchmark framework: the workflow orchestrates the existing web, native SQLite, direct-sync, Wasm, and Rust-core runners. This PR adds a focused mode that samples those same implementations at representative sizes.

Commands

  • /bench or /bench quick — focused cross-runtime signal (27 result rows)
  • /bench web — the existing full browser suite, including browser sync
  • /bench full — the existing exhaustive cross-runtime suite

Manual workflow runs expose the same quick, web, and full choices, with quick as the default.

What quick keeps

  • Browser memory and OPFS operation coverage
  • Native SQLite operation and note-path coverage
  • Native direct-sync coverage
  • Wasm and Rust-core coverage

The browser-specific sync matrix remains in /bench web. This avoids making old base/head revisions use different browser instrumentation while keeping a fast sync signal through the existing native direct-sync runner.

Other cleanup

  • Checks out the small quick orchestrator from the trusted workflow revision, so it can benchmark older open PRs that do not contain the new script yet.
  • Avoids a duplicate successful aggregation pass.
  • Removes unnecessary full-history checkouts and unused Rust components.
  • Reuses the existing native-SQLite package-script alias in the full command.
  • Documents how the package-local and root-level runners relate.

The focused suite is not a replacement for /bench full; it is the routine feedback path.

Measured impact

The exhaustive run took 56m47s. The validated focused run took 16m42s, a roughly 71% wall-time reduction.

Focused-run breakdown:

  • Full setup and two project builds: about 8m50s
  • Base measurements: 3m48s
  • Head measurements: 3m48s
  • Results: 27 rows per revision, with identical groups

Build/setup is now the largest remaining fixed cost. The companion iteration fix in #215 removes additional hidden native oversampling.

Validation

  • Prettier, Node syntax, Rust formatting, and git diff --check
  • Verified pnpm forwards the browser filter through both package-script levels
  • Playwright test discovery confirms quick selects exactly the two generic browser tests
  • Independent diff review: no blockers; expected result count is 27
  • Live workflow run: success in 16m42s
  • Artifact contains 27 base rows and 27 head rows; PR comparison comment updated successfully

Merged correctness/performance fix: #215. Its benchmark regression tests remain in the native test command after the rebase.

@marcus-pousette marcus-pousette added the performance Measurable runtime, storage, or responsiveness improvement label Jul 13, 2026
@github-actions

github-actions Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Benchmarks

Triggered by workflow_dispatch - Run: https://github.com/cybersemics/treecrdt/actions/runs/29268135321
Suite: quick - Base command: success - Head command: success
Base: main (683b623) - Head: perf/benchmark-focused-commands (1b34e70)
Compared entries: 27 - Improved: 18 - Regressed: 9

Impl Storage Workload Base ops/s Head ops/s Δ ops/s Base p50 ms Head p50 ms Δ p50 Status
core-inmem-crdt memory insert-move-1000 13829.97 13789.78 -0.29% 144.61 145.03 +0.29%
sqlite-node file insert-into-balanced-tree-fanout10-10000 478.27 473.72 -0.95% 2.09 2.11 +0.96%
sqlite-node file insert-move-1000 18292.93 19410.90 +6.11% 109.33 103.03 -5.76% 🟡
sqlite-node file read-children-payloads-fanout10-10000 14322.21 14947.32 +4.36% 0.77 0.74 -4.18%
sqlite-node file sync-balanced-children-cold-start-fanout10-1000 355.03 385.26 +8.51% 28.17 25.96 -7.84% 🟡
sqlite-node file sync-balanced-children-payloads-cold-start-fanout10-1000 635.62 691.56 +8.80% 33.04 30.37 -8.09% 🟡
sqlite-node file sync-one-missing-1000 10.31 10.73 +4.02% 96.95 93.21 -3.86%
sqlite-node memory insert-into-balanced-tree-fanout10-10000 1416.28 1432.13 +1.12% 0.71 0.70 -1.11%
sqlite-node memory insert-move-1000 17434.72 17674.00 +1.37% 114.71 113.16 -1.35%
sqlite-node memory read-children-payloads-fanout10-10000 22464.20 22312.24 -0.68% 0.49 0.49 +0.68%
sqlite-node memory sync-balanced-children-cold-start-fanout10-1000 344.94 398.17 +15.43% 28.99 25.12 -13.37% 🟠
sqlite-node memory sync-balanced-children-payloads-cold-start-fanout10-1000 639.51 698.70 +9.26% 32.84 30.06 -8.47% 🟡
sqlite-node memory sync-one-missing-1000 10.06 10.44 +3.76% 99.37 95.77 -3.63%
wa-sqlite browser-memory insert-chain-100 3003.91 2846.98 -5.22% 33.29 35.13 +5.51% 🟡
wa-sqlite browser-memory insert-chain-1000 400.87 383.30 -4.38% 2494.57 2608.92 +4.58%
wa-sqlite browser-memory insert-move-100 5148.67 5462.99 +6.10% 38.84 36.61 -5.75% 🟡
wa-sqlite browser-memory insert-move-1000 7784.98 7433.70 -4.51% 256.91 269.04 +4.73%
wa-sqlite browser-memory replay-log-100 3081.90 3004.36 -2.52% 32.45 33.29 +2.58%
wa-sqlite browser-memory replay-log-1000 402.21 382.07 -5.01% 2486.27 2617.31 +5.27% 🟡
wa-sqlite browser-opfs-coop-sync insert-chain-100 1156.17 1299.17 +12.37% 86.49 76.97 -11.01% 🟠
wa-sqlite browser-opfs-coop-sync insert-chain-1000 375.02 380.37 +1.43% 2666.52 2629.04 -1.41%
wa-sqlite browser-opfs-coop-sync insert-move-100 1928.55 2073.83 +7.53% 103.71 96.44 -7.01% 🟡
wa-sqlite browser-opfs-coop-sync insert-move-1000 3667.24 4055.52 +10.59% 545.37 493.16 -9.57% 🟠
wa-sqlite browser-opfs-coop-sync replay-log-100 1276.81 1307.15 +2.38% 78.32 76.50 -2.32%
wa-sqlite browser-opfs-coop-sync replay-log-1000 374.74 381.40 +1.78% 2668.50 2621.92 -1.75%
wasm memory insert-chain-1000 26345.45 28481.85 +8.11% 37.96 35.11 -7.50% 🟡
wasm memory insert-move-1000 27417.27 27045.18 -1.36% 72.95 73.95 +1.38%

Status: ✅ <5% change | 🟡 5–10% | 🟠 10–30% | 🔴 >30%

Artifacts:

  • base summary: bench-base/benchmarks/summary.json
  • head summary: bench-head/benchmarks/summary.json
  • command logs: bench-base/benchmark.log, bench-head/benchmark.log

@marcus-pousette
marcus-pousette force-pushed the perf/benchmark-focused-commands branch 2 times, most recently from 2e83cb2 to be736f1 Compare July 13, 2026 10:16
@marcus-pousette
marcus-pousette force-pushed the perf/benchmark-focused-commands branch from be736f1 to 1b34e70 Compare July 13, 2026 16:22
@marcus-pousette

Copy link
Copy Markdown
Collaborator Author

/bench

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance Measurable runtime, storage, or responsiveness improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants