Skip to content

fix(bench): honor explicit iteration counts#215

Merged
Antonov548 merged 1 commit into
mainfrom
fix/benchmark-iteration-precedence
Jul 13, 2026
Merged

fix(bench): honor explicit iteration counts#215
Antonov548 merged 1 commit into
mainfrom
fix/benchmark-iteration-precedence

Conversation

@marcus-pousette

Copy link
Copy Markdown
Collaborator

Why

The shared benchmark runner's small-workload sampling floor currently overrides an explicit iteration count.

That silently multiplies callers which already perform their own outer sampling. In the SQLite note-path suite, the intended 2–3 samples become 20–30 seeded database runs per result: 200 database seedings per revision while the artifact still reports only 2–3 samples. The 100-operation SQLite benchmark is multiplied similarly.

In the successful /bench run for #167, note paths consumed about 5m12s per revision. Most of that work was unreported nested sampling.

What changed

  • Treat workload and environment iteration counts as explicit requests.
  • Apply the automatic 7–10-sample floor only when neither supplies a count.
  • Add dependency-free regression tests for workload, environment, and fallback precedence.
  • Include the benchmark package test in the native Node test command so CI runs it.

Expected impact

The note-path matrix now performs its intended 20 seeded runs per revision instead of 200. Based on run 29234317596, that should save roughly nine minutes across a full base/head comparison while making the reported sample counts truthful.

Unconfigured tiny workloads retain the existing adaptive sampling behavior.

Validation

  • pnpm -C packages/treecrdt-benchmark test
  • Prettier check on all changed files
  • git diff --check

@marcus-pousette marcus-pousette added bug Something isn't working area:sqlite SQLite extension, sqlite-node, wa-sqlite persistence, and migrations labels Jul 13, 2026
@Antonov548
Antonov548 merged commit 63154a4 into main Jul 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:sqlite SQLite extension, sqlite-node, wa-sqlite persistence, and migrations bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants