fix(bench): honor explicit iteration counts#215
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The shared benchmark runner's small-workload sampling floor currently overrides an explicit iteration count.
That silently multiplies callers which already perform their own outer sampling. In the SQLite note-path suite, the intended 2–3 samples become 20–30 seeded database runs per result: 200 database seedings per revision while the artifact still reports only 2–3 samples. The 100-operation SQLite benchmark is multiplied similarly.
In the successful
/benchrun for #167, note paths consumed about 5m12s per revision. Most of that work was unreported nested sampling.What changed
Expected impact
The note-path matrix now performs its intended 20 seeded runs per revision instead of 200. Based on run 29234317596, that should save roughly nine minutes across a full base/head comparison while making the reported sample counts truthful.
Unconfigured tiny workloads retain the existing adaptive sampling behavior.
Validation
pnpm -C packages/treecrdt-benchmark testgit diff --check