Repository navigation
iterate on CI bench - #2298
Merged
Merged
iterate on CI bench#2298
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
ArthurZucker
approved these changes
Aug 6, 2026
Comment on lines
+10
to
+11
| # multi-thread sweep (1/2/4/8/device-max) per fixture group, per-implementation | ||
| # memory footprint (RSS), and stripped minimal-binary size. The release dep is |
Collaborator
There was a problem hiding this comment.
Memory footprint should be HEAP ALLOCATION, its the best proxy afaik
Collaborator
Author
There was a problem hiding this comment.
Yep, allocs + instruction counts coming
SBrandeis
force-pushed
the
ci/better-benches
branch
from
August 6, 2026 11:15
27e19df to
41805d8
Compare
A callgrind pass over the 64 MB corpus costs 10+ minutes per model in CI (the release baseline runs at ~3.5 MB/s natively, ~50x slower simulated), which pushed the bench shards past 40 minutes. The allocation lane keeps the deterministic work tripwire and is free: it rides along the memory children that run anyway. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The released crate is pinned, so its numbers only move when the machine does, yet timing its few-MB/s encodes dominated every bench run. The plan job now resolves cached baseline numbers from the HF Hub dataset, keyed on a hash of everything that shapes them (fixture_bench.rs, the model manifest, the Makefile fixture lists); on a hit the bench jobs run --baseline-from and skip all release timing, re-measuring one canary fixture per model so the report can flag runner drift (fence ±10%). On a miss the run measures the release and the report job seeds the cache. The shard matrix becomes one job per model (wall clock = slowest model, jobs named after what they bench), REPS drops 5 -> 3, and the thread sweep narrows to 2/4 threads; the thread chart's linear reference now anchors on the sweep's first point instead of assuming a 1-thread row. Verified locally on gpt2: cached-mode run finishes in 17 s vs ~7 min for the measuring run, stitches the cached numbers byte-identically, passes the id gate, and the canary drifted +1.1%. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…iming process #2305 landed `PipelineTokenizer::decode` on the base branch, so the reason this branch dropped the decode phase — "it's a loud stub, there is nothing to compare yet" — no longer holds. Decode returns as phase 4, reshaped for what the branch changed underneath it. The release still mints the id stream both decoders replay, so decode is judged on decode alone, but it mints ONCE per model now instead of once per lane, over a capped prefix of each fixture (DECODE_SAMPLE_BYTES). Minting runs at release encode speed — the slowest thing in the bench, and exactly what --baseline-from exists to skip — while decoding is two orders of magnitude faster, so a full-corpus stream would cost far more to prepare than to measure. Throughput is denominated in the input bytes the ids came from: free, identical in both modes, and on the same scale as the encode charts. --baseline-from covers decode_mbps and decode_threads; the thread sweep is keyed per fixture group like the encode one, and both share a generalized par_mbps. The memory children grow a decode leg with per-fixture allocation rows, bracketed so the id buffer lands in neither encode_bytes nor decode_bytes. WordPiece still keeps no id -> token map, so those models report a null pipeline series ("pending") while the release's is measured — the gap stays on the chart instead of vanishing from it. The counting allocator is now confined to those children. Its four shared atomics were running in the process that measures wall time, and multi-threaded allocation traffic serializes on them hard enough to invert a scaling curve. On gpt2, baseline MB/s at 2 -> 4 threads, counting off vs on: encode 9.5 -> 17.6 vs 4.8 -> 3.1, decode 37.5 -> 64.8 vs 13.1 -> 7.5. The pipeline's own numbers moved by noise, because the contention scales with how often an implementation allocates and the release allocates ~3000x more per decode — so the lane was quietly distorting the exact comparison the sweeps exist to draw. Single-thread numbers were unaffected; uncontended atomics are cheap. The CI gate fails on text_match as well as ids_match, and the workflow's claim that the decode oracle is #[ignore]d is gone: it runs, and its two WordPiece tests fail on purpose. Verified on gpt2: decode geomean x9.87 vs the release (352 MB/s on English, matching #2305's own measurements), allocation counts bit-identical across two independent runs in both lanes, cached-baseline mode stitches the decode numbers and finishes in 26 s. bert-base-uncased exercises the pending path end to end. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SBrandeis
force-pushed
the
ci/better-benches
branch
from
August 6, 2026 13:52
fd6588a to
460b4cb
Compare
SBrandeis
marked this pull request as ready for review
August 6, 2026 14:07
ArthurZucker
added a commit
that referenced
this pull request
Sep 2, 2026
Squashed from 3 commits: - feat(tk-encode): implement PipelineTokenizer::decode (8-10x over the legacy path) (#2305) - feat(decode): wire wordpiece with a naive reverse vocab lookup (#2309) - iterate on CI bench (#2298) Area: core-decode Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PipelineTokenizer benchmark
10 / 10 models supported — PipelineTokenizer vs
tokenizersv0.23.1 (latest release) · whole corpus · ~10 kB chunks · cold caches · add_special_tokens on · 2/4-thread sweep per group460b4cb94 · 2026-08-06 14:03 UTC· Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 32 coresDecode — the release encodes each fixture's decode sample (
add_special_tokens=true) and both implementations decode those SAME ids withskip_special_tokens=false, so the comparison is decode alone. MB/s counts the input bytes the ids came from, the same denominator as the encode charts.vs base branch (
f793dbf5c) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.Work vs base (
f793dbf5c, allocation lane):allocation lane measured on: Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · glibc 2.39
bert-base-uncased — normalizer-heavy WordPiece · ×4.93 vs v0.23.1 · ×0.98 vs base · decode pending
Memory (RSS MB, load+encode+decode): v0.23.1 7+11+0 (peak 38) · Pipeline 4+4+— (peak 15)
Allocations (encode pass, whole corpus): v0.23.1: 159.5M allocs, 52.04 GB allocated, peak live 37 MB; Pipeline: 61.6k allocs, 0.94 GB allocated, peak live 16 MB
deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×14.77 vs v0.23.1 · ×0.63 vs base · decode ×9.83
Memory (RSS MB, load+encode+decode): v0.23.1 57+0+0 (peak 75) · Pipeline 41+0+0 (peak 88)
Allocations (encode pass, whole corpus): v0.23.1: 153.2M allocs, 38.67 GB allocated, peak live 66 MB; Pipeline: 28.0k allocs, 0.23 GB allocated, peak live 74 MB
Allocations (decode pass, decode sample): v0.23.1: 15.3M allocs, 0.77 GB allocated; Pipeline: 7.4k allocs, 0.06 GB allocated
gemma-4 — byte-fallback BPE, Metaspace-style split (gemma-4) · ×2.91 vs v0.23.1 · ×0.07 vs base · decode ×1.54
Memory (RSS MB, load+encode+decode): v0.23.1 286+0+0 (peak 366) · Pipeline 1+0+0 (peak 365)
Allocations (encode pass, whole corpus): v0.23.1: 29.1M allocs, 23.69 GB allocated, peak live 310 MB; Pipeline: 65.2k allocs, 2.35 GB allocated, peak live 310 MB
Allocations (decode pass, decode sample): v0.23.1: 18.9M allocs, 1.79 GB allocated; Pipeline: 18.9M allocs, 1.79 GB allocated
gpt2 — gpt2 ByteLevel regex · ×15.05 vs v0.23.1 · ×0.48 vs base · decode ×30.49
Memory (RSS MB, load+encode+decode): v0.23.1 21+12+0 (peak 82) · Pipeline 6+4+0 (peak 84)
Allocations (encode pass, whole corpus): v0.23.1: 243.3M allocs, 38.17 GB allocated, peak live 79 MB; Pipeline: 34.6k allocs, 0.43 GB allocated, peak live 80 MB
Allocations (decode pass, decode sample): v0.23.1: 30.9M allocs, 1.36 GB allocated; Pipeline: 6.0k allocs, 0.07 GB allocated
gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×8.12 vs v0.23.1 · ×0.39 vs base · decode ×16.39
Memory (RSS MB, load+encode+decode): v0.23.1 236+0+0 (peak 310) · Pipeline 1+0+0 (peak 311)
Allocations (encode pass, whole corpus): v0.23.1: 127.8M allocs, 29.89 GB allocated, peak live 272 MB; Pipeline: 28.4k allocs, 0.23 GB allocated, peak live 272 MB
Allocations (decode pass, decode sample): v0.23.1: 14.5M allocs, 0.71 GB allocated; Pipeline: 7.4k allocs, 0.05 GB allocated
glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×10.97 vs v0.23.1 · ×0.52 vs base · decode ×14.48
Memory (RSS MB, load+encode+decode): v0.23.1 164+0+0 (peak 225) · Pipeline 1+0+0 (peak 226)
Allocations (encode pass, whole corpus): v0.23.1: 154.7M allocs, 33.00 GB allocated, peak live 212 MB; Pipeline: 29.5k allocs, 0.25 GB allocated, peak live 212 MB
Allocations (decode pass, decode sample): v0.23.1: 18.3M allocs, 0.90 GB allocated; Pipeline: 6.8k allocs, 0.05 GB allocated
llama-2 — model-bounded BPE, no pre-tokenizer · ×4.89 vs v0.23.1 · ×0.14 vs base · decode ×1.43
Memory (RSS MB, load+encode+decode): v0.23.1 20+6+0 (peak 63) · Pipeline 7+3+1 (peak 71)
Allocations (encode pass, whole corpus): v0.23.1: 63.0M allocs, 28.31 GB allocated, peak live 61 MB; Pipeline: 67.7k allocs, 0.56 GB allocated, peak live 72 MB
Allocations (decode pass, decode sample): v0.23.1: 35.7M allocs, 3.03 GB allocated; Pipeline: 35.7M allocs, 3.03 GB allocated
llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×13.86 vs v0.23.1 · ×0.58 vs base · decode ×19.48
Memory (RSS MB, load+encode+decode): v0.23.1 68+2+0 (peak 90) · Pipeline 36+0+0 (peak 103)
Allocations (encode pass, whole corpus): v0.23.1: 152.3M allocs, 33.83 GB allocated, peak live 91 MB; Pipeline: 28.7k allocs, 0.24 GB allocated, peak live 92 MB
Allocations (decode pass, decode sample): v0.23.1: 17.3M allocs, 0.88 GB allocated; Pipeline: 9.6k allocs, 0.06 GB allocated
mistral-small-4 — tekken byte-level BPE, 1k added specials (mistral-small-4) · ×9.44 vs v0.23.1 · ×0.50 vs base · decode ×10.74
Memory (RSS MB, load+encode+decode): v0.23.1 151+0+0 (peak 189) · Pipeline 9+0+0 (peak 189)
Allocations (encode pass, whole corpus): v0.23.1: 135.2M allocs, 31.26 GB allocated, peak live 185 MB; Pipeline: 28.7k allocs, 0.24 GB allocated, peak live 185 MB
Allocations (decode pass, decode sample): v0.23.1: 16.6M allocs, 0.81 GB allocated; Pipeline: 7.1k allocs, 0.05 GB allocated
t5-base — Unigram + Metaspace · ×2.93 vs v0.23.1 · ×0.53 vs base · decode ×0.96
Memory (RSS MB, load+encode+decode): v0.23.1 31+10+0 (peak 56) · Pipeline 26+14+1 (peak 94)
Allocations (encode pass, whole corpus): v0.23.1: 260.7M allocs, 35.16 GB allocated, peak live 53 MB; Pipeline: 50.7M allocs, 4.73 GB allocated, peak live 107 MB
Allocations (decode pass, decode sample): v0.23.1: 12.0M allocs, 0.64 GB allocated; Pipeline: 12.0M allocs, 0.64 GB allocated