Skip to content

iterate on CI bench - #2298

Merged
SBrandeis merged 6 commits into
feat/train_encode_splitfrom
ci/better-benches
Aug 6, 2026
Merged

SBrandeis merged 6 commits into
feat/train_encode_splitfrom
ci/better-benches

Conversation

@SBrandeis

@SBrandeis SBrandeis commented Aug 5, 2026 •

Copy link
Copy Markdown
Collaborator

PipelineTokenizer benchmark

10 / 10 models supported — PipelineTokenizer vs tokenizers v0.23.1 (latest release) · whole corpus · ~10 kB chunks · cold caches · add_special_tokens on · 2/4-thread sweep per group

460b4cb94 · 2026-08-06 14:03 UTC · Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 32 cores

Per-model encode throughput vs latest release

Per-fixture encode throughput vs latest release, across models

Decode — the release encodes each fixture's decode sample (add_special_tokens=true) and both implementations decode those SAME ids with skip_special_tokens=false, so the comparison is decode alone. MB/s counts the input bytes the ids came from, the same denominator as the encode charts.

Per-model decode throughput vs latest release

vs base branch (f793dbf5c) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.

Per-model encode throughput vs base branch

Work vs base (f793dbf5c, allocation lane):

  • encode allocations: no data in the base run yet
  • decode allocations: no data in the base run yet

Encode allocations vs latest release

allocation lane measured on: Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · glibc 2.39

Per-model memory footprint

Per-model decode memory footprint

Minimal encode binary size

bert-base-uncased — normalizer-heavy WordPiece · ×4.93 vs v0.23.1 · ×0.98 vs base · decode pending bert-base-uncased speedup bert-base-uncased input-size response bert-base-uncased thread scaling (lang) bert-base-uncased thread scaling (modalities) bert-base-uncased decode speedup bert-base-uncased decode thread scaling (lang) bert-base-uncased decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 7+11+0 (peak 38) · Pipeline 4+4+— (peak 15)

Allocations (encode pass, whole corpus): v0.23.1: 159.5M allocs, 52.04 GB allocated, peak live 37 MB; Pipeline: 61.6k allocs, 0.94 GB allocated, peak live 16 MB

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Decode allocs/MB Δ base Ids
amh_Ethi lang 8.0 27.4 ×3.43 pending 660 (×1,461) ×1.01 match
arb_Arab lang 4.2 25.8 ×6.19 pending 752 (×3,636) ×0.97 match
cmn_Hani lang 3.7 19.5 ×5.23 pending 855 (×4,638) ×1.01 match
eng_Latn lang 4.4 19.1 ×4.30 pending 670 (×3,178) ×0.99 match
hin_Deva lang 6.4 28.3 ×4.46 pending 642 (×2,434) ×1.00 match
jpn_Jpan lang 4.2 27.4 ×6.50 pending 864 (×3,342) ×0.98 match
rus_Cyrl lang 3.6 25.3 ×6.97 pending 752 (×4,727) ×0.95 match
tam_Taml lang 7.0 38.4 ×5.52 pending 644 (×2,168) ×0.97 match
tha_Thai lang 8.2 32.4 ×3.95 pending 622 (×1,546) ×0.98 match
added_normalized_sparse modalities 5.2 18.4 ×3.53 pending 68,958 (×23.29) ×0.96 match
added_special_sparse modalities 3.9 21.3 ×5.45 pending 89,671 (×38.89) ×0.95 match
agentic-traces modalities 3.7 18.7 ×5.02 pending 702 (×3,726) ×1.00 match
agentic_swe modalities 3.8 19.5 ×5.08 pending 755 (×3,476) ×0.99 match
code_mixed modalities 3.8 19.3 ×5.11 pending 776 (×3,615) ×0.99 match
math_latex modalities 3.9 18.6 ×4.76 pending 661 (×3,725) ×0.98 match
deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×14.77 vs v0.23.1 · ×0.63 vs base · decode ×9.83 deepseek-v4 speedup deepseek-v4 input-size response deepseek-v4 thread scaling (lang) deepseek-v4 thread scaling (modalities) deepseek-v4 decode speedup deepseek-v4 decode thread scaling (lang) deepseek-v4 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 57+0+0 (peak 75) · Pipeline 41+0+0 (peak 88)

Allocations (encode pass, whole corpus): v0.23.1: 153.2M allocs, 38.67 GB allocated, peak live 66 MB; Pipeline: 28.0k allocs, 0.23 GB allocated, peak live 74 MB

Allocations (decode pass, decode sample): v0.23.1: 15.3M allocs, 0.77 GB allocated; Pipeline: 7.4k allocs, 0.06 GB allocated

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Decode allocs/MB Δ base Ids
amh_Ethi lang 4.3 68.2 ×15.82 ×13.55 570 (×3,582) ×0.87 match
arb_Arab lang 4.2 56.1 ×13.44 ×9.06 378 (×5,022) ×0.60 match
cmn_Hani lang 3.5 46.3 ×13.28 ×8.44 397 (×4,758) ×0.48 match
eng_Latn lang 3.0 66.6 ×21.91 ×9.65 385 (×7,925) ×0.95 match
hin_Deva lang 5.3 85.0 ×16.18 ×9.98 366 (×4,352) ×0.76 match
jpn_Jpan lang 3.9 43.5 ×11.23 ×9.16 418 (×3,717) ×0.39 match
rus_Cyrl lang 4.2 44.8 ×10.58 ×7.86 376 (×4,319) ×0.46 match
tam_Taml lang 5.7 49.5 ×8.66 ×8.10 368 (×2,674) ×0.45 match
tha_Thai lang 5.6 23.2 ×4.13 ×8.47 357 (×1,476) ×0.16 match
added_normalized_sparse modalities 5.1 82.8 ×16.22 ×12.96 490 (×5,070) ×0.73 match
added_special_sparse modalities 3.8 56.4 ×14.85 ×9.11 391 (×8,796) ×0.85 match
agentic-traces modalities 2.6 64.2 ×24.92 ×10.06 430 (×8,982) ×1.02 match
agentic_swe modalities 2.8 73.1 ×25.97 ×11.15 470 (×7,846) ×0.90 match
code_mixed modalities 2.8 65.2 ×23.29 ×12.01 467 (×7,449) ×0.87 match
math_latex modalities 2.7 60.8 ×22.57 ×9.82 387 (×9,256) ×0.90 match
gemma-4 — byte-fallback BPE, Metaspace-style split (gemma-4) · ×2.91 vs v0.23.1 · ×0.07 vs base · decode ×1.54 gemma-4 speedup gemma-4 input-size response gemma-4 thread scaling (lang) gemma-4 thread scaling (modalities) gemma-4 decode speedup gemma-4 decode thread scaling (lang) gemma-4 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 286+0+0 (peak 366) · Pipeline 1+0+0 (peak 365)

Allocations (encode pass, whole corpus): v0.23.1: 29.1M allocs, 23.69 GB allocated, peak live 310 MB; Pipeline: 65.2k allocs, 2.35 GB allocated, peak live 310 MB

Allocations (decode pass, decode sample): v0.23.1: 18.9M allocs, 1.79 GB allocated; Pipeline: 18.9M allocs, 1.79 GB allocated

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Decode allocs/MB Δ base Ids
amh_Ethi lang 11.7 47.6 ×4.07 ×1.50 780 (×521) ×0.12 match
arb_Arab lang 7.4 23.0 ×3.09 ×1.45 753 (×514) ×0.07 match
cmn_Hani lang 13.2 54.8 ×4.15 ×1.23 831 (×350) ×0.07 match
eng_Latn lang 4.0 9.1 ×2.29 ×1.69 768 (×732) ×0.05 match
hin_Deva lang 9.7 28.3 ×2.92 ×1.65 732 (×375) ×0.08 match
jpn_Jpan lang 13.5 44.5 ×3.29 ×1.22 764 (×303) ×0.06 match
rus_Cyrl lang 7.3 19.5 ×2.66 ×1.43 752 (×414) ×0.05 match
tam_Taml lang 11.3 28.3 ×2.51 ×1.44 736 (×261) ×0.05 match
tha_Thai lang 12.5 34.1 ×2.74 ×1.32 711 (×234) ×0.05 match
added_normalized_sparse modalities 4.4 11.0 ×2.52 ×1.73 881 (×659) ×0.05 match
added_special_sparse modalities 4.1 12.4 ×3.01 ×1.90 156,197 (×9.98) ×0.23 match
agentic-traces modalities 4.2 11.0 ×2.62 ×1.55 788 (×825) ×0.06 match
agentic_swe modalities 4.3 13.5 ×3.14 ×1.80 849 (×1,004) ×0.08 match
code_mixed modalities 4.3 12.2 ×2.82 ×1.74 873 (×797) ×0.07 match
math_latex modalities 4.1 9.8 ×2.40 ×1.61 740 (×792) ×0.05 match
gpt2 — gpt2 ByteLevel regex · ×15.05 vs v0.23.1 · ×0.48 vs base · decode ×30.49 gpt2 speedup gpt2 input-size response gpt2 thread scaling (lang) gpt2 thread scaling (modalities) gpt2 decode speedup gpt2 decode thread scaling (lang) gpt2 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 21+12+0 (peak 82) · Pipeline 6+4+0 (peak 84)

Allocations (encode pass, whole corpus): v0.23.1: 243.3M allocs, 38.17 GB allocated, peak live 79 MB; Pipeline: 34.6k allocs, 0.43 GB allocated, peak live 80 MB

Allocations (decode pass, decode sample): v0.23.1: 30.9M allocs, 1.36 GB allocated; Pipeline: 6.0k allocs, 0.07 GB allocated

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Decode allocs/MB Δ base Ids
amh_Ethi lang 3.7 70.3 ×18.86 ×46.42 570 (×4,658) ×0.81 match
arb_Arab lang 3.3 62.3 ×18.63 ×23.54 564 (×5,616) ×0.70 match
cmn_Hani lang 3.2 41.8 ×13.09 ×29.53 572 (×4,864) ×0.41 match
eng_Latn lang 2.9 67.7 ×23.47 ×21.08 386 (×8,670) ×0.72 match
hin_Deva lang 2.5 68.7 ×27.34 ×37.04 549 (×7,411) ×0.86 match
jpn_Jpan lang 3.7 35.4 ×9.67 ×24.96 504 (×4,683) ×0.27 match
rus_Cyrl lang 3.7 58.1 ×15.55 ×24.27 564 (×5,089) ×0.54 match
tam_Taml lang 2.3 71.7 ×31.32 ×58.39 553 (×10,027) ×0.88 match
tha_Thai lang 3.0 61.0 ×20.62 ×49.94 533 (×6,617) ×0.65 match
added_normalized_sparse modalities 4.1 8.6 ×2.09 ×30.15 490 (×6,465) ×0.06 match
added_special_sparse modalities 3.3 7.1 ×2.16 ×19.24 391 (×8,582) ×0.08 match
agentic-traces modalities 2.5 55.7 ×22.08 ×26.98 441 (×9,456) ×0.70 match
agentic_swe modalities 2.7 68.8 ×25.39 ×35.32 491 (×8,756) ×0.72 match
code_mixed modalities 2.4 63.1 ×26.02 ×32.20 496 (×8,611) ×0.71 match
math_latex modalities 2.4 57.1 ×24.28 ×23.09 406 (×9,294) ×0.67 match
gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×8.12 vs v0.23.1 · ×0.39 vs base · decode ×16.39 gpt-oss speedup gpt-oss input-size response gpt-oss thread scaling (lang) gpt-oss thread scaling (modalities) gpt-oss decode speedup gpt-oss decode thread scaling (lang) gpt-oss decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 236+0+0 (peak 310) · Pipeline 1+0+0 (peak 311)

Allocations (encode pass, whole corpus): v0.23.1: 127.8M allocs, 29.89 GB allocated, peak live 272 MB; Pipeline: 28.4k allocs, 0.23 GB allocated, peak live 272 MB

Allocations (decode pass, decode sample): v0.23.1: 14.5M allocs, 0.71 GB allocated; Pipeline: 7.4k allocs, 0.05 GB allocated

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Decode allocs/MB Δ base Ids
amh_Ethi lang 3.9 59.9 ×15.50 ×25.14 570 (×3,431) ×0.79 match
arb_Arab lang 4.4 47.4 ×10.75 ×13.03 378 (×4,522) ×0.54 match
cmn_Hani lang 4.1 24.3 ×6.00 ×14.79 474 (×2,360) ×0.25 match
eng_Latn lang 3.7 56.9 ×15.35 ×17.12 383 (×7,145) ×0.84 match
hin_Deva lang 6.1 66.7 ×10.98 ×12.01 366 (×3,680) ×0.61 match
jpn_Jpan lang 4.5 29.4 ×6.51 ×16.64 475 (×1,990) ×0.25 match
rus_Cyrl lang 4.5 35.6 ×7.84 ×10.47 376 (×3,878) ×0.36 match
tam_Taml lang 5.6 35.1 ×6.30 ×12.62 368 (×2,372) ×0.30 match
tha_Thai lang 5.5 18.8 ×3.41 ×14.27 358 (×1,345) ×0.13 match
added_normalized_sparse modalities 5.4 11.0 ×2.04 ×27.48 490 (×4,723) ×0.09 match
added_special_sparse modalities 4.6 8.7 ×1.89 ×16.24 391 (×7,345) ×0.11 match
agentic-traces modalities 3.6 51.1 ×14.28 ×18.38 414 (×7,406) ×0.76 match
agentic_swe modalities 3.7 64.7 ×17.55 ×19.61 458 (×6,524) ×0.78 match
code_mixed modalities 3.6 63.5 ×17.79 ×20.11 447 (×6,828) ×0.80 match
math_latex modalities 3.4 47.4 ×13.84 ×16.33 386 (×7,848) ×0.72 match
glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×10.97 vs v0.23.1 · ×0.52 vs base · decode ×14.48 glm-5.2 speedup glm-5.2 input-size response glm-5.2 thread scaling (lang) glm-5.2 thread scaling (modalities) glm-5.2 decode speedup glm-5.2 decode thread scaling (lang) glm-5.2 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 164+0+0 (peak 225) · Pipeline 1+0+0 (peak 226)

Allocations (encode pass, whole corpus): v0.23.1: 154.7M allocs, 33.00 GB allocated, peak live 212 MB; Pipeline: 29.5k allocs, 0.25 GB allocated, peak live 212 MB

Allocations (decode pass, decode sample): v0.23.1: 18.3M allocs, 0.90 GB allocated; Pipeline: 6.8k allocs, 0.05 GB allocated

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Decode allocs/MB Δ base Ids
amh_Ethi lang 4.3 74.4 ×17.18 ×25.61 570 (×3,814) ×0.96 match
arb_Arab lang 4.6 58.1 ×12.61 ×10.80 384 (×4,896) ×0.63 match
cmn_Hani lang 4.6 24.6 ×5.34 ×11.70 425 (×2,520) ×0.24 match
eng_Latn lang 3.9 66.6 ×17.21 ×13.32 384 (×7,146) ×0.91 match
hin_Deva lang 4.0 80.6 ×19.97 ×12.09 458 (×6,443) ×0.95 match
jpn_Jpan lang 5.0 27.3 ×5.42 ×12.83 433 (×2,153) ×0.23 match
rus_Cyrl lang 4.9 40.2 ×8.29 ×9.27 376 (×3,881) ×0.40 match
tam_Taml lang 3.9 85.6 ×21.87 ×18.68 464 (×6,456) ×0.96 match
tha_Thai lang 4.7 68.6 ×14.52 ×19.01 445 (×3,747) ×0.77 match
added_normalized_sparse modalities 4.4 12.7 ×2.87 ×18.47 490 (×4,755) ×0.10 match
added_special_sparse modalities 3.6 10.5 ×2.89 ×11.86 391 (×7,379) ×0.17 match
agentic-traces modalities 3.4 57.6 ×16.73 ×15.03 422 (×7,221) ×0.85 match
agentic_swe modalities 3.7 68.9 ×18.48 ×16.31 460 (×6,451) ×0.83 match
code_mixed modalities 3.5 59.4 ×17.21 ×15.65 446 (×6,650) ×0.75 match
math_latex modalities 3.4 55.1 ×16.00 ×13.91 390 (×7,773) ×0.81 match
llama-2 — model-bounded BPE, no pre-tokenizer · ×4.89 vs v0.23.1 · ×0.14 vs base · decode ×1.43 llama-2 speedup llama-2 input-size response llama-2 thread scaling (lang) llama-2 thread scaling (modalities) llama-2 decode speedup llama-2 decode thread scaling (lang) llama-2 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 20+6+0 (peak 63) · Pipeline 7+3+1 (peak 71)

Allocations (encode pass, whole corpus): v0.23.1: 63.0M allocs, 28.31 GB allocated, peak live 61 MB; Pipeline: 67.7k allocs, 0.56 GB allocated, peak live 72 MB

Allocations (decode pass, decode sample): v0.23.1: 35.7M allocs, 3.03 GB allocated; Pipeline: 35.7M allocs, 3.03 GB allocated

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Decode allocs/MB Δ base Ids
amh_Ethi lang 5.6 91.0 ×16.35 ×1.27 850 (×3,793) ×0.27 match
arb_Arab lang 12.2 76.3 ×6.24 ×1.36 845 (×844) ×0.23 match
cmn_Hani lang 11.0 109.6 ×9.92 ×1.24 849 (×1,468) ×0.13 match
eng_Latn lang 4.5 11.1 ×2.48 ×1.52 823 (×744) ×0.06 match
hin_Deva lang 14.1 111.4 ×7.90 ×1.38 824 (×847) ×0.31 match
jpn_Jpan lang 16.0 136.5 ×8.54 ×1.26 859 (×828) ×0.14 match
rus_Cyrl lang 9.0 29.9 ×3.34 ×1.36 753 (×505) ×0.08 match
tam_Taml lang 15.3 126.4 ×8.29 ×1.33 829 (×938) ×0.26 match
tha_Thai lang 19.0 128.9 ×6.77 ×1.29 800 (×638) ×0.20 match
added_normalized_sparse modalities 5.2 15.6 ×3.02 ×1.51 881 (×772) ×0.09 match
added_special_sparse modalities 4.7 15.9 ×3.41 ×1.78 145,162 (×11.23) ×0.33 match
agentic-traces modalities 4.8 13.5 ×2.79 ×1.45 791 (×916) ×0.07 match
agentic_swe modalities 4.5 13.8 ×3.04 ×1.68 851 (×1,059) ×0.10 match
code_mixed modalities 4.7 13.9 ×2.96 ×1.60 877 (×1,010) ×0.09 match
math_latex modalities 4.6 12.3 ×2.67 ×1.48 762 (×860) ×0.07 match
llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×13.86 vs v0.23.1 · ×0.58 vs base · decode ×19.48 llama-3 speedup llama-3 input-size response llama-3 thread scaling (lang) llama-3 thread scaling (modalities) llama-3 decode speedup llama-3 decode thread scaling (lang) llama-3 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 68+2+0 (peak 90) · Pipeline 36+0+0 (peak 103)

Allocations (encode pass, whole corpus): v0.23.1: 152.3M allocs, 33.83 GB allocated, peak live 91 MB; Pipeline: 28.7k allocs, 0.24 GB allocated, peak live 92 MB

Allocations (decode pass, decode sample): v0.23.1: 17.3M allocs, 0.88 GB allocated; Pipeline: 9.6k allocs, 0.06 GB allocated

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Decode allocs/MB Δ base Ids
amh_Ethi lang 4.3 78.6 ×18.20 ×26.96 570 (×3,818) ×1.06 match
arb_Arab lang 4.6 57.8 ×12.61 ×13.04 379 (×4,900) ×0.56 match
cmn_Hani lang 4.7 25.8 ×5.53 ×14.19 476 (×2,372) ×0.23 match
eng_Latn lang 4.0 70.2 ×17.59 ×15.78 384 (×7,149) ×0.90 match
hin_Deva lang 4.3 110.8 ×25.96 ×15.43 366 (×7,163) ×0.98 match
jpn_Jpan lang 5.0 27.2 ×5.47 ×18.95 433 (×2,151) ×0.22 match
rus_Cyrl lang 4.5 38.8 ×8.56 ×13.51 376 (×3,985) ×0.36 match
tam_Taml lang 3.6 94.5 ×25.99 ×27.68 464 (×6,460) ×0.99 match
tha_Thai lang 4.4 52.7 ×11.97 ×17.36 356 (×4,118) ×0.48 match
added_normalized_sparse modalities 4.2 22.2 ×5.28 ×30.63 490 (×4,760) ×0.16 match
added_special_sparse modalities 3.4 65.2 ×19.34 ×20.03 391 (×7,385) ×0.79 match
agentic-traces modalities 3.4 64.0 ×18.95 ×21.08 414 (×7,327) ×0.85 match
agentic_swe modalities 3.5 77.4 ×22.06 ×23.40 458 (×6,458) ×0.84 match
code_mixed modalities 3.3 67.3 ×20.31 ×24.57 446 (×6,650) ×0.76 match
math_latex modalities 3.2 58.8 ×18.27 ×19.81 388 (×7,791) ×0.79 match
mistral-small-4 — tekken byte-level BPE, 1k added specials (mistral-small-4) · ×9.44 vs v0.23.1 · ×0.50 vs base · decode ×10.74 mistral-small-4 speedup mistral-small-4 input-size response mistral-small-4 thread scaling (lang) mistral-small-4 thread scaling (modalities) mistral-small-4 decode speedup mistral-small-4 decode thread scaling (lang) mistral-small-4 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 151+0+0 (peak 189) · Pipeline 9+0+0 (peak 189)

Allocations (encode pass, whole corpus): v0.23.1: 135.2M allocs, 31.26 GB allocated, peak live 185 MB; Pipeline: 28.7k allocs, 0.24 GB allocated, peak live 185 MB

Allocations (decode pass, decode sample): v0.23.1: 16.6M allocs, 0.81 GB allocated; Pipeline: 7.1k allocs, 0.05 GB allocated

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Decode allocs/MB Δ base Ids
amh_Ethi lang 4.4 63.2 ×14.47 ×19.01 570 (×3,847) ×0.97 match
arb_Arab lang 4.9 49.1 ×10.05 ×8.94 378 (×4,557) ×0.56 match
cmn_Hani lang 4.6 32.0 ×6.97 ×11.72 476 (×2,565) ×0.35 match
eng_Latn lang 3.8 55.9 ×14.55 ×9.77 388 (×7,243) ×0.85 match
hin_Deva lang 6.2 71.5 ×11.58 ×8.42 366 (×3,846) ×0.70 match
jpn_Jpan lang 5.0 35.5 ×7.11 ×9.97 471 (×2,080) ×0.33 match
rus_Cyrl lang 4.7 39.1 ×8.35 ×7.98 376 (×4,023) ×0.41 match
tam_Taml lang 6.2 38.7 ×6.21 ×9.07 368 (×2,471) ×0.34 match
tha_Thai lang 6.2 21.7 ×3.47 ×10.25 358 (×1,515) ×0.16 match
added_normalized_sparse modalities 5.5 36.4 ×6.62 ×13.99 490 (×4,858) ×0.33 match
added_special_sparse modalities 4.5 14.2 ×3.13 ×10.44 391 (×7,459) ×0.20 match
agentic-traces modalities 3.3 56.0 ×16.84 ×11.37 438 (×7,687) ×0.91 match
agentic_swe modalities 3.4 65.3 ×19.07 ×12.26 471 (×6,995) ×0.86 match
code_mixed modalities 3.4 63.4 ×18.76 ×11.88 465 (×6,767) ×0.86 match
math_latex modalities 3.4 51.7 ×15.36 ×10.03 396 (×7,918) ×0.80 match
t5-base — Unigram + Metaspace · ×2.93 vs v0.23.1 · ×0.53 vs base · decode ×0.96 t5-base speedup t5-base input-size response t5-base thread scaling (lang) t5-base thread scaling (modalities) t5-base decode speedup t5-base decode thread scaling (lang) t5-base decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 31+10+0 (peak 56) · Pipeline 26+14+1 (peak 94)

Allocations (encode pass, whole corpus): v0.23.1: 260.7M allocs, 35.16 GB allocated, peak live 53 MB; Pipeline: 50.7M allocs, 4.73 GB allocated, peak live 107 MB

Allocations (decode pass, decode sample): v0.23.1: 12.0M allocs, 0.64 GB allocated; Pipeline: 12.0M allocs, 0.64 GB allocated

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Decode allocs/MB Δ base Ids
amh_Ethi lang 6.5 22.8 ×3.49 ×0.94 425,812 (×6.45) ×0.64 match
arb_Arab lang 5.0 18.9 ×3.74 ×0.93 563,067 (×7.24) ×0.64 match
cmn_Hani lang 10.2 16.5 ×1.61 ×0.94 933,252 (×1.44) ×0.32 match
eng_Latn lang 2.8 9.4 ×3.38 ×0.97 856,403 (×7.71) ×0.51 match
hin_Deva lang 6.7 25.8 ×3.86 ×0.94 208,773 (×15.20) ×0.78 match
jpn_Jpan lang 10.5 16.8 ×1.59 ×0.96 825,987 (×1.52) ×0.38 match
rus_Cyrl lang 4.2 12.6 ×3.00 ×0.97 962,975 (×4.39) ×0.44 match
tam_Taml lang 8.7 24.4 ×2.80 ×0.95 484,863 (×4.28) ×0.56 match
tha_Thai lang 10.6 17.9 ×1.69 ×0.95 769,838 (×1.71) ×0.39 match
added_normalized_sparse modalities 4.9 18.4 ×3.78 ×0.98 130,340 (×49.33) ×0.88 match
added_special_sparse modalities 5.0 19.1 ×3.85 ×0.99 162,215 (×32.48) ×0.97 match
agentic-traces modalities 2.9 9.3 ×3.23 ×0.98 1,041,988 (×5.73) ×0.48 match
agentic_swe modalities 3.5 10.7 ×3.08 ×0.98 863,771 (×6.10) ×0.48 match
code_mixed modalities 3.0 9.8 ×3.22 ×0.98 995,011 (×5.88) ×0.47 match
math_latex modalities 2.7 9.0 ×3.38 ×0.98 890,667 (×7.35) ×0.48 match

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Comment thread .github/workflows/pipeline-bench.yml Outdated
Comment on lines +10 to +11
# multi-thread sweep (1/2/4/8/device-max) per fixture group, per-implementation
# memory footprint (RSS), and stripped minimal-binary size. The release dep is

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Memory footprint should be HEAP ALLOCATION, its the best proxy afaik

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, allocs + instruction counts coming

SBrandeis and others added 6 commits August 6, 2026 15:10
A callgrind pass over the 64 MB corpus costs 10+ minutes per model in CI
(the release baseline runs at ~3.5 MB/s natively, ~50x slower simulated),
which pushed the bench shards past 40 minutes. The allocation lane keeps
the deterministic work tripwire and is free: it rides along the memory
children that run anyway.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The released crate is pinned, so its numbers only move when the machine
does, yet timing its few-MB/s encodes dominated every bench run. The plan
job now resolves cached baseline numbers from the HF Hub dataset, keyed on
a hash of everything that shapes them (fixture_bench.rs, the model
manifest, the Makefile fixture lists); on a hit the bench jobs run
--baseline-from and skip all release timing, re-measuring one canary
fixture per model so the report can flag runner drift (fence ±10%). On a
miss the run measures the release and the report job seeds the cache.

The shard matrix becomes one job per model (wall clock = slowest model,
jobs named after what they bench), REPS drops 5 -> 3, and the thread sweep
narrows to 2/4 threads; the thread chart's linear reference now anchors on
the sweep's first point instead of assuming a 1-thread row.

Verified locally on gpt2: cached-mode run finishes in 17 s vs ~7 min for
the measuring run, stitches the cached numbers byte-identically, passes
the id gate, and the canary drifted +1.1%.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…iming process

#2305 landed `PipelineTokenizer::decode` on the base branch, so the reason this
branch dropped the decode phase — "it's a loud stub, there is nothing to compare
yet" — no longer holds. Decode returns as phase 4, reshaped for what the branch
changed underneath it.

The release still mints the id stream both decoders replay, so decode is judged
on decode alone, but it mints ONCE per model now instead of once per lane, over
a capped prefix of each fixture (DECODE_SAMPLE_BYTES). Minting runs at release
encode speed — the slowest thing in the bench, and exactly what --baseline-from
exists to skip — while decoding is two orders of magnitude faster, so a
full-corpus stream would cost far more to prepare than to measure. Throughput is
denominated in the input bytes the ids came from: free, identical in both modes,
and on the same scale as the encode charts. --baseline-from covers decode_mbps
and decode_threads; the thread sweep is keyed per fixture group like the encode
one, and both share a generalized par_mbps. The memory children grow a decode
leg with per-fixture allocation rows, bracketed so the id buffer lands in
neither encode_bytes nor decode_bytes. WordPiece still keeps no id -> token map,
so those models report a null pipeline series ("pending") while the release's is
measured — the gap stays on the chart instead of vanishing from it.

The counting allocator is now confined to those children. Its four shared
atomics were running in the process that measures wall time, and multi-threaded
allocation traffic serializes on them hard enough to invert a scaling curve. On
gpt2, baseline MB/s at 2 -> 4 threads, counting off vs on: encode 9.5 -> 17.6 vs
4.8 -> 3.1, decode 37.5 -> 64.8 vs 13.1 -> 7.5. The pipeline's own numbers moved
by noise, because the contention scales with how often an implementation
allocates and the release allocates ~3000x more per decode — so the lane was
quietly distorting the exact comparison the sweeps exist to draw. Single-thread
numbers were unaffected; uncontended atomics are cheap.

The CI gate fails on text_match as well as ids_match, and the workflow's claim
that the decode oracle is #[ignore]d is gone: it runs, and its two WordPiece
tests fail on purpose.

Verified on gpt2: decode geomean x9.87 vs the release (352 MB/s on English,
matching #2305's own measurements), allocation counts bit-identical across two
independent runs in both lanes, cached-baseline mode stitches the decode numbers
and finishes in 26 s. bert-base-uncased exercises the pending path end to end.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@SBrandeis
SBrandeis marked this pull request as ready for review August 6, 2026 14:07
@SBrandeis
SBrandeis merged commit 16d4b93 into feat/train_encode_split Aug 6, 2026
51 of 56 checks passed
@SBrandeis
SBrandeis deleted the ci/better-benches branch August 6, 2026 14:08
ArthurZucker added a commit that referenced this pull request Sep 2, 2026
Squashed from 3 commits:
  - feat(tk-encode): implement PipelineTokenizer::decode (8-10x over the legacy path) (#2305)
  - feat(decode): wire wordpiece with a naive reverse vocab lookup (#2309)
  - iterate on CI bench (#2298)

Area: core-decode
Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants