Skip to content

feat(decode): wire wordpiece with a naive reverse vocab lookup - #2309

Merged
SBrandeis merged 2 commits into
feat/train_encode_splitfrom
feat/decoded-wordpiece
Aug 6, 2026
Merged

SBrandeis merged 2 commits into
feat/train_encode_splitfrom
feat/decoded-wordpiece

Conversation

@SBrandeis

@SBrandeis SBrandeis commented Aug 6, 2026 •

Copy link
Copy Markdown
Collaborator

PipelineTokenizer benchmark

10 / 10 models supported — PipelineTokenizer vs tokenizers v0.23.1 (latest release) · ~10 kB inputs · add_special_tokens on · single thread + 1/2/4/8/max-thread sweep

d8904fe95 · 2026-08-06 13:33 UTC · Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 96 cores

Per-model encode throughput vs latest release

vs base branch (f793dbf5c) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.

Per-model encode throughput vs base branch

Per-model memory footprint

Minimal encode binary size

Decode

Round-trip: v0.23.1 encode_fast produces the id streams (same fixtures, add_special_tokens=true); both implementations decode those SAME ids with skip_special_tokens=false. MB/s counts decoded text bytes.

Per-model decode throughput vs latest release

Per-model decode memory footprint

bert-base-uncased — normalizer-heavy WordPiece · ×5.16 vs v0.23.1 · ×1.00 vs base · decode ×1.10 bert-base-uncased speedup bert-base-uncased stage decomposition bert-base-uncased thread scaling bert-base-uncased decode speedup bert-base-uncased decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 12+0 (peak 11) · Pipeline 13+0 (peak 16)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 7.8 26.7 ×3.42 ×0.99 3% (1.2) 75% (27.4) 13% (4.8) 7% (2.5) 1% (0.5) match
arb_Arab lang 4.2 26.4 ×6.29 ×1.00 3% (1.2) 75% (27.5) 9% (3.3) 11% (4.1) 1% (0.5) match
ben_Beng lang 6.0 36.8 ×6.15 ×0.99 4% (1.2) 72% (19.1) 11% (2.9) 11% (2.9) 2% (0.5) match
cmn_Hani lang 3.8 19.3 ×5.05 ×1.00 2% (1.2) 71% (37.2) 11% (5.8) 13% (6.7) 2% (1.2) match
ell_Grek lang 3.8 26.0 ×6.93 ×1.01 3% (1.2) 74% (28.6) 9% (3.3) 12% (4.8) 2% (0.6) match
eng_Latn lang 4.5 19.1 ×4.28 ×0.99 5% (2.5) 72% (38.7) 8% (4.4) 11% (5.8) 4% (2.2) match
heb_Hebr lang 4.2 20.7 ×4.97 ×1.01 2% (1.2) 77% (37.5) 8% (3.7) 11% (5.4) 1% (0.7) match
hin_Deva lang 6.4 28.6 ×4.44 ×1.01 3% (1.2) 77% (27.0) 9% (3.2) 9% (3.3) 1% (0.5) match
jpn_Jpan lang 4.2 28.2 ×6.66 ×1.00 3% (1.2) 66% (23.2) 13% (4.6) 17% (5.8) 1% (0.3) match
kat_Geor lang 6.1 29.7 ×4.83 ×1.01 3% (1.2) 78% (26.5) 9% (3.1) 8% (2.8) 1% (0.3) match
kor_Hang lang 2.4 19.5 ×8.08 ×1.00 2% (1.2) 66% (35.1) 14% (7.3) 17% (8.9) 1% (0.3) match
rus_Cyrl lang 3.6 26.9 ×7.39 ×1.01 3% (1.2) 73% (27.4) 9% (3.2) 15% (5.6) 1% (0.3) match
tam_Taml lang 7.0 40.3 ×5.78 ×1.02 5% (1.2) 74% (18.6) 10% (2.5) 9% (2.4) 2% (0.4) match
tha_Thai lang 8.5 34.0 ×3.98 ×1.02 4% (1.2) 84% (24.9) 7% (2.2) 4% (1.1) 1% (0.4) match
added_normalized_dense modalities 6.8 20.0 ×2.96 ×1.00 3% (1.3) 79% (40.7) 13% (6.9) 2% (0.9) 4% (1.9) match
added_normalized_sparse modalities 5.3 19.2 ×3.62 ×1.00 3% (1.9) 74% (39.9) 14% (7.4) 5% (2.9) 3% (1.8) match
added_special_dense modalities 5.3 39.0 ×7.39 ×1.00 20% (5.1) 42% (10.7) 33% (8.5) 5% (1.2) 0% (0.0) match
added_special_sparse modalities 3.9 22.2 ×5.65 ×0.99 8% (3.6) 63% (28.6) 19% (8.7) 8% (3.7) 1% (0.7) match
agentic-traces modalities 3.9 18.8 ×4.85 ×1.00 4% (2.4) 70% (38.5) 9% (4.9) 13% (7.2) 4% (2.1) match
agentic_swe modalities 4.1 19.8 ×4.81 ×1.00 4% (1.9) 74% (38.7) 7% (3.7) 11% (5.8) 4% (2.1) match
code_mixed modalities 3.9 19.6 ×4.99 ×1.00 4% (2.2) 73% (38.5) 8% (4.3) 11% (5.8) 4% (2.0) match
math_latex modalities 4.0 18.9 ×4.70 ×1.00 4% (2.5) 70% (38.5) 9% (4.7) 13% (7.4) 4% (2.2) match
deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×22.51 vs v0.23.1 · ×1.04 vs base · decode ×8.37 deepseek-v4 speedup deepseek-v4 stage decomposition deepseek-v4 thread scaling deepseek-v4 decode speedup deepseek-v4 decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 62+0 (peak 67) · Pipeline 93+0 (peak 92)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 4.3 77.1 ×18.04 ×0.99 6% (0.7) 0% (0.0) 46% (4.7) 47% (4.8) 1% (0.1) match
arb_Arab lang 4.0 93.6 ×23.16 ×1.00 7% (0.6) 0% (0.0) 38% (3.4) 57% (5.1) 0% (0.0) match
ben_Beng lang 6.1 118.9 ×19.48 ×1.00 7% (0.6) 0% (0.0) 38% (3.2) 53% (4.3) 2% (0.1) match
cmn_Hani lang 3.5 95.5 ×26.91 ×0.99 6% (0.8) 0% (0.0) 25% (3.2) 65% (8.4) 4% (0.5) match
ell_Grek lang 4.5 98.1 ×21.67 ×1.02 5% (0.6) 0% (0.0) 29% (3.4) 66% (7.9) 0% (0.0) match
eng_Latn lang 3.1 70.3 ×22.89 ×1.00 16% (2.0) 0% (0.0) 41% (5.1) 42% (5.2) 1% (0.1) match
heb_Hebr lang 3.9 90.7 ×22.98 ×1.01 4% (0.6) 0% (0.0) 24% (3.6) 72% (11.0) 0% (0.1) match
hin_Deva lang 5.8 113.6 ×19.51 ×1.02 7% (0.6) 0% (0.0) 38% (3.3) 55% (4.9) 0% (0.0) match
jpn_Jpan lang 4.3 114.2 ×26.85 ×1.02 6% (0.7) 0% (0.0) 27% (3.0) 69% (7.8) 0% (0.0) match
kat_Geor lang 5.9 120.3 ×20.41 ×1.02 5% (0.6) 0% (0.0) 27% (3.0) 68% (7.6) 0% (0.0) match
kor_Hang lang 3.7 78.4 ×21.32 ×1.02 3% (0.6) 0% (0.0) 20% (3.7) 76% (13.8) 0% (0.0) match
rus_Cyrl lang 4.5 110.0 ×24.62 ×1.13 4% (0.6) 0% (0.0) 22% (3.3) 73% (10.9) 0% (0.0) match
tam_Taml lang 6.3 144.5 ×22.94 ×1.31 6% (0.6) 0% (0.0) 26% (2.7) 68% (7.2) 0% (0.1) match
tha_Thai lang 7.1 190.5 ×27.02 ×1.29 6% (0.6) 0% (0.0) 21% (2.3) 73% (7.8) 0% (0.0) match
added_normalized_dense modalities 5.5 146.0 ×26.37 ×1.01 11% (0.8) 0% (0.0) 42% (2.8) 46% (3.1) 1% (0.1) match
added_normalized_sparse modalities 5.0 114.0 ×22.62 ×1.01 15% (1.3) 0% (0.0) 45% (3.8) 40% (3.3) 1% (0.1) match
added_special_dense modalities 3.4 59.4 ×17.45 ×0.98 42% (6.6) 5% (0.8) 38% (6.0) 14% (2.2) 1% (0.2) match
added_special_sparse modalities 3.8 67.3 ×17.71 ×1.02 27% (3.9) 2% (0.3) 49% (7.0) 20% (2.9) 2% (0.2) match
agentic-traces modalities 3.0 68.1 ×22.79 ×1.08 13% (1.8) 0% (0.0) 40% (5.4) 46% (6.2) 1% (0.1) match
agentic_swe modalities 3.0 86.2 ×28.75 ×1.06 13% (1.3) 0% (0.0) 37% (3.9) 49% (5.1) 1% (0.1) match
code_mixed modalities 3.5 78.8 ×22.51 ×1.05 13% (1.6) 0% (0.0) 40% (4.6) 46% (5.3) 1% (0.1) match
math_latex modalities 2.9 69.0 ×23.85 ×1.02 15% (1.9) 0% (0.0) 42% (5.4) 43% (5.5) 1% (0.1) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.63 3.13 4.75 5.25 41.8 20.2 9.2 — 8.8× / 8.0× 4.3× / 3.8× 1.9× / 1.7× —
arb_Arab 1.00 2.67 3.44 5.11 49.7 23.3 10.3 — 14.4× / 9.7× 6.8× / 4.6× 3.0× / 2.0× —
ben_Beng 1.46 2.70 3.16 4.40 36.0 16.1 7.5 — 11.4× / 8.2× 5.1× / 3.7× 2.4× / 1.7× —
cmn_Hani 1.13 2.08 3.24 4.19 59.5 33.8 14.2 — 18.4× / 14.2× 10.4× / 8.1× 4.4× / 3.4× —
ell_Grek 0.58 2.68 3.44 5.53 47.4 21.1 9.5 — 13.8× / 8.6× 6.1× / 3.8× 2.8× / 1.7× —
eng_Latn 0.10 0.89 5.10 5.90 65.4 39.3 16.3 — 12.8× / 11.1× 7.7× / 6.7× 3.2× / 2.8× —
heb_Hebr 1.00 2.74 3.60 5.33 51.1 24.2 10.8 — 14.2× / 9.6× 6.7× / 4.5× 3.0× / 2.0× —
hin_Deva 1.36 2.83 3.35 4.82 37.9 18.6 8.6 — 11.3× / 7.9× 5.6× / 3.9× 2.6× / 1.8× —
jpn_Jpan 1.56 3.22 3.04 4.70 52.7 27.5 11.7 — 17.3× / 11.2× 9.0× / 5.9× 3.9× / 2.5× —
kat_Geor 1.39 2.24 2.98 3.83 32.5 15.1 7.2 — 10.9× / 8.5× 5.1× / 3.9× 2.4× / 1.9× —
kor_Hang 1.08 2.61 3.69 5.22 50.2 27.0 11.9 — 13.6× / 9.6× 7.3× / 5.2× 3.2× / 2.3× —
rus_Cyrl 1.02 2.63 3.34 4.96 47.7 20.9 9.5 — 14.3× / 9.6× 6.2× / 4.2× 2.8× / 1.9× —
tam_Taml 0.92 2.68 2.75 4.51 31.7 13.7 6.6 — 11.5× / 7.0× 5.0× / 3.0× 2.4× / 1.5× —
tha_Thai 1.35 2.32 2.27 3.25 27.4 10.1 5.2 — 12.1× / 8.4× 4.4× / 3.1× 2.3× / 1.6× —
added_normalized_dense 0.06 0.88 2.77 3.60 42.4 19.9 9.2 — 15.3× / 11.8× 7.2× / 5.5× 3.3× / 2.5× —
added_normalized_sparse 0.07 0.88 3.77 4.58 48.9 25.6 11.6 — 13.0× / 10.7× 6.8× / 5.6× 3.1× / 2.5× —
added_special_dense 0.06 0.88 6.00 6.83 170.1 97.0 38.2 — 28.3× / 24.9× 16.2× / 14.2× 6.4× / 5.6× —
added_special_sparse 0.06 0.88 6.98 7.81 100.7 57.3 24.2 — 14.4× / 12.9× 8.2× / 7.3× 3.5× / 3.1× —
agentic-traces 0.57 0.92 5.39 5.75 79.2 53.2 20.4 — 14.7× / 13.8× 9.9× / 9.3× 3.8× / 3.5× —
agentic_swe 0.59 0.86 3.86 4.13 95.3 65.2 23.5 — 24.7× / 23.1× 16.9× / 15.8× 6.1× / 5.7× —
code_mixed 0.08 0.86 4.65 5.43 73.0 53.7 18.5 — 15.7× / 13.4× 11.6× / 9.9× 4.0× / 3.4× —
math_latex 0.56 0.93 5.42 5.79 76.9 47.2 19.3 — 14.2× / 13.3× 8.7× / 8.1× 3.6× / 3.3× —
gemma-4 — byte-fallback BPE, Metaspace-style split (gemma-4) · ×38.91 vs v0.23.1 · ×0.99 vs base · decode ×1.50 gemma-4 speedup gemma-4 stage decomposition gemma-4 thread scaling gemma-4 decode speedup gemma-4 decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 304+0 (peak 370) · Pipeline 274+0 (peak 370)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 10.4 388.1 ×37.17 ×0.99 29% (0.7) 55% (1.3) 1% (0.0) 16% (0.4) 0% (0.0) match
arb_Arab lang 7.1 350.2 ×49.31 ×1.01 24% (0.6) 61% (1.6) 1% (0.0) 14% (0.4) 0% (0.0) match
ben_Beng lang 9.9 463.5 ×46.89 ×0.99 31% (0.6) 52% (1.0) 1% (0.0) 16% (0.3) 0% (0.0) match
cmn_Hani lang 12.4 719.7 ×57.99 ×0.98 53% (0.6) 17% (0.2) 2% (0.0) 30% (0.3) 0% (0.0) match
ell_Grek lang 7.7 345.9 ×45.00 ×1.01 24% (0.6) 62% (1.6) 1% (0.0) 14% (0.4) 0% (0.0) match
eng_Latn lang 3.9 178.0 ×46.05 ×1.00 36% (2.0) 56% (3.0) 1% (0.0) 8% (0.4) 0% (0.0) match
heb_Hebr lang 8.1 334.6 ×41.53 ×0.95 23% (0.6) 61% (1.6) 1% (0.0) 15% (0.4) 0% (0.0) match
hin_Deva lang 9.8 377.9 ×38.75 ×1.01 26% (0.6) 60% (1.4) 1% (0.0) 14% (0.3) 0% (0.0) match
jpn_Jpan lang 12.5 763.3 ×60.95 ×0.96 55% (0.6) 15% (0.2) 2% (0.0) 30% (0.3) 0% (0.0) match
kat_Geor lang 11.9 493.4 ×41.34 ×0.98 33% (0.6) 49% (0.9) 1% (0.0) 18% (0.3) 0% (0.0) match
kor_Hang lang 9.4 353.7 ×37.46 ×1.01 24% (0.6) 61% (1.6) 1% (0.0) 15% (0.4) 0% (0.0) match
rus_Cyrl lang 7.1 380.5 ×53.65 ×1.02 25% (0.6) 60% (1.4) 1% (0.0) 15% (0.4) 0% (0.0) match
tam_Taml lang 11.0 524.2 ×47.68 ×0.99 35% (0.6) 47% (0.8) 1% (0.0) 18% (0.3) 0% (0.0) match
tha_Thai lang 13.5 639.8 ×47.42 ×0.99 44% (0.6) 33% (0.5) 2% (0.0) 22% (0.3) 0% (0.0) match
added_normalized_dense modalities 5.2 313.0 ×59.93 ×0.99 26% (0.8) 61% (1.8) 1% (0.0) 13% (0.4) 0% (0.0) match
added_normalized_sparse modalities 4.6 206.0 ×44.43 ×0.98 29% (1.3) 63% (2.8) 0% (0.0) 10% (0.4) 0% (0.0) match
added_special_dense modalities 4.7 37.7 ×7.96 ×0.97 46% (11.3) 21% (5.2) 22% (5.5) 12% (2.9) 0% (0.0) match
added_special_sparse modalities 7.2 49.9 ×6.97 ×0.94 29% (5.2) 38% (6.8) 23% (4.1) 12% (2.1) 0% (0.0) match
agentic-traces modalities 4.3 193.4 ×44.64 ×1.01 36% (1.8) 55% (2.7) 1% (0.0) 10% (0.5) 0% (0.0) match
agentic_swe modalities 4.0 164.6 ×40.88 ×1.00 22% (1.3) 69% (4.0) 0% (0.0) 9% (0.5) 0% (0.0) match
code_mixed modalities 4.1 175.0 ×42.19 ×1.00 28% (1.6) 63% (3.4) 1% (0.0) 9% (0.5) 0% (0.0) match
math_latex modalities 4.2 185.1 ×44.24 ×1.00 36% (1.9) 56% (2.9) 1% (0.0) 8% (0.4) 0% (0.0) match
gpt2 — gpt2 ByteLevel regex · ×25.57 vs v0.23.1 · ×0.98 vs base · decode ×16.51 gpt2 speedup gpt2 stage decomposition gpt2 thread scaling gpt2 decode speedup gpt2 decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 25+2 (peak 27) · Pipeline 32+0 (peak 32)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 3.9 86.2 ×21.95 ×1.00 7% (0.7) 0% (0.0) 39% (3.6) 56% (5.2) 0% (0.0) match
arb_Arab lang 3.7 88.5 ×23.63 ×0.99 7% (0.6) 0% (0.0) 25% (2.3) 69% (6.3) 0% (0.0) match
ben_Beng lang 2.9 70.4 ×24.53 ×0.99 5% (0.6) 0% (0.0) 23% (3.0) 72% (9.7) 1% (0.1) match
cmn_Hani lang 3.7 101.1 ×27.26 ×1.00 5% (0.6) 0% (0.0) 18% (2.3) 91% (11.9) 0% (0.0) match
ell_Grek lang 4.2 100.0 ×23.91 ×0.96 6% (0.6) 0% (0.0) 21% (2.2) 75% (7.6) 0% (0.0) match
eng_Latn lang 3.4 93.0 ×27.51 ×0.99 19% (2.0) 0% (0.0) 29% (2.9) 52% (5.3) 0% (0.0) match
heb_Hebr lang 3.8 93.7 ×24.69 ×0.97 5% (0.6) 0% (0.0) 19% (2.3) 76% (9.4) 0% (0.1) match
hin_Deva lang 3.0 79.5 ×26.08 ×0.99 5% (0.6) 0% (0.0) 25% (3.1) 69% (8.4) 0% (0.0) match
jpn_Jpan lang 4.5 127.6 ×28.46 ×0.97 5% (0.6) 0% (0.0) 17% (2.1) 82% (10.1) 0% (0.0) match
kat_Geor lang 4.6 132.3 ×28.61 ×0.97 9% (0.6) 0% (0.0) 29% (2.1) 64% (4.5) 0% (0.0) match
kor_Hang lang 3.3 83.3 ×25.11 ×0.95 5% (0.6) 0% (0.0) 19% (2.4) 82% (10.6) 0% (0.0) match
rus_Cyrl lang 4.1 104.5 ×25.39 ×0.97 6% (0.6) 0% (0.0) 19% (2.1) 76% (8.3) 0% (0.0) match
tam_Taml lang 2.7 80.5 ×30.05 ×0.98 5% (0.6) 0% (0.0) 23% (2.7) 71% (8.5) 1% (0.1) match
tha_Thai lang 3.5 92.5 ×26.51 ×0.98 6% (0.6) 0% (0.0) 23% (2.5) 76% (8.5) 0% (0.0) match
added_normalized_dense modalities 5.8 177.0 ×30.68 ×1.02 14% (0.7) 0% (0.0) 20% (1.1) 64% (3.3) 2% (0.1) match
added_normalized_sparse modalities 5.0 134.6 ×27.18 ×0.97 18% (1.3) 0% (0.0) 28% (2.0) 53% (3.7) 2% (0.1) match
added_special_dense modalities 4.0 78.8 ×19.66 ×0.94 42% (5.0) 5% (0.6) 33% (3.8) 18% (2.1) 2% (0.2) match
added_special_sparse modalities 4.1 81.0 ×19.78 ×0.95 28% (3.2) 2% (0.2) 39% (4.5) 29% (3.3) 2% (0.3) match
agentic-traces modalities 3.1 78.3 ×25.47 ×0.99 14% (1.8) 0% (0.0) 28% (3.5) 58% (7.2) 0% (0.0) match
agentic_swe modalities 3.3 94.1 ×28.22 ×0.99 13% (1.3) 0% (0.0) 25% (2.4) 62% (6.1) 1% (0.1) match
code_mixed modalities 3.5 87.7 ×25.03 ×0.99 15% (1.6) 0% (0.0) 27% (2.9) 58% (6.2) 1% (0.1) match
math_latex modalities 3.2 83.6 ×26.43 ×0.98 17% (1.9) 0% (0.0) 30% (3.3) 53% (5.9) 0% (0.0) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.62 3.17 3.63 4.19 28.4 21.4 5.8 4.9 7.8× / 6.8× 5.9× / 5.1× 1.6× / 1.4× 1.3× / 1.2×
arb_Arab 1.01 2.68 2.27 3.93 33.1 26.5 6.7 5.2 14.6× / 8.4× 11.7× / 6.7× 3.0× / 1.7× 2.3× / 1.3×
ben_Beng 1.47 2.71 3.03 4.27 68.3 57.4 13.8 4.0 22.5× / 16.0× 18.9× / 13.4× 4.6× / 3.2× 1.3× / 0.9×
cmn_Hani 1.13 2.11 2.33 3.31 26.8 21.1 5.9 2.4 11.5× / 8.1× 9.1× / 6.4× 2.5× / 1.8× 1.0× / 0.7×
ell_Grek 0.58 2.73 2.18 4.33 29.3 22.5 5.9 4.8 13.4× / 6.8× 10.3× / 5.2× 2.7× / 1.4× 2.2× / 1.1×
eng_Latn 0.12 0.89 2.93 3.70 46.2 43.0 12.1 4.0 15.8× / 12.5× 14.7× / 11.6× 4.1× / 3.3× 1.4× / 1.1×
heb_Hebr 1.00 2.76 2.30 4.07 32.8 26.7 7.0 3.0 14.2× / 8.0× 11.6× / 6.6× 3.0× / 1.7× 1.3× / 0.7×
hin_Deva 1.36 2.84 3.10 4.58 64.3 55.8 13.4 4.3 20.7× / 14.0× 18.0× / 12.2× 4.3× / 2.9× 1.4× / 0.9×
jpn_Jpan 1.57 3.25 2.13 3.81 23.9 17.6 5.0 3.9 11.2× / 6.3× 8.2× / 4.6× 2.3× / 1.3× 1.8× / 1.0×
kat_Geor 1.38 2.23 2.05 2.90 17.9 14.4 4.1 2.0 8.7× / 6.2× 7.0× / 5.0× 2.0× / 1.4× 1.0× / 0.7×
kor_Hang 1.08 2.63 2.44 3.99 32.8 28.5 7.6 3.6 13.5× / 8.2× 11.7× / 7.1× 3.1× / 1.9× 1.5× / 0.9×
rus_Cyrl 1.02 2.64 2.10 3.73 27.9 21.7 5.8 2.6 13.3× / 7.5× 10.3× / 5.8× 2.8× / 1.6× 1.2× / 0.7×
tam_Taml 0.92 2.70 2.70 4.47 71.6 59.7 14.3 3.8 26.5× / 16.0× 22.1× / 13.3× 5.3× / 3.2× 1.4× / 0.9×
tha_Thai 1.35 2.34 2.53 3.51 41.0 32.4 8.8 3.2 16.2× / 11.7× 12.8× / 9.2× 3.5× / 2.5× 1.3× / 0.9×
added_normalized_dense 0.06 0.88 1.05 1.88 24.4 23.1 6.4 2.0 23.2× / 13.0× 22.0× / 12.3× 6.1× / 3.4× 1.9× / 1.1×
added_normalized_sparse 0.06 0.88 1.95 2.78 32.3 31.4 8.6 2.9 16.6× / 11.6× 16.1× / 11.3× 4.4× / 3.1× 1.5× / 1.0×
added_special_dense 0.06 0.88 3.85 4.67 97.9 97.7 19.9 3.1 25.5× / 21.0× 25.4× / 20.9× 5.2× / 4.3× 0.8× / 0.7×
added_special_sparse 0.06 0.88 4.45 5.28 63.4 62.3 14.5 3.6 14.2× / 12.0× 14.0× / 11.8× 3.3× / 2.7× 0.8× / 0.7×
agentic-traces 0.57 0.94 3.49 3.85 61.3 61.3 15.4 4.9 17.6× / 15.9× 17.6× / 15.9× 4.4× / 4.0× 1.4× / 1.3×
agentic_swe 0.54 0.87 2.42 2.75 62.5 70.0 14.7 3.7 25.8× / 22.7× 28.9× / 25.4× 6.0× / 5.3× 1.5× / 1.3×
code_mixed 0.07 0.87 2.91 3.71 60.9 67.9 15.4 4.2 20.9× / 16.4× 23.3× / 18.3× 5.3× / 4.2× 1.4× / 1.1×
math_latex 0.59 0.94 3.28 3.63 54.5 52.1 13.9 4.4 16.6× / 15.0× 15.9× / 14.3× 4.2× / 3.8× 1.3× / 1.2×
gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×18.98 vs v0.23.1 · ×0.98 vs base · decode ×10.23 gpt-oss speedup gpt-oss stage decomposition gpt-oss thread scaling gpt-oss decode speedup gpt-oss decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 241+0 (peak 313) · Pipeline 234+0 (peak 316)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 4.0 74.7 ×18.88 ×0.98 6% (0.7) 0% (0.0) 47% (5.0) 49% (5.1) 0% (0.0) match
arb_Arab lang 4.5 87.4 ×19.30 ×0.99 6% (0.6) 0% (0.0) 39% (3.8) 57% (5.6) 0% (0.0) match
ben_Beng lang 6.6 107.1 ×16.17 ×0.99 7% (0.6) 0% (0.0) 43% (3.9) 52% (4.7) 0% (0.0) match
cmn_Hani lang 4.6 97.0 ×21.04 ×0.98 3% (0.6) 0% (0.0) 16% (3.3) 88% (18.5) 0% (0.0) match
ell_Grek lang 5.0 97.3 ×19.35 ×0.98 5% (0.6) 0% (0.0) 27% (3.3) 71% (8.8) 0% (0.0) match
eng_Latn lang 3.9 67.1 ×17.23 ×0.99 16% (2.0) 0% (0.0) 36% (4.6) 48% (6.1) 0% (0.0) match
heb_Hebr lang 4.4 86.2 ×19.45 ×0.99 4% (0.6) 0% (0.0) 25% (3.8) 73% (11.0) 0% (0.0) match
hin_Deva lang 6.8 105.2 ×15.55 ×0.96 6% (0.6) 0% (0.0) 43% (4.0) 51% (4.8) 0% (0.0) match
jpn_Jpan lang 5.4 112.0 ×20.64 ×0.96 4% (0.6) 0% (0.0) 20% (3.1) 84% (12.9) 0% (0.0) match
kat_Geor lang 6.7 119.8 ×17.93 ×0.94 5% (0.6) 0% (0.0) 24% (2.9) 78% (9.6) 0% (0.0) match
kor_Hang lang 4.0 72.3 ×18.06 ×0.98 3% (0.6) 0% (0.0) 18% (3.9) 81% (17.4) 0% (0.0) match
rus_Cyrl lang 5.0 96.6 ×19.22 ×0.97 4% (0.6) 0% (0.0) 20% (3.2) 78% (12.4) 0% (0.0) match
tam_Taml lang 6.8 117.1 ×17.21 ×0.99 5% (0.6) 0% (0.0) 26% (3.4) 71% (9.3) 0% (0.0) match
tha_Thai lang 7.9 136.9 ×17.41 ×0.98 5% (0.6) 0% (0.0) 25% (3.4) 73% (9.8) 0% (0.0) match
added_normalized_dense modalities 5.3 151.6 ×28.55 ×1.00 12% (0.8) 0% (0.0) 36% (2.3) 51% (3.2) 1% (0.0) match
added_normalized_sparse modalities 5.3 118.6 ×22.16 ×0.99 16% (1.3) 0% (0.0) 39% (3.2) 45% (3.6) 2% (0.1) match
added_special_dense modalities 4.3 73.2 ×17.14 ×0.98 39% (5.1) 3% (0.4) 37% (4.8) 20% (2.6) 1% (0.1) match
added_special_sparse modalities 4.4 74.2 ×16.69 ×0.95 25% (3.2) 2% (0.2) 48% (6.1) 24% (3.1) 1% (0.1) match
agentic-traces modalities 3.6 66.6 ×18.36 ×0.99 13% (1.8) 0% (0.0) 37% (5.0) 51% (6.9) 0% (0.0) match
agentic_swe modalities 3.7 82.7 ×22.12 ×0.99 12% (1.3) 0% (0.0) 35% (3.7) 53% (5.6) 0% (0.0) match
code_mixed modalities 3.9 78.6 ×20.04 ×0.99 14% (1.6) 0% (0.0) 39% (4.4) 48% (5.4) 0% (0.0) match
math_latex modalities 3.5 65.0 ×18.73 ×0.98 13% (1.9) 0% (0.0) 34% (4.9) 54% (7.8) 0% (0.0) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.61 3.16 4.96 5.51 30.0 14.1 7.0 4.7 6.0× / 5.4× 2.8× / 2.6× 1.4× / 1.3× 0.9× / 0.9×
arb_Arab 1.02 2.66 3.80 5.45 34.4 16.4 7.5 5.0 9.0× / 6.3× 4.3× / 3.0× 2.0× / 1.4× 1.3× / 0.9×
ben_Beng 1.45 2.70 3.91 5.15 24.0 10.7 5.4 2.9 6.2× / 4.7× 2.7× / 2.1× 1.4× / 1.0× 0.7× / 0.6×
cmn_Hani 1.13 2.08 3.30 4.25 21.8 10.8 5.4 2.4 6.6× / 5.1× 3.3× / 2.5× 1.6× / 1.3× 0.7× / 0.6×
ell_Grek 0.58 2.67 3.29 5.38 30.3 15.6 6.9 5.1 9.2× / 5.6× 4.7× / 2.9× 2.1× / 1.3× 1.5× / 0.9×
eng_Latn 0.11 0.89 4.57 5.35 44.3 29.6 13.6 4.1 9.7× / 8.3× 6.5× / 5.5× 3.0× / 2.5× 0.9× / 0.8×
heb_Hebr 1.00 2.75 3.85 5.60 33.7 17.9 8.0 2.9 8.8× / 6.0× 4.6× / 3.2× 2.1× / 1.4× 0.8× / 0.5×
hin_Deva 1.37 2.85 4.03 5.51 26.1 12.8 6.3 3.1 6.5× / 4.7× 3.2× / 2.3× 1.6× / 1.2× 0.8× / 0.6×
jpn_Jpan 1.56 3.24 3.08 4.76 20.3 9.3 4.7 3.7 6.6× / 4.3× 3.0× / 1.9× 1.5× / 1.0× 1.2× / 0.8×
kat_Geor 1.40 2.24 2.91 3.75 18.9 10.1 4.6 2.1 6.5× / 5.0× 3.5× / 2.7× 1.6× / 1.2× 0.7× / 0.6×
kor_Hang 1.09 2.63 3.89 5.43 33.5 19.1 8.7 3.6 8.6× / 6.2× 4.9× / 3.5× 2.2× / 1.6× 0.9× / 0.7×
rus_Cyrl 1.02 2.63 3.22 4.84 29.1 15.4 6.6 4.8 9.0× / 6.0× 4.8× / 3.2× 2.1× / 1.4× 1.5× / 1.0×
tam_Taml 0.92 2.77 3.44 5.29 19.0 8.8 4.2 3.0 5.5× / 3.6× 2.6× / 1.7× 1.2× / 0.8× 0.9× / 0.6×
tha_Thai 1.35 2.33 3.42 4.40 13.1 5.7 2.8 2.2 3.8× / 3.0× 1.7× / 1.3× 0.8× / 0.6× 0.7× / 0.5×
added_normalized_dense 0.06 0.88 2.28 3.10 34.4 17.9 11.3 2.0 15.1× / 11.1× 7.9× / 5.8× 5.0× / 3.7× 0.9× / 0.7×
added_normalized_sparse 0.06 0.88 3.16 3.98 36.7 20.8 11.4 2.9 11.6× / 9.2× 6.6× / 5.2× 3.6× / 2.9× 0.9× / 0.7×
added_special_dense 0.06 0.88 4.83 5.66 87.1 67.7 24.0 3.4 18.0× / 15.4× 14.0× / 12.0× 5.0× / 4.2× 0.7× / 0.6×
added_special_sparse 0.06 0.88 6.08 6.91 58.5 41.4 17.1 3.6 9.6× / 8.5× 6.8× / 6.0× 2.8× / 2.5× 0.6× / 0.5×
agentic-traces 0.57 0.92 4.98 5.33 53.8 42.5 17.1 4.9 10.8× / 10.1× 8.5× / 8.0× 3.4× / 3.2× 1.0× / 0.9×
agentic_swe 0.56 0.86 3.71 4.01 54.3 49.8 17.4 3.5 14.6× / 13.5× 13.4× / 12.4× 4.7× / 4.3× 0.9× / 0.9×
code_mixed 0.07 0.86 4.37 5.16 53.9 48.1 17.2 4.2 12.4× / 10.5× 11.0× / 9.3× 3.9× / 3.3× 1.0× / 0.8×
math_latex 0.58 0.92 4.85 5.19 51.3 35.9 15.7 4.5 10.6× / 9.9× 7.4× / 6.9× 3.2× / 3.0× 0.9× / 0.9×
glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×19.67 vs v0.23.1 · ×1.00 vs base · decode ×12.05 glm-5.2 speedup glm-5.2 stage decomposition glm-5.2 thread scaling glm-5.2 decode speedup glm-5.2 decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 169+0 (peak 230) · Pipeline 170+0 (peak 231)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 4.1 79.0 ×19.21 ×1.02 12% (1.2) 0% (0.0) 37% (3.7) 54% (5.4) 0% (0.0) match
arb_Arab lang 4.6 94.1 ×20.34 ×1.02 13% (1.2) 0% (0.0) 26% (2.3) 63% (5.7) 0% (0.0) match
ben_Beng lang 4.3 89.5 ×20.96 ×1.01 11% (1.2) 0% (0.0) 30% (3.1) 59% (6.2) 0% (0.0) match
cmn_Hani lang 4.9 102.5 ×20.80 ×1.01 6% (1.2) 0% (0.0) 12% (2.4) 79% (16.3) 4% (0.8) match
ell_Grek lang 5.2 102.5 ×19.75 ×1.00 9% (1.2) 0% (0.0) 18% (2.2) 75% (9.3) 0% (0.0) match
eng_Latn lang 3.8 72.3 ×18.91 ×0.99 22% (2.5) 0% (0.0) 26% (3.0) 57% (6.6) 0% (0.0) match
heb_Hebr lang 4.2 82.8 ×19.53 ×0.98 9% (1.2) 0% (0.0) 18% (2.3) 77% (10.1) 0% (0.0) match
hin_Deva lang 3.9 84.9 ×21.50 ×1.01 10% (1.2) 0% (0.0) 29% (3.2) 61% (6.9) 0% (0.0) match
jpn_Jpan lang 5.7 118.3 ×20.67 ×1.01 7% (1.2) 0% (0.0) 12% (2.2) 83% (14.7) 0% (0.0) match
kat_Geor lang 6.4 116.1 ×18.08 ×0.97 12% (1.2) 0% (0.0) 22% (2.1) 70% (6.7) 0% (0.0) match
kor_Hang lang 3.9 74.3 ×18.87 ×1.00 6% (1.2) 0% (0.0) 12% (2.5) 85% (17.7) 0% (0.0) match
rus_Cyrl lang 4.9 101.1 ×20.56 ×1.01 7% (1.2) 0% (0.0) 14% (2.2) 77% (12.4) 2% (0.4) match
tam_Taml lang 3.9 89.8 ×23.32 ×1.00 11% (1.2) 0% (0.0) 26% (2.7) 64% (6.8) 0% (0.0) match
tha_Thai lang 5.0 91.6 ×18.51 ×1.03 11% (1.2) 0% (0.0) 23% (2.6) 73% (8.2) 0% (0.0) match
added_normalized_dense modalities 6.0 168.6 ×28.23 ×1.01 24% (1.3) 0% (0.0) 19% (1.1) 54% (3.0) 3% (0.1) match
added_normalized_sparse modalities 5.5 129.9 ×23.80 ×1.01 25% (1.9) 0% (0.0) 27% (2.0) 47% (3.5) 2% (0.1) match
added_special_dense modalities 4.1 45.1 ×10.87 ×0.94 59% (12.8) 2% (0.4) 25% (5.4) 12% (2.7) 1% (0.3) match
added_special_sparse modalities 4.3 60.9 ×14.03 ×0.96 45% (7.0) 1% (0.1) 33% (5.2) 21% (3.3) 0% (0.0) match
agentic-traces modalities 3.5 69.2 ×19.84 ×1.01 18% (2.5) 0% (0.0) 28% (3.8) 55% (7.3) 0% (0.0) match
agentic_swe modalities 3.8 84.3 ×22.46 ×1.01 19% (2.0) 0% (0.0) 28% (2.9) 54% (5.7) 0% (0.0) match
code_mixed modalities 3.9 81.6 ×20.86 ×1.03 20% (2.2) 0% (0.0) 31% (3.3) 50% (5.4) 0% (0.0) match
math_latex modalities 3.8 68.7 ×18.30 ×1.02 18% (2.5) 0% (0.0) 25% (3.5) 56% (8.0) 2% (0.2) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.63 3.14 3.67 4.19 27.3 16.8 6.0 4.8 7.4× / 6.5× 4.6× / 4.0× 1.6× / 1.4× 1.3× / 1.2×
arb_Arab 1.00 2.68 2.35 4.03 31.4 19.7 6.9 5.2 13.4× / 7.8× 8.4× / 4.9× 2.9× / 1.7× 2.2× / 1.3×
ben_Beng 1.46 2.70 3.15 4.39 45.5 28.1 10.3 3.8 14.5× / 10.4× 8.9× / 6.4× 3.3× / 2.3× 1.2× / 0.9×
cmn_Hani 1.13 2.08 2.41 3.35 19.5 12.0 4.7 2.4 8.1× / 5.8× 5.0× / 3.6× 2.0× / 1.4× 1.0× / 0.7×
ell_Grek 0.59 2.67 2.24 4.32 28.5 17.8 6.2 4.8 12.7× / 6.6× 7.9× / 4.1× 2.8× / 1.4× 2.1× / 1.1×
eng_Latn 0.10 0.89 3.05 3.85 45.1 34.4 12.6 3.8 14.8× / 11.7× 11.3× / 8.9× 4.1× / 3.3× 1.3× / 1.0×
heb_Hebr 1.00 2.74 2.34 4.09 31.5 20.9 7.1 3.0 13.4× / 7.7× 8.9× / 5.1× 3.0× / 1.7× 1.3× / 0.7×
hin_Deva 1.36 2.84 3.25 4.73 48.0 31.7 11.3 4.1 14.8× / 10.1× 9.8× / 6.7× 3.5× / 2.4× 1.3× / 0.9×
jpn_Jpan 1.57 3.23 2.20 3.86 18.1 10.1 4.1 3.8 8.2× / 4.7× 4.6× / 2.6× 1.9× / 1.1× 1.7× / 1.0×
kat_Geor 1.39 2.22 2.11 2.95 17.3 11.3 4.2 2.1 8.2× / 5.9× 5.4× / 3.8× 2.0× / 1.4× 1.0× / 0.7×
kor_Hang 1.08 2.54 2.53 4.00 31.5 22.4 7.7 3.6 12.4× / 7.9× 8.8× / 5.6× 3.0× / 1.9× 1.4× / 0.9×
rus_Cyrl 1.02 2.63 2.20 3.81 27.1 17.3 6.0 2.5 12.3× / 7.1× 7.8× / 4.5× 2.7× / 1.6× 1.1× / 0.6×
tam_Taml 0.92 2.68 2.74 4.50 44.8 27.8 9.9 3.5 16.4× / 10.0× 10.2× / 6.2× 3.6× / 2.2× 1.3× / 0.8×
tha_Thai 1.36 2.33 2.62 3.59 28.4 17.0 6.8 3.0 10.9× / 7.9× 6.5× / 4.7× 2.6× / 1.9× 1.2× / 0.8×
added_normalized_dense 0.06 0.88 1.08 1.91 24.0 17.2 6.6 2.0 22.2× / 12.6× 15.9× / 9.0× 6.1× / 3.5× 1.8× / 1.0×
added_normalized_sparse 0.06 0.88 1.97 2.79 31.8 22.5 8.9 2.7 16.2× / 11.4× 11.5× / 8.1× 4.5× / 3.2× 1.4× / 1.0×
added_special_dense 0.06 0.88 5.43 6.26 95.9 77.1 21.1 3.4 17.7× / 15.3× 14.2× / 12.3× 3.9× / 3.4× 0.6× / 0.5×
added_special_sparse 0.06 0.88 5.18 6.01 61.7 46.8 15.2 3.6 11.9× / 10.3× 9.0× / 7.8× 2.9× / 2.5× 0.7× / 0.6×
agentic-traces 0.56 0.92 3.76 4.11 54.0 45.6 15.3 4.6 14.4× / 13.1× 12.2× / 11.1× 4.1× / 3.7× 1.2× / 1.1×
agentic_swe 0.52 0.87 2.90 3.25 56.4 53.3 15.7 3.5 19.5× / 17.4× 18.4× / 16.4× 5.4× / 4.8× 1.2× / 1.1×
code_mixed 0.08 0.87 3.33 4.12 54.7 52.4 16.0 4.0 16.4× / 13.3× 15.7× / 12.7× 4.8× / 3.9× 1.2× / 1.0×
math_latex 0.57 0.93 3.47 3.83 51.9 41.0 14.4 4.3 15.0× / 13.6× 11.8× / 10.7× 4.1× / 3.8× 1.2× / 1.1×
llama-2 — model-bounded BPE, no pre-tokenizer · ×36.95 vs v0.23.1 · ×1.02 vs base · decode ×1.50 llama-2 speedup llama-2 stage decomposition llama-2 thread scaling llama-2 decode speedup llama-2 decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 19+0 (peak 23) · Pipeline 29+0 (peak 29)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 4.8 344.8 ×71.89 ×1.02 2% (0.1) 76% (1.9) 0% (0.0) 23% (0.6) 0% (0.0) match
arb_Arab lang 10.5 338.8 ×32.41 ×1.03 2% (0.0) 82% (2.2) 0% (0.0) 17% (0.4) 0% (0.0) match
ben_Beng lang 11.2 433.0 ×38.62 ×1.01 3% (0.1) 76% (1.5) 0% (0.0) 21% (0.4) 0% (0.0) match
cmn_Hani lang 9.3 868.3 ×93.06 ×1.01 9% (0.1) 44% (0.4) 0% (0.0) 51% (0.4) 0% (0.0) match
ell_Grek lang 10.5 334.9 ×32.02 ×1.02 2% (0.1) 81% (2.2) 0% (0.0) 21% (0.6) 0% (0.0) match
eng_Latn lang 4.2 171.4 ×40.38 ×1.01 1% (0.1) 91% (5.2) 0% (0.0) 9% (0.5) 0% (0.0) match
heb_Hebr lang 10.3 330.8 ×32.04 ×1.02 2% (0.0) 82% (2.2) 0% (0.0) 19% (0.5) 0% (0.0) match
hin_Deva lang 12.2 367.0 ×30.11 ×1.01 2% (0.1) 82% (2.0) 0% (0.0) 18% (0.4) 0% (0.0) match
jpn_Jpan lang 12.7 981.2 ×76.97 ×1.02 10% (0.1) 42% (0.3) 0% (0.0) 51% (0.4) 0% (0.0) match
kat_Geor lang 14.4 466.1 ×32.33 ×1.01 4% (0.1) 75% (1.4) 0% (0.0) 22% (0.4) 0% (0.0) match
kor_Hang lang 7.4 323.6 ×43.58 ×1.02 3% (0.1) 81% (2.3) 0% (0.0) 18% (0.5) 0% (0.0) match
rus_Cyrl lang 8.2 379.9 ×46.44 ×1.04 2% (0.1) 84% (2.0) 0% (0.0) 15% (0.4) 0% (0.0) match
tam_Taml lang 12.8 496.7 ×38.68 ×1.02 3% (0.1) 75% (1.3) 0% (0.0) 24% (0.4) 0% (0.0) match
tha_Thai lang 15.6 659.1 ×42.19 ×1.03 4% (0.1) 65% (0.8) 0% (0.0) 32% (0.4) 0% (0.0) match
added_normalized_dense modalities 5.7 265.0 ×46.43 ×1.01 3% (0.1) 84% (2.6) 0% (0.0) 13% (0.4) 1% (0.0) match
added_normalized_sparse modalities 5.1 186.1 ×36.80 ×1.04 1% (0.0) 92% (4.5) 0% (0.0) 9% (0.4) 0% (0.0) match
added_special_dense modalities 3.9 40.6 ×10.41 ×1.00 22% (5.2) 62% (14.4) 4% (0.8) 12% (2.7) 0% (0.1) match
added_special_sparse modalities 5.5 50.6 ×9.17 ×1.06 13% (2.2) 73% (12.4) 2% (0.4) 13% (2.2) 0% (0.0) match
agentic-traces modalities 4.5 186.1 ×41.07 ×1.01 3% (0.1) 89% (4.5) 0% (0.0) 11% (0.5) 0% (0.0) match
agentic_swe modalities 4.1 139.9 ×34.20 ×1.01 1% (0.1) 92% (6.3) 0% (0.0) 8% (0.5) 0% (0.0) match
code_mixed modalities 4.3 163.9 ×38.10 ×1.00 1% (0.0) 91% (5.4) 0% (0.0) 9% (0.6) 0% (0.0) match
math_latex modalities 4.4 180.8 ×41.01 ×1.01 1% (0.1) 91% (4.8) 0% (0.0) 9% (0.5) 0% (0.0) match
llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×22.26 vs v0.23.1 · ×1.06 vs base · decode ×13.00 llama-3 speedup llama-3 stage decomposition llama-3 thread scaling llama-3 decode speedup llama-3 decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 73+0 (peak 95) · Pipeline 108+0 (peak 107)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 4.4 86.2 ×19.64 ×1.17 7% (0.7) 0% (0.0) 40% (3.7) 53% (5.0) 0% (0.0) match
arb_Arab lang 4.9 109.3 ×22.50 ×1.06 7% (0.6) 0% (0.0) 28% (2.3) 64% (5.3) 1% (0.1) match
ben_Beng lang 4.2 89.6 ×21.45 ×1.00 6% (0.6) 0% (0.0) 29% (3.1) 64% (6.9) 1% (0.1) match
cmn_Hani lang 5.4 136.8 ×25.46 ×1.24 4% (0.6) 0% (0.0) 15% (2.4) 83% (13.1) 0% (0.0) match
ell_Grek lang 5.4 115.7 ×21.31 ×1.02 5% (0.6) 0% (0.0) 19% (2.2) 76% (8.9) 0% (0.0) match
eng_Latn lang 4.3 86.6 ×19.93 ×1.12 18% (2.0) 0% (0.0) 28% (3.1) 53% (5.8) 1% (0.1) match
heb_Hebr lang 4.5 97.2 ×21.66 ×1.08 5% (0.6) 0% (0.0) 19% (2.3) 76% (9.3) 0% (0.0) match
hin_Deva lang 4.5 114.0 ×25.28 ×1.00 8% (0.6) 0% (0.0) 40% (3.3) 51% (4.1) 1% (0.1) match
jpn_Jpan lang 6.2 156.3 ×25.27 ×1.27 4% (0.6) 0% (0.0) 14% (2.2) 82% (12.6) 0% (0.0) match
kat_Geor lang 5.8 135.0 ×23.28 ×1.05 8% (0.6) 0% (0.0) 26% (2.1) 67% (5.3) 0% (0.0) match
kor_Hang lang 4.3 84.0 ×19.44 ×1.00 3% (0.6) 0% (0.0) 13% (2.5) 83% (15.7) 0% (0.0) match
rus_Cyrl lang 5.4 131.9 ×24.28 ×1.21 4% (0.6) 0% (0.0) 14% (2.2) 82% (12.3) 0% (0.0) match
tam_Taml lang 4.1 96.7 ×23.50 ×1.02 6% (0.6) 0% (0.0) 27% (2.7) 67% (6.8) 1% (0.1) match
tha_Thai lang 5.5 120.0 ×21.68 ×1.08 6% (0.6) 0% (0.0) 24% (2.6) 70% (7.6) 0% (0.0) match
added_normalized_dense modalities 6.1 176.9 ×29.21 ×1.01 15% (0.8) 0% (0.0) 21% (1.1) 65% (3.4) 0% (0.0) match
added_normalized_sparse modalities 5.3 134.1 ×25.15 ×0.98 20% (1.3) 0% (0.0) 28% (1.9) 53% (3.6) 1% (0.0) match
added_special_dense modalities 4.1 72.8 ×17.56 ×0.97 39% (5.0) 3% (0.5) 37% (4.8) 22% (2.8) 0% (0.0) match
added_special_sparse modalities 4.4 80.3 ×18.35 ×0.98 28% (3.2) 2% (0.2) 41% (4.8) 30% (3.4) 0% (0.0) match
agentic-traces modalities 3.6 79.4 ×22.07 ×1.06 15% (1.8) 0% (0.0) 31% (3.7) 54% (6.5) 0% (0.0) match
agentic_swe modalities 4.0 97.5 ×24.23 ×1.06 13% (1.3) 0% (0.0) 30% (2.9) 56% (5.4) 1% (0.1) match
code_mixed modalities 4.1 90.1 ×21.76 ×1.02 15% (1.6) 0% (0.0) 33% (3.3) 51% (5.2) 1% (0.1) match
math_latex modalities 3.9 78.8 ×20.29 ×1.06 16% (1.9) 0% (0.0) 29% (3.5) 54% (6.5) 1% (0.1) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.62 3.15 3.71 4.23 27.1 16.1 6.0 4.8 7.3× / 6.4× 4.3× / 3.8× 1.6× / 1.4× 1.3× / 1.1×
arb_Arab 1.00 2.67 2.33 3.99 30.2 20.0 6.9 5.2 13.0× / 7.5× 8.6× / 5.0× 3.0× / 1.7× 2.2× / 1.3×
ben_Beng 1.45 2.69 3.14 4.37 43.1 28.1 10.4 3.8 13.7× / 9.9× 9.0× / 6.4× 3.3× / 2.4× 1.2× / 0.9×
cmn_Hani 1.13 2.07 2.41 3.35 19.0 11.7 4.8 2.4 7.9× / 5.7× 4.9× / 3.5× 2.0× / 1.4× 1.0× / 0.7×
ell_Grek 0.58 2.67 2.24 4.33 27.5 17.2 6.2 4.8 12.3× / 6.4× 7.7× / 4.0× 2.8× / 1.4× 2.1× / 1.1×
eng_Latn 0.10 0.89 3.05 3.84 41.6 33.3 12.5 3.9 13.6× / 10.8× 10.9× / 8.7× 4.1× / 3.3× 1.3× / 1.0×
heb_Hebr 1.00 2.74 2.33 4.07 29.9 19.6 7.1 3.0 12.9× / 7.4× 8.4× / 4.8× 3.0× / 1.7× 1.3× / 0.7×
hin_Deva 1.36 2.84 3.26 4.74 46.3 31.0 11.2 4.1 14.2× / 9.8× 9.5× / 6.5× 3.4× / 2.4× 1.3× / 0.9×
jpn_Jpan 1.56 3.20 2.19 3.82 17.6 10.0 4.1 3.7 8.1× / 4.6× 4.6× / 2.6× 1.9× / 1.1× 1.7× / 1.0×
kat_Geor 1.38 2.27 2.09 2.98 16.6 11.3 4.2 2.2 8.0× / 5.6× 5.4× / 3.8× 2.0× / 1.4× 1.0× / 0.7×
kor_Hang 1.08 2.56 2.52 4.00 29.8 21.8 7.7 3.7 11.8× / 7.5× 8.7× / 5.5× 3.1× / 1.9× 1.5× / 0.9×
rus_Cyrl 1.02 2.63 2.17 3.78 26.0 16.8 6.1 2.5 12.0× / 6.9× 7.8× / 4.5× 2.8× / 1.6× 1.1× / 0.7×
tam_Taml 0.92 2.70 2.74 4.52 42.7 26.8 9.9 3.5 15.6× / 9.4× 9.8× / 5.9× 3.6× / 2.2× 1.3× / 0.8×
tha_Thai 1.36 2.35 2.60 3.59 27.2 16.5 6.9 3.1 10.5× / 7.6× 6.3× / 4.6× 2.6× / 1.9× 1.2× / 0.9×
added_normalized_dense 0.06 0.88 1.07 1.90 22.6 17.1 6.6 2.0 21.0× / 11.9× 15.9× / 9.0× 6.2× / 3.5× 1.8× / 1.0×
added_normalized_sparse 0.06 0.88 1.89 2.71 29.6 22.9 9.0 2.7 15.7× / 10.9× 12.2× / 8.5× 4.8× / 3.3× 1.4× / 1.0×
added_special_dense 0.06 0.88 4.78 5.61 90.7 75.2 21.1 3.4 19.0× / 16.2× 15.7× / 13.4× 4.4× / 3.8× 0.7× / 0.6×
added_special_sparse 0.06 0.88 4.82 5.65 57.8 47.1 15.1 3.6 12.0× / 10.2× 9.8× / 8.3× 3.1× / 2.7× 0.7× / 0.6×
agentic-traces 0.57 0.92 3.74 4.09 50.3 44.8 15.2 4.7 13.5× / 12.3× 12.0× / 11.0× 4.1× / 3.7× 1.2× / 1.1×
agentic_swe 0.51 0.86 2.88 3.23 52.8 52.6 15.7 3.4 18.3× / 16.3× 18.3× / 16.3× 5.4× / 4.8× 1.2× / 1.1×
code_mixed 0.09 0.87 3.30 4.08 50.7 49.8 15.4 4.1 15.4× / 12.4× 15.1× / 12.2× 4.7× / 3.8× 1.2× / 1.0×
math_latex 0.57 0.92 3.47 3.82 48.7 40.5 14.3 4.3 14.0× / 12.7× 11.7× / 10.6× 4.1× / 3.7× 1.2× / 1.1×
mistral-small-4 — tekken byte-level BPE, 1k added specials (mistral-small-4) · ×18.69 vs v0.23.1 · ×1.04 vs base · decode ×8.80 mistral-small-4 speedup mistral-small-4 stage decomposition mistral-small-4 thread scaling mistral-small-4 decode speedup mistral-small-4 decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 151+0 (peak 193) · Pipeline 145+0 (peak 194)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 3.8 74.1 ×19.65 ×1.14 11% (1.2) 0% (0.0) 43% (4.8) 44% (4.9) 2% (0.2) match
arb_Arab lang 5.0 99.9 ×20.13 ×1.13 12% (1.2) 0% (0.0) 35% (3.3) 53% (5.1) 0% (0.0) match
ben_Beng lang 6.8 110.8 ×16.22 ×1.07 12% (1.2) 0% (0.0) 39% (3.7) 48% (4.6) 0% (0.0) match
cmn_Hani lang 5.1 95.1 ×18.67 ×1.03 5% (1.2) 0% (0.0) 16% (3.5) 86% (18.7) 0% (0.0) match
ell_Grek lang 5.3 97.0 ×18.34 ×1.03 8% (1.2) 0% (0.0) 23% (3.2) 70% (9.7) 0% (0.0) match
eng_Latn lang 3.7 66.5 ×17.80 ×1.01 18% (2.5) 0% (0.0) 33% (4.6) 46% (6.5) 3% (0.4) match
heb_Hebr lang 4.7 89.2 ×18.98 ×1.05 7% (1.2) 0% (0.0) 21% (3.4) 73% (12.0) 0% (0.0) match
hin_Deva lang 6.9 112.8 ×16.29 ×1.11 11% (1.2) 0% (0.0) 37% (3.8) 51% (5.2) 0% (0.0) match
jpn_Jpan lang 5.8 118.5 ×20.27 ×1.11 8% (1.2) 0% (0.0) 23% (3.2) 84% (12.0) 0% (0.0) match
kat_Geor lang 6.8 123.7 ×18.10 ×1.10 9% (1.2) 0% (0.0) 23% (2.9) 68% (8.7) 0% (0.0) match
kor_Hang lang 4.2 75.4 ×17.84 ×1.03 6% (1.2) 0% (0.0) 18% (3.7) 77% (16.4) 0% (0.0) match
rus_Cyrl lang 4.9 95.5 ×19.48 ×1.01 7% (1.2) 0% (0.0) 19% (3.1) 74% (12.4) 0% (0.0) match
tam_Taml lang 6.9 114.0 ×16.53 ×1.01 9% (1.2) 0% (0.0) 25% (3.2) 66% (8.6) 0% (0.0) match
tha_Thai lang 8.1 134.7 ×16.56 ×1.02 9% (1.2) 0% (0.0) 26% (3.2) 66% (8.2) 0% (0.0) match
added_normalized_dense modalities 5.5 138.4 ×24.98 ×0.99 19% (1.3) 0% (0.0) 32% (2.2) 48% (3.3) 1% (0.0) match
added_normalized_sparse modalities 5.2 108.4 ×20.88 ×0.99 21% (1.9) 0% (0.0) 36% (3.2) 42% (3.8) 1% (0.0) match
added_special_dense modalities 4.3 69.9 ×16.42 ×1.00 37% (5.1) 3% (0.4) 38% (5.2) 22% (3.0) 1% (0.1) match
added_special_sparse modalities 4.4 70.3 ×15.89 ×0.98 27% (3.6) 1% (0.2) 44% (5.9) 29% (3.8) 0% (0.0) match
agentic-traces modalities 3.4 64.1 ×19.12 ×1.05 16% (2.4) 0% (0.0) 35% (5.0) 48% (7.0) 1% (0.1) match
agentic_swe modalities 3.3 76.1 ×22.72 ×1.00 16% (1.9) 0% (0.0) 32% (3.8) 51% (6.1) 1% (0.1) match
code_mixed modalities 3.6 73.2 ×20.27 ×0.99 17% (2.2) 0% (0.0) 36% (4.4) 46% (5.6) 1% (0.1) match
math_latex modalities 3.5 65.3 ×18.73 ×1.01 18% (2.5) 0% (0.0) 34% (4.9) 53% (7.5) 0% (0.0) match

Pre-tokenize: classify + fsm vs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm). ×vs = engine ÷ our pipeline (SIMD / scalar classify); onig & pcre2 (JIT) are C, fancy is pure-Rust fancy-regex, logos is a compile-time DFA lexer (approximate grammar; n/a for deepseek).

Fixture classify SIMD classify scalar pipe (SIMD cls + fsm) pipe (scalar cls + fsm) onig fancy pcre2 logos ×vs onig ×vs fancy ×vs pcre2 ×vs logos
amh_Ethi 2.62 3.15 4.76 5.29 26.8 14.2 6.8 — 5.6× / 5.1× 3.0× / 2.7× 1.4× / 1.3× —
arb_Arab 1.00 2.67 3.32 4.99 31.2 16.6 7.4 — 9.4× / 6.3× 5.0× / 3.3× 2.2× / 1.5× —
ben_Beng 1.46 2.68 3.69 4.91 22.0 11.1 5.4 — 6.0× / 4.5× 3.0× / 2.3× 1.5× / 1.1× —
cmn_Hani 1.12 2.11 3.47 4.46 20.7 11.7 5.6 — 6.0× / 4.6× 3.4× / 2.6× 1.6× / 1.3× —
ell_Grek 0.58 2.70 3.24 5.36 27.6 16.1 6.9 — 8.5× / 5.2× 4.9× / 3.0× 2.1× / 1.3× —
eng_Latn 0.12 0.90 4.59 5.38 39.6 31.1 13.8 — 8.6× / 7.4× 6.8× / 5.8× 3.0× / 2.6× —
heb_Hebr 1.01 2.77 3.43 5.20 30.3 17.6 7.9 — 8.8× / 5.8× 5.1× / 3.4× 2.3× / 1.5× —
hin_Deva 1.36 2.86 3.83 5.33 23.4 13.1 6.2 — 6.1× / 4.4× 3.4× / 2.4× 1.6× / 1.2× —
jpn_Jpan 1.56 3.21 3.22 4.87 19.5 9.7 4.8 — 6.1× / 4.0× 3.0× / 2.0× 1.5× / 1.0× —
kat_Geor 1.39 2.26 2.94 3.82 17.3 10.5 4.6 — 5.9× / 4.5× 3.6× / 2.7× 1.5× / 1.2× —
kor_Hang 1.11 2.55 3.73 5.18 30.1 19.6 8.8 — 8.1× / 5.8× 5.2× / 3.8× 2.3× / 1.7× —
rus_Cyrl 1.02 2.65 3.14 4.77 26.7 15.4 6.6 — 8.5× / 5.6× 4.9× / 3.2× 2.1× / 1.4× —
tam_Taml 0.92 2.68 3.24 5.00 17.7 8.9 4.3 — 5.5× / 3.5× 2.8× / 1.8× 1.3× / 0.9× —
tha_Thai 1.36 2.32 3.20 4.16 12.7 5.8 2.8 — 4.0× / 3.1× 1.8× / 1.4× 0.9× / 0.7× —
added_normalized_dense 0.06 0.88 2.20 3.02 30.4 17.3 11.5 — 13.8× / 10.1× 7.9× / 5.7× 5.2× / 3.8× —
added_normalized_sparse 0.06 0.88 3.21 4.03 31.4 20.5 11.6 — 9.8× / 7.8× 6.4× / 5.1× 3.6× / 2.9× —
added_special_dense 0.06 0.88 5.18 6.01 77.9 67.9 23.8 — 15.0× / 13.0× 13.1× / 11.3× 4.6× / 4.0× —
added_special_sparse 0.06 0.88 5.93 6.76 50.2 42.3 16.3 — 8.5× / 7.4× 7.1× / 6.3× 2.7× / 2.4× —
agentic-traces 0.57 0.92 5.04 5.39 49.7 43.9 17.3 — 9.9× / 9.2× 8.7× / 8.1× 3.4× / 3.2× —
agentic_swe 0.51 0.86 3.82 4.17 54.2 54.3 18.5 — 14.2× / 13.0× 14.2× / 13.0× 4.9× / 4.4× —
code_mixed 0.09 0.87 4.42 5.20 49.3 49.5 17.5 — 11.2× / 9.5× 11.2× / 9.5× 4.0× / 3.4× —
math_latex 0.57 0.92 4.86 5.22 46.3 38.1 16.2 — 9.5× / 8.9× 7.8× / 7.3× 3.3× / 3.1× —
t5-base — Unigram + Metaspace · ×5.59 vs v0.23.1 · ×1.00 vs base · decode ×0.95 t5-base speedup t5-base stage decomposition t5-base thread scaling t5-base decode speedup t5-base decode thread scaling

Memory (RSS MB, load+encode): v0.23.1 34+2 (peak 34) · Pipeline 61+0 (peak 65)

Fixture Group v0.23.1 MB/s Pipeline MB/s Speedup Δ base added-token normalize pre-tokenize model post Ids
amh_Ethi lang 5.4 35.7 ×6.59 ×1.00 2% (0.7) 80% (21.8) 9% (2.4) 10% (2.6) 0% (0.0) match
arb_Arab lang 4.2 29.4 ×6.97 ×1.00 2% (0.6) 76% (27.0) 8% (2.8) 15% (5.4) 0% (0.0) match
ben_Beng lang 7.4 34.5 ×4.65 ×1.02 2% (0.6) 78% (24.4) 5% (1.7) 15% (4.8) 0% (0.0) match
cmn_Hani lang 12.5 50.6 ×4.06 ×0.98 2% (0.6) 67% (17.0) 2% (0.6) 28% (7.2) 0% (0.0) match
ell_Grek lang 4.6 29.4 ×6.42 ×1.00 1% (0.6) 64% (27.4) 6% (2.5) 28% (12.1) 0% (0.0) match
eng_Latn lang 2.6 18.5 ×7.27 ×1.00 2% (2.0) 45% (41.0) 5% (4.7) 45% (40.7) 2% (2.0) match
heb_Hebr lang 4.1 31.8 ×7.75 ×0.99 2% (0.6) 58% (24.0) 7% (2.8) 34% (13.8) 0% (0.0) match
hin_Deva lang 6.0 33.3 ×5.51 ×1.00 2% (0.6) 76% (24.4) 7% (2.3) 15% (4.8) 0% (0.0) match
jpn_Jpan lang 13.6 44.3 ×3.26 ×0.99 2% (0.6) 74% (20.1) 1% (0.2) 23% (6.1) 0% (0.0) match
kat_Geor lang 8.1 42.6 ×5.29 ×0.99 2% (0.6) 63% (19.0) 5% (1.5) 30% (9.0) 0% (0.0) match
kor_Hang lang 4.2 30.4 ×7.19 ×1.00 1% (0.6) 53% (24.6) 6% (2.9) 40% (18.5) 0% (0.0) match
rus_Cyrl lang 4.2 29.0 ×6.94 ×1.01 1% (0.6) 50% (27.1) 4% (2.1) 49% (26.6) 0% (0.0) match
tam_Taml lang 9.0 43.8 ×4.88 ×1.01 2% (0.6) 66% (18.8) 5% (1.3) 28% (7.9) 0% (0.0) match
tha_Thai lang 12.8 46.5 ×3.65 ×1.01 2% (0.6) 67% (18.9) 3% (0.8) 27% (7.7) 1% (0.2) match
added_normalized_dense modalities 4.6 22.6 ×4.94 ×0.98 2% (0.8) 83% (38.1) 5% (2.2) 11% (4.9) 0% (0.0) match
added_normalized_sparse modalities 3.9 20.6 ×5.25 ×0.98 3% (1.4) 74% (39.4) 7% (3.9) 16% (8.4) 0% (0.0) match
added_special_dense modalities 6.2 36.6 ×5.89 ×1.00 19% (5.1) 64% (16.8) 12% (3.1) 4% (1.1) 1% (0.2) match
added_special_sparse modalities 4.1 20.2 ×4.90 ×1.03 6% (3.2) 64% (34.0) 20% (10.6) 4% (1.9) 6% (3.4) match
agentic-traces modalities 2.8 19.2 ×6.82 ×0.98 2% (1.8) 49% (40.3) 5% (3.8) 46% (38.0) 0% (0.0) match
agentic_swe modalities 4.3 22.5 ×5.27 ×1.00 2% (1.3) 61% (37.5) 4% (2.3) 31% (19.2) 1% (0.7) match
code_mixed modalities 3.7 20.8 ×5.56 ×1.00 2% (1.6) 57% (39.3) 4% (2.7) 36% (25.0) 0% (0.0) match
math_latex modalities 2.7 18.6 ×6.92 ×0.99 2% (1.9) 49% (41.0) 6% (4.8) 42% (35.3) 0% (0.1) match

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SBrandeis
SBrandeis marked this pull request as ready for review August 6, 2026 13:34
@SBrandeis SBrandeis changed the title wordpiece reverse vocab lookup feat(decode): wire wordpiece with a naive reverse vocab lookup Aug 6, 2026
@SBrandeis
SBrandeis merged commit c2b7135 into feat/train_encode_split Aug 6, 2026
38 of 42 checks passed
@SBrandeis
SBrandeis deleted the feat/decoded-wordpiece branch August 6, 2026 13:57
ArthurZucker added a commit that referenced this pull request Sep 2, 2026
Squashed from 3 commits:
  - feat(tk-encode): implement PipelineTokenizer::decode (8-10x over the legacy path) (#2305)
  - feat(decode): wire wordpiece with a naive reverse vocab lookup (#2309)
  - iterate on CI bench (#2298)

Area: core-decode
Co-authored-by: Simon Brandeis <33657802+SBrandeis@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants