Skip to content

chore(tk-encode): remove Unigram sampling - #2707

Open
SBrandeis wants to merge 46 commits into
mainfrom
chore/rm-unigram-sampling
Open

SBrandeis wants to merge 46 commits into
mainfrom
chore/rm-unigram-sampling

Conversation

@SBrandeis

Copy link
Copy Markdown
Collaborator

Unigram::alpha and nbest_size drew a tokenization at random from the lattice (subword regularization). No tokenizer.json sets them, and since the fields went pub(crate) nothing outside the tests could. Lattice::sample and sample_nbest go with them, and so does the rand dependency, which only they used. Unigram::tokenize now always goes through the word cache.

Deviation from 0.23: its Rust fields and Python Unigram(alpha=, nbest_size=) have no equivalent. Lattice::nbest stays, since the Unigram trainer prunes with it.

@SBrandeis
SBrandeis added this pull request to stack #2702 November 17, 2026 15:43
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SBrandeis
SBrandeis removed this pull request from stack #2702 November 17, 2026 15:48
Nothing outside bitcannon uses them. Atom itself stays pub: it is the value
set classify() writes into its public tags output.
Json, its accessors, MAX_DEPTH and json::Error's fields are only used by the
reader in tk-serialize. json::Error itself stays pub for now: the umbrella
re-exports the json module, so the compiler cannot tell who names it.
Buckets, Bucket, AddedTokenFlags, BucketVocabStore and its key hashing, plus
AddedVocabulary's lookup accessors, are only used inside tk-encode.
get_vocab, get_vocab_size, id_to_token and token_to_id on Unigram, WordLevel,
PipelineWordPiece and PipelineModel are only called by PipelineTokenizer, which
keeps its own pub lookups.
…the crate

Lattice's insert, piece, len, bos_node, eos_node, sample and sample_nbest,
Node::new, Unigram's alpha and nbest_size fields, and UnigramError are only
used inside tk-encode.
…rate

WordLevelBuilder::new and WordPieceBuilder::new are reached through
WordLevel::builder() and WordPiece::builder(); nothing outside tk-encode reads
WordPiece's max_input_chars_per_word field.
PipelineTokenizer's resolve_padding and resolve_truncation, PreTokenizerScratch's
split_on_bytes, split_on_bits, take_pair and put_pair, and Template's
has_type_ids and post_process are only called inside tk-encode.
The Pattern trait, Invert, Split's Search enum and search field, and
whitespace::is_word_char are only used by tk-encode's pre-tokenizers.
Only its Default impl calls it; tk-serialize builds the unit struct directly.
…egex::find_iter to the crate

Only tk-encode uses them.
Unigram, its only caller, calls content() directly. Now that the store is
crate-private the alias was dead in builds without the unigram feature.
ENV_VARIABLE, get_parallelism and num_threads are only read by tk-encode's
pool setup; set_num_threads stays pub as the library knob.
Since MB went pub(crate), a target without a classify kernel (scalar wasm32,
riscv64) flags it as dead, which broke the bitcannon wasm clippy job.
split, its SplitPolicy, split_delimiter and split_matches are only called by
tk-encode's pre-tokenizers. split and SplitPolicy need no re-export at all.
Only Split and the regex recognizer use it. Its last pub use, Split::grammar,
was removed as test-only.
Adding a variant to these is no longer a breaking change:

- errors: bpe/wordlevel/wordpiece `Error`, `ConvertError`, `UnigramTrainerError`
- component dispatch: `PipelineModel`, `PipelineNormalizer`, `PipelinePreTokenizer`,
  `DecoderRuntime`, `ModelWrapper`, `TrainerWrapper`. Their feature-gated variants
  already made downstream exhaustive matches break under feature unification.
- trainer options: `ProgressFormat`, `ParityVariant`

Closed sets stay exhaustive: the directions, `Override`, `Input(s)`, and the value
sets `tokenizer.json` fixes (truncation/padding strategy, split behaviour, prepend
scheme, string-or-regex patterns).

tk-serialize's writers now need a wildcard arm. It returns an `Err` naming the
component kind, so a component added without a writer fails `to_json` loudly
instead of failing to compile; the decoder writer becomes fallible for it.
…nd decoder

`decoders::metaspace::PrependScheme` and `normalizers::metaspace::PrependBehavior`
were the same three values. Keep one, named after the `prepend_scheme` key, in
`normalizers::metaspace`; drop its unused `Display` impl. tk-serialize's two
readers and two writers of it fold into one each; error messages are unchanged.
Applies the rule now in tokenizers-rust-style/public-api.md: a public type stays
exhaustive only when one more variant or field would make it a different concept.
"Nothing planned" and "mirrors tokenizer.json" no longer close a type.

Enums: TruncationStrategy, PaddingStrategy, SplitDelimiterBehavior, PrependScheme,
ReplacePattern, SplitPattern, bitcannon's Atom, PipelineModelScratch.
Structs: EncodeOptions, PaddingParams, TruncationParams, BpeConfig,
FromPretrainedParameters, AddedToken, Prepend, FixedLength, Punctuation.

Fallout:
- tk-serialize's writers match these with a wildcard arm returning an Err, so
  write_padding/write_truncation and the prepend_scheme/pattern helpers become
  fallible, and padding_to_json/truncation_to_json now return Result<String>.
- Struct literals outside tk-encode (tk-serialize readers, tk-train, the three
  bindings, the oracle and wasm tests) start from default() and assign fields.
- tk-train's AddedToken serde shim drops `serde(remote)`, whose derive builds the
  foreign struct with a literal, for a local struct with From both ways.
- Python: Padding.length, Padding.__repr__/__reduce__, Tokenizer.to_str/save/__repr__
  raise on a padding strategy the binding cannot express; same Python signatures.
0.23 also encodes pre-tokenized input, so Single/Pair is not the whole concept.
A flat batch (one buffer plus offsets, as in #2361) is a plausible third variant.
`vocab_r` is the inverse of `vocab`, but both were pub, so any caller could
change one without the other. The WordLevel trainer kept them in step by hand,
assigning both maps on the model.

The fields are private now, with `vocab()` and `unk_token()` getters (the shape
Unigram's `vocab()` already has). The trainer rebuilds the model through
`WordLevel::builder()`, which derives `vocab_r`, and passes the model's own
`unk_token` along: the builder would otherwise reset it to `<unk>`, which
`training_keeps_the_models_unk_token` pins.
…r::new

The single template must not reference sequence B or carry an infix, and the
pair template must reference B. Only tk-serialize's reader checked this. With
pub fields, `PipelineTokenizer::from_parts` took any processor, and in a release
build a broken template gave an `Encoding` whose `type_ids` and ids differ in
length (tk-encode had a debug_assert for the infix only).

The fields are private now, with `single()` and `pair()` getters. `new` checks
both templates and reports every broken rule at once. The reader calls it and
drops its own check, so a config that broke the rules is still refused, with
the new wording.
It only bounded Model::Scratch and required nothing, so it constrained
nothing. Removing it frees the name for PipelineModelScratch.
Frees the noun for the enum users hold (PipelineNormalizer). The trait
names the capability, like std's Read or Hash.
Frees the noun for the enum users hold (PipelinePreTokenizer).
Frees the noun for the enum users hold (PipelineModel). Its methods are
tokenize_pipeline and tokenize_spans, so the verb is the one they share.
The generic parameter that carried the bare name becomes M.
The Python and wasm bindings define their own Tokenizer, so they reach the
core one through its module (pipeline::Tokenizer). The Node class keeps
its JS name, PipelineTokenizer.
…onfig

Gives WordPiece BPE's shape: a plain WordPieceConfig with pub fields and
a Default, and no builder. The builder only copied its fields into the
struct, which a struct literal does directly. The reader looks up the
vocab through closures, as the BPE arm does, so the config needs no
get_vocab_size/token_to_id of its own. This frees the WordPiece name for
the runtime model.
Built with WordPiece::from_config, as BPE is with BPE::from_config,
in place of TryFrom<WordPieceConfig>.
The Python and Node bindings imported core types under Pipeline* aliases
(PipelineEncoding, PipelineEncodeOptions, Pipeline) to avoid clashing
with their own classes. They now spell them pipeline::Encoding and so on,
like the Tokenizer and DecodeStream sites.
The suffix told it apart from the legacy Model::tokenize, which is gone.
Tokenize::tokenize now pairs with Tokenize::tokenize_spans.
…ntime to Decoder

Matches the other pipeline stages: the trait names the capability
(Normalize, PreTokenize, Tokenize) and the noun goes to the enum the
Tokenizer holds (Normalizer, PreTokenizer, Model). Both renames land
together because the enum takes the name the trait gives up.
The `train` feature re-exported `tk-train`'s trainers from `tokenizers`.
Once `tokenizers` is 1.0, that would freeze a trainer API we plan to
redesign after v1 (no tokenizer-level `train`, mutate-in-place `train`,
two ways to configure most trainers, indicatif and ahash types in public
signatures). `tk-train` stays a 0.x crate that users depend on directly;
adding `train` back to `tokenizers` later is a non-breaking change.

Removes the `train`, `esaxx_fast`, `parity-aware-bpe` and `progressbar`
features from `tokenizers` (all four are still on `tk-train`).
Unigram::alpha and nbest_size drew a tokenization at random from the
lattice (subword regularization). No tokenizer.json sets them, and since
the fields went pub(crate) nothing outside the tests could. Lattice::sample
and sample_nbest go with them, and so does the rand dependency, which only
they used. Unigram::tokenize now always goes through the word cache.

Deviation from 0.23: its Rust fields and Python Unigram(alpha=, nbest_size=)
have no equivalent. Lattice::nbest stays, since the Unigram trainer prunes
with it.
@SBrandeis
SBrandeis force-pushed the chore/rm-unigram-sampling branch from d2d5647 to 84f42d1 Compare November 17, 2026 15:57
@SBrandeis
SBrandeis force-pushed the chore/trainer-api branch 3 times, most recently from 2264f25 to f2061c7 Compare November 17, 2026 16:32
Base automatically changed from chore/trainer-api to main November 17, 2026 16:38

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants