Repository navigation
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Nothing outside bitcannon uses them. Atom itself stays pub: it is the value set classify() writes into its public tags output.
Json, its accessors, MAX_DEPTH and json::Error's fields are only used by the reader in tk-serialize. json::Error itself stays pub for now: the umbrella re-exports the json module, so the compiler cannot tell who names it.
Buckets, Bucket, AddedTokenFlags, BucketVocabStore and its key hashing, plus AddedVocabulary's lookup accessors, are only used inside tk-encode.
get_vocab, get_vocab_size, id_to_token and token_to_id on Unigram, WordLevel, PipelineWordPiece and PipelineModel are only called by PipelineTokenizer, which keeps its own pub lookups.
…the crate Lattice's insert, piece, len, bos_node, eos_node, sample and sample_nbest, Node::new, Unigram's alpha and nbest_size fields, and UnigramError are only used inside tk-encode.
Only the BPE model uses them.
…rate WordLevelBuilder::new and WordPieceBuilder::new are reached through WordLevel::builder() and WordPiece::builder(); nothing outside tk-encode reads WordPiece's max_input_chars_per_word field.
PipelineTokenizer's resolve_padding and resolve_truncation, PreTokenizerScratch's split_on_bytes, split_on_bits, take_pair and put_pair, and Template's has_type_ids and post_process are only called inside tk-encode.
The Pattern trait, Invert, Split's Search enum and search field, and whitespace::is_word_char are only used by tk-encode's pre-tokenizers.
Only its Default impl calls it; tk-serialize builds the unit struct directly.
…egex::find_iter to the crate Only tk-encode uses them.
Unigram, its only caller, calls content() directly. Now that the store is crate-private the alias was dead in builds without the unigram feature.
ENV_VARIABLE, get_parallelism and num_threads are only read by tk-encode's pool setup; set_num_threads stays pub as the library knob.
Since MB went pub(crate), a target without a classify kernel (scalar wasm32, riscv64) flags it as dead, which broke the bitcannon wasm clippy job.
split, its SplitPolicy, split_delimiter and split_matches are only called by tk-encode's pre-tokenizers. split and SplitPolicy need no re-export at all.
Only Split and the regex recognizer use it. Its last pub use, Split::grammar, was removed as test-only.
Adding a variant to these is no longer a breaking change: - errors: bpe/wordlevel/wordpiece `Error`, `ConvertError`, `UnigramTrainerError` - component dispatch: `PipelineModel`, `PipelineNormalizer`, `PipelinePreTokenizer`, `DecoderRuntime`, `ModelWrapper`, `TrainerWrapper`. Their feature-gated variants already made downstream exhaustive matches break under feature unification. - trainer options: `ProgressFormat`, `ParityVariant` Closed sets stay exhaustive: the directions, `Override`, `Input(s)`, and the value sets `tokenizer.json` fixes (truncation/padding strategy, split behaviour, prepend scheme, string-or-regex patterns). tk-serialize's writers now need a wildcard arm. It returns an `Err` naming the component kind, so a component added without a writer fails `to_json` loudly instead of failing to compile; the decoder writer becomes fallible for it.
…nd decoder `decoders::metaspace::PrependScheme` and `normalizers::metaspace::PrependBehavior` were the same three values. Keep one, named after the `prepend_scheme` key, in `normalizers::metaspace`; drop its unused `Display` impl. tk-serialize's two readers and two writers of it fold into one each; error messages are unchanged.
Applies the rule now in tokenizers-rust-style/public-api.md: a public type stays exhaustive only when one more variant or field would make it a different concept. "Nothing planned" and "mirrors tokenizer.json" no longer close a type. Enums: TruncationStrategy, PaddingStrategy, SplitDelimiterBehavior, PrependScheme, ReplacePattern, SplitPattern, bitcannon's Atom, PipelineModelScratch. Structs: EncodeOptions, PaddingParams, TruncationParams, BpeConfig, FromPretrainedParameters, AddedToken, Prepend, FixedLength, Punctuation. Fallout: - tk-serialize's writers match these with a wildcard arm returning an Err, so write_padding/write_truncation and the prepend_scheme/pattern helpers become fallible, and padding_to_json/truncation_to_json now return Result<String>. - Struct literals outside tk-encode (tk-serialize readers, tk-train, the three bindings, the oracle and wasm tests) start from default() and assign fields. - tk-train's AddedToken serde shim drops `serde(remote)`, whose derive builds the foreign struct with a literal, for a local struct with From both ways. - Python: Padding.length, Padding.__repr__/__reduce__, Tokenizer.to_str/save/__repr__ raise on a padding strategy the binding cannot express; same Python signatures.
0.23 also encodes pre-tokenized input, so Single/Pair is not the whole concept.
A flat batch (one buffer plus offsets, as in #2361) is a plausible third variant.
`vocab_r` is the inverse of `vocab`, but both were pub, so any caller could change one without the other. The WordLevel trainer kept them in step by hand, assigning both maps on the model. The fields are private now, with `vocab()` and `unk_token()` getters (the shape Unigram's `vocab()` already has). The trainer rebuilds the model through `WordLevel::builder()`, which derives `vocab_r`, and passes the model's own `unk_token` along: the builder would otherwise reset it to `<unk>`, which `training_keeps_the_models_unk_token` pins.
…r::new The single template must not reference sequence B or carry an infix, and the pair template must reference B. Only tk-serialize's reader checked this. With pub fields, `PipelineTokenizer::from_parts` took any processor, and in a release build a broken template gave an `Encoding` whose `type_ids` and ids differ in length (tk-encode had a debug_assert for the infix only). The fields are private now, with `single()` and `pair()` getters. `new` checks both templates and reports every broken rule at once. The reader calls it and drops its own check, so a config that broke the rules is still refused, with the new wording.
It only bounded Model::Scratch and required nothing, so it constrained nothing. Removing it frees the name for PipelineModelScratch.
Frees the noun for the enum users hold (PipelineNormalizer). The trait names the capability, like std's Read or Hash.
Frees the noun for the enum users hold (PipelinePreTokenizer).
Frees the noun for the enum users hold (PipelineModel). Its methods are tokenize_pipeline and tokenize_spans, so the verb is the one they share.
The generic parameter that carried the bare name becomes M.
The Python and wasm bindings define their own Tokenizer, so they reach the core one through its module (pipeline::Tokenizer). The Node class keeps its JS name, PipelineTokenizer.
…onfig Gives WordPiece BPE's shape: a plain WordPieceConfig with pub fields and a Default, and no builder. The builder only copied its fields into the struct, which a struct literal does directly. The reader looks up the vocab through closures, as the BPE arm does, so the config needs no get_vocab_size/token_to_id of its own. This frees the WordPiece name for the runtime model.
Built with WordPiece::from_config, as BPE is with BPE::from_config, in place of TryFrom<WordPieceConfig>.
The Python and Node bindings imported core types under Pipeline* aliases (PipelineEncoding, PipelineEncodeOptions, Pipeline) to avoid clashing with their own classes. They now spell them pipeline::Encoding and so on, like the Tokenizer and DecodeStream sites.
The suffix told it apart from the legacy Model::tokenize, which is gone. Tokenize::tokenize now pairs with Tokenize::tokenize_spans.
…ntime to Decoder Matches the other pipeline stages: the trait names the capability (Normalize, PreTokenize, Tokenize) and the noun goes to the enum the Tokenizer holds (Normalizer, PreTokenizer, Model). Both renames land together because the enum takes the name the trait gives up.
The `train` feature re-exported `tk-train`'s trainers from `tokenizers`. Once `tokenizers` is 1.0, that would freeze a trainer API we plan to redesign after v1 (no tokenizer-level `train`, mutate-in-place `train`, two ways to configure most trainers, indicatif and ahash types in public signatures). `tk-train` stays a 0.x crate that users depend on directly; adding `train` back to `tokenizers` later is a non-breaking change. Removes the `train`, `esaxx_fast`, `parity-aware-bpe` and `progressbar` features from `tokenizers` (all four are still on `tk-train`).
Unigram::alpha and nbest_size drew a tokenization at random from the lattice (subword regularization). No tokenizer.json sets them, and since the fields went pub(crate) nothing outside the tests could. Lattice::sample and sample_nbest go with them, and so does the rand dependency, which only they used. Unigram::tokenize now always goes through the word cache. Deviation from 0.23: its Rust fields and Python Unigram(alpha=, nbest_size=) have no equivalent. Lattice::nbest stays, since the Unigram trainer prunes with it.
SBrandeis
force-pushed
the
chore/rm-unigram-sampling
branch
from
November 17, 2026 15:57
d2d5647 to
84f42d1
Compare
SBrandeis
force-pushed
the
chore/trainer-api
branch
3 times, most recently
from
November 17, 2026 16:32
2264f25 to
f2061c7
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Unigram::alpha and nbest_size drew a tokenization at random from the lattice (subword regularization). No tokenizer.json sets them, and since the fields went pub(crate) nothing outside the tests could. Lattice::sample and sample_nbest go with them, and so does the rand dependency, which only they used. Unigram::tokenize now always goes through the word cache.
Deviation from 0.23: its Rust fields and Python Unigram(alpha=, nbest_size=) have no equivalent. Lattice::nbest stays, since the Unigram trainer prunes with it.