Paper: What Language is This? Ask Your Tokenizer (ICML 2026)
Fast multilingual language identification using unigram language models. A shared token vocabulary is trained across languages, each language gets its own token probability distribution over it, and a text is labeled with the language that scores it highest (Rust-accelerated, Rayon-parallel batch inference). The released model covers 1,940 language-script combinations and ships with calibrated inference (default), per-language decision thresholds that can be extended one language at a time, and language subsetting for latency-sensitive deployments.
Prerequisites:
- Python 3.9 or newer. The suite is run on 3.9, 3.11, 3.12, 3.13 and 3.14; the Rust extension is an abi3 build, so one compiled extension serves all of them
- A Rust toolchain for the mandatory tokenizers build. With rustup there is nothing to choose: the bundled fork pins its toolchain version and rustup fetches it. Without rustup, 1.85 or newer
- cmake and a C++ compiler, only if you build the optional SentencePiece CLI (used by the
sptraining method)
# Clone with submodules (required for the custom tokenizers)
git clone --recurse-submodules https://github.com/Ahmetcanyvz/UNILID.git && cd UNILID
# Safe to re-run, and the fix if the clone did not pull the submodule
git submodule update --init tokenizers
python3 -m venv .venv
source .venv/bin/activate
# Install unilid. This does not install or build the tokenizers extension,
# so prediction needs the build below as well.
pip install -e .
# Check the checkout before building
python doctor.py
# Build the custom tokenizers extension (REQUIRED for inference; standard
# HuggingFace tokenizers will NOT work)
pip uninstall tokenizers -y
pip install maturin
cd tokenizers/bindings/python
unset CONDA_PREFIX # if using conda
maturin develop --release
cd ../../..
# Should now print "ready"
python doctor.pypython doctor.py reports the submodule checkout, the Rust version, the
SentencePiece CLI, and whether the built extension has the methods model loading
requires, and prints the fix for whatever is missing. Two build failures it
exists to catch, because neither error message names its own cause:
- A Cargo error about
?weak dependency features and the nightly channel means the toolchain is too old, not that a nightly one is needed. The fork'srust-toolchainfiles pin a version, so rustup users do not see this; anyone building with a distribution-packaged Rust needs 1.85 or newer. - A Cargo manifest parse error usually means the
tokenizerssubmodule was never checked out. Rungit submodule update --init tokenizers.
Optional extras: pip install -e ".[train]" for training (torch, transformers,
sentencepiece, ujson), pip install -e ".[dev]" for the test suite
(python -m pytest tests/). The suite passes on [dev] alone; the two tests
that exercise the SentencePiece training path skip unless both the spm_train
binary and the sentencepiece package are present, which needs the build in
Training below plus the [train] extra.
| Model | Languages | Training Data | Calibration | Download |
|---|---|---|---|---|
| unilid-1940-calibrated | 1940 language-script combinations | 60M samples | bundled (version-2 file) | HuggingFace Hub |
| unilid-1940 | 1940 language-script combinations | 60M samples | none (version-1 file) | HuggingFace Hub |
The two files store the same weights and differ only in the container. The
calibrated file bundles the calibration artifact (161 KB), which the same
repository also offers on its own as calibration.json. The stored weights are
the base ones either way, so that one download serves both modes: pass
calibrated=False for base inference. unilid-1940.unilid is the version-1
file, for base inference without the bundled calibration and for supplying a
calibration artifact of your own. Either file is 780 MB; loading builds the
float32 weight matrix in memory, so plan for roughly 2 to 3 GB of free RAM.
pip install huggingface_hub
python -c "from huggingface_hub import hf_hub_download; \
print(hf_hub_download('cmeister/unilid-1940', 'unilid-1940-calibrated.unilid', local_dir='.'))"
# or directly:
# wget https://huggingface.co/cmeister/unilid-1940/resolve/main/unilid-1940-calibrated.unilid
# the version-1 file, with no bundled calibration:
# wget https://huggingface.co/cmeister/unilid-1940/resolve/main/unilid-1940.unilidfrom unilid import load_model
model = load_model("unilid-1940-calibrated.unilid")
lang, tokens, score = model.predict("The quick brown fox jumps over the lazy dog.")
print(lang) # 'eng_Latn'
texts = [
"The quick brown fox jumps over the lazy dog.",
"Der schnelle braune Fuchs springt über den faulen Hund.",
"Le renard brun rapide saute par-dessus le chien paresseux.",
]
for text, (lang, tokens, score) in zip(texts, model.predict_batch(texts)):
print(f"{lang}: {text[:50]}")Prediction defaults to calibrated inference: a shared constant replaces each language's unseen-token log-probabilities at load time, and close decisions that land in two error-prone groups of languages are re-examined against each language's own threshold. On the GlotLID-C test pool this raises macro F1 from 0.933 to 0.956. The mechanism, the constants, and the full measured effects (including the evaluations where calibration lowers a metric) are described in REPRODUCING.md and specified in the UNILID paper.
Base (uncalibrated) inference is one flag away:
base_model = load_model("unilid-1940-calibrated.unilid", calibrated=False)Loading a model that has no calibration artifact (any version-1 .unilid
file, self-trained models included) with default arguments raises
UnilidCalibrationError; pass calibrated=False for such files.
Models trained before 0.3.0 placed a fifth of every row's probability on each of the four special tokens. Such a file still loads and still reproduces the numbers published against it. To tell the two generations apart, sum the probabilities of a row's real tokens: 1.0 for a file trained by 0.3.0 or later, 0.2 for an earlier one. Version 0.3.0 and later measure this at every load and print it.
Batch inference uses Rayon and defaults to all CPU cores; limit it with
RAYON_NUM_THREADS=4 python ....
Scoring cost is linear in the number of languages, so restricting the model to the languages a deployment can actually encounter reduces latency proportionally (1,940 to 100 languages cuts the scoring loop by roughly 19x).
model = load_model("unilid-1940-calibrated.unilid",
languages=["eng_Latn", "deu_Latn", "fra_Latn"])or write a smaller model file once and load it anywhere:
unilid-calibrate subset unilid-1940-calibrated.unilid -o european.unilid \
--langs-file my_languages.txtBase (uncalibrated) predictions on a subset are exact: the decision rule is an
argmax over the included languages. Under calibrated inference the
re-examination thresholds are carried over from the full model by default;
because removing languages can only raise a prediction's margin, carried
thresholds make re-examination fire at most as often as calibrated, and
otherwise behave identically. Re-estimating the thresholds against the subset
model is optional: pass --recalibrate <corpus_dir> (expects
<corpus_dir>/<lang>_train.txt per retained low-resource language) to run the
per-language threshold recipe against the subset.
A new language can be added to an existing calibrated model without retraining
anything else. A complete runnable walkthrough on toy data ships with the
repository: bash examples/add_language/run_example.sh builds a small
calibrated model from scratch, adds a fourth language, and compares predictions
before and after (see
examples/add_language/README.md).
unilid-add-language unilid-1940-calibrated.unilid xyz_Latn xyz_train.txt -o extended.unilidfrom unilid import add_language
summary = add_language("unilid-1940-calibrated.unilid", "xyz_Latn",
"xyz_train.txt", "extended.unilid")This trains the new language's token distribution over the model's existing
vocabulary (fixed-vocabulary EM; the default sp method needs the SentencePiece
CLI, --method em is pure Python), appends its weight row (existing rows are
copied byte-identically), and calibrates it from its own data alone: a
re-examination threshold is estimated from its own training lines when it has
fewer than 18,000 of them, and it becomes a replacement candidate only with at
least 100,000. Three caveats, stated here and in the paper: the four-language
high-entropy group is not recomputed (the one non-incremental piece of the
calibration), existing languages' thresholds are kept, and the 100,000-line
candidate requirement is a per-artifact constant that an uncapped corpus
deployment should choose deliberately.
The new row is put on the same scale as the model's existing rows before it is stored, and the command prints what it did. Only the probability mass a row places on tokens that can affect a score is comparable across languages, and a row carrying more of it than the others scores higher by a constant per token, for reasons that have nothing to do with the language. Models written before special tokens were excluded from the distribution hold part of their mass on those tokens, the 2026-08-11 release exactly 0.2 of it, so a freshly trained row is scaled down to match rather than being given a silent advantage over all 1,940 languages already in the file. For a model trained by 0.3.0 or later there is nothing to scale down, and the command reports which of the two cases it found.
The new language is trained over the base model's existing vocabulary, which
add_language cannot extend. A language whose text uses byte values that
vocabulary does not cover has most of its input fall to <unk>, and its
accuracy stays poor however much training data it has. For a real language that
means starting from the released 1,940-language model rather than from a small
model of your own: the toy walkthrough's 0.98 comes from a 300-token vocabulary
over constructed data and does not carry over. The third bullet of
examples/add_language/README.md covers this.
unilid-calibrate manages the calibration artifact directly:
| Subcommand | Does |
|---|---|
export MODEL -o cal.json |
Write the bundled calibration to standalone JSON |
bundle MODEL cal.json -o OUT |
Attach a calibration to a version-1 model (writes version 2) |
estimate MODEL LANG train.txt -o OUT |
Re-estimate one language's threshold |
subset MODEL -o OUT --langs ... [--recalibrate DIR] |
Restrict to a language subset |
train.py is the single entry point for training a model from scratch. It
loads data, splits per-language corpus files, trains a shared base tokenizer,
re-estimates per-language token probabilities, and saves everything plus a
training_summary.json. A model trained this way is a base model with no
calibration artifact; load it with calibrated=False (deriving a full
calibration is described in the paper's development-protocol appendix, and
examples/add_language/build_calibration.py
shows the bootstrap on a small model).
The default per-language method (sp) requires the forked SentencePiece CLI
(both the pip package from the [train] extra and the compiled binary):
git submodule update --init sentencepiece # not pulled by the install steps above
cd sentencepiece && mkdir -p build && cd build
cmake .. && make -j$(nproc) && sudo make install # or -DCMAKE_INSTALL_PREFIX=$HOME/.local
cd ../..Verify with spm_train --help, or python doctor.py.
Which steps need that binary, exactly. The base tokenizer and the per-language probabilities are two separate training steps with separate flags:
| Step | Flag | Needs spm_train |
|---|---|---|
| Base vocabulary | --base-training-method hf (default) or bpe |
no |
| Base vocabulary | --base-training-method soft or hard |
yes, unless both --base-seed-vocab hf and --base-em-impl custom are set |
| Per-language | --per-lang-counts-method sp (default) |
yes |
| Per-language | --per-lang-counts-method soft or hard |
no |
The custom EM base methods use SentencePiece for two sub-steps by default:
suffix-array seeding of the initial vocabulary (--base-seed-vocab) and the
maximization step (--base-em-impl). Both defaults are sp, so
--base-training-method soft alone still needs the binary. A missing binary is
an error naming the flag that asked for it, never a silent switch to another
method.
Provide exactly one of --fasttext, --wili-dir, or --tsv.
--fasttext FILE: one sample per line, __label__ plus the language code, a
space, then the text (the format of GlotLID
and FastText LID):
__label__eng Hello world
__label__deu Hallo Welt
--wili-dir DIR: a directory with aligned x_train.txt (texts) and
y_train.txt (language codes), as in
WiLI-2018.
--tsv FILE: tab-separated id, lang, text, as in
Tatoeba's sentences.csv.
# GlotLID-format data, 100K vocab, byte-level
python train.py --fasttext data/glotlid/train.txt --vocab-size 100000 \
--byte-level --max-base-samples-per-lang 10000
# Custom EM for both base and languages, with no SentencePiece binary anywhere
python train.py --wili-dir data/wili/ --vocab-size 50000 \
--base-training-method soft --base-seed-vocab hf --base-em-impl custom \
--per-lang-counts-method hard
# Resume a partially completed run
python train.py --fasttext data/train.txt --vocab-size 100000 \
--results-dir results_100k --reuse-corpus --reuse-base --skip-existing-langs
# Seed the vocabulary from an existing tokenizer (e.g. LLaMA)
python train.py --fasttext data/train.txt --vocab-size 100000 \
--initial-vocab path/to/llama/tokenizer.jsonInput (mutually exclusive, one required): --fasttext FILE,
--wili-dir DIR, --tsv FILE.
Training:
| Flag | Default | Description |
|---|---|---|
--vocab-size |
100000 |
Vocabulary size |
--base-training-method |
hf |
Base tokenizer training: hf (HuggingFace UnigramTrainer), bpe (HuggingFace BPE), soft (custom soft-EM), hard (custom hard-EM) |
--base-seed-vocab |
sp |
Initial vocabulary for the soft/hard base methods: sp (SentencePiece suffix-array seeding, needs spm_train), hf (pure Python). Ignored by hf and bpe. |
--base-em-impl |
sp |
EM implementation for the soft/hard base methods: sp (needs spm_train), custom (pure Python). Ignored by hf and bpe. |
--per-lang-counts-method |
sp |
Per-language probability estimation: sp (SentencePiece EM, C implementation), soft (custom soft-EM), hard (custom hard-EM). All use EM; sp is fastest. |
--byte-level / --char-level |
--byte-level |
Byte-level or character-level tokenization |
--initial-vocab FILE |
None | Seed vocabulary from an existing tokenizer (.json) or a text file (one token per line) |
--seed |
42 |
Random seed |
--max-samples |
None | Limit total input lines (for debugging) |
Sampling:
| Flag | Default | Description |
|---|---|---|
--max-base-samples-per-lang |
10000 |
Max samples per language for base tokenizer training |
--max-lang-samples-per-lang |
None | Cap per-language training data |
--shared-samples-per-lang |
None | Use the same subsample for base and per-language training |
Orchestration:
| Flag | Default | Description |
|---|---|---|
--lang-batch-size |
10 |
Languages trained per batch (controls memory) |
--results-dir |
results_{K}k |
Output directory |
--corpus-dir |
None | Reuse a pre-split corpus directory |
--base-tokenizer-path |
None | Path to load/save the base tokenizer |
--reuse-corpus / --no-reuse-corpus |
True |
Reuse existing corpus files if found |
--reuse-base / --no-reuse-base |
True |
Reuse an existing base tokenizer if found |
--skip-existing-langs / --no-skip-existing-langs |
True |
Skip languages with existing tokenizers |
results_100k/
training_summary.json # Full training config, timing, file paths
corpus/ # Per-language text files
corpus_base_sampled/ # Subsampled files for base training
tokenizers/
langspec_base_tokenizer.json # Shared base tokenizer
langspec_sp_eng.tokenizer.json
... # Per-language tokenizers with metadata
A trained model packs into a single binary file (one file instead of hundreds of JSON tokenizers, roughly 16x smaller, memory-mapped weights):
Header (32 bytes):
magic "UNILID\x00\x00"; version uint32 (1 = base model; 2 = calibration
appended); num_langs, vocab_size, base_tok_len, langs_len (uint32); 4
reserved bytes
Body:
base_tokenizer JSON; langs JSON array; weights float32[num_langs * vocab_size]
Version 2 only, after the weights:
calibration_len uint64 little-endian; calibration JSON
The stored weights are always the base (unclamped) matrix; the unseen-token constant is applied at load time when calibrated inference is active, so one file serves both modes. Package version 0.1.0 rejects version-2 files with an error rather than silently returning base predictions.
Each row is a distribution over the vocabulary's real tokens. The four special
tokens (<s>, </s>, <pad>, <unk>) hold no probability mass: their stored
weights are never read when scoring, because the scorer takes its unknown-token
score from a single model-wide constant and the other three are reachable only
by text containing those literal substrings. Mass placed on them would be mass
taken from the tokens that do decide a prediction, lowering all of them by a
constant. Models trained before version 0.3.0 do carry such mass, the
1,940-language model released on 2026-08-11 exactly 0.8 of every row, which
lowers each of its real tokens by 1.609 nats. That is one reason its
unseen-token values sit above the
-27.63 training floor, at a measured median of -17.66, though not the whole
reason: removing the mass moves the median only to -16.05, so most of the gap has
another origin. Those files still load and score exactly as before, and
add_language matches their scale when extending them.
Version 0.3.0 and later state which generation wrote a file, on every load. The
loader prints the smallest and largest real-token mass over the rows it loaded:
0.200 per row is what the sp training path produced before 0.3.0, and 1.000 is a
distribution normalized over the real tokens alone. The measurement is exact,
over every row and every column, and it is blocked so that a 1,940 by 100,000
matrix is never converted to float64 in one piece. The special columns are
located through the tokenizer by token string, not by position, because a base
tokenizer converted from an LLM's has them at non-contiguous high indices. For a
pre-0.3.0 sp file the loader adds a warning naming the two routes to the
corrected generation, then loads the file as before. A mass a little under 1.000
is what the em training path produced before 0.3.0, where the unknown token's
probability was the only mass outside the real tokens; that reading and any
other figure print a longer warning, and the file still loads. No part of the
report is fatal: a measurement that fails is itself reported, and the load
continues. eval.py prints the same report to standard error. The container
header cannot record the generation instead: it encodes only version 1 against
version 2, and FORMAT_VERSION_MAX = 2 makes a version-3 file unreadable to
every reader already published.
Pack, unpack, and bundle:
unilid-convert results_100k -o my_model.unilid # pack (version 1)
unilid-convert results_100k -o my_model.unilid --calibration cal.json # pack + bundle (version 2)
unilid-convert model.unilid --unpack # unpack (writes calibration.json for version 2)
python convert.py ... # same pack/unpack argumentsunpack_unilid writes the bundled calibration to calibration.json beside the
tokenizers, and save_unilid refuses to repack a directory containing
calibration.json unless the calibration is passed explicitly, so a calibrated
model cannot be downgraded to version 1 by accident.
eval.py streams predictions over a text file or evaluates against labeled
data. It scores in base (uncalibrated) mode and refuses a version-2 model file
unless --base is passed; calibrated predictions are produced through the
Python API.
# Predictions to TSV (text \t lang \t score) or JSONL
python eval.py --model model.unilid --input texts.txt --output predictions.tsv
python eval.py --model model.unilid --input texts.txt --output out.jsonl --format jsonl
# Metrics (accuracy, macro F1/precision/recall, throughput) on fastText-format labels
python eval.py --model model.unilid --input test.txt --fasttext --output results.json
python eval.py --model model.unilid --input test.txt --fasttext --lang-only # ignore scriptsInference:
from unilid import load_model
model = load_model("unilid-1940-calibrated.unilid") # calibrated (default)
model = load_model("model.unilid", calibrated=False) # base behavior
model = load_model("model.unilid", calibration="cal.json") # standalone calibration file
model = load_model("model.unilid", languages=["eng_Latn", "deu_Latn"]) # subset
lang, tokens, score = model.predict("Hello world")
results = model.predict_batch(["Hello world", "Hallo Welt"])
model.num_languages, model.langs # language inventory
model.calibrated # True under the default
model.calibration # the loaded Calibration (or None)
model.last_reexamination_stats # per-group counts of the last batch
tok = model.tokenizer # the shared base tokenizer
tok.encode("Hello world").tokenspredict and predict_batch return (lang, tokens, score); under calibrated
inference, tokens and score are the segmentation and score under the
finally predicted language.
Training and customization:
from unilid import (
add_language, # add one language to a calibrated model
train_standard_tokenizer, # one shared tokenizer over languages
train_language_specific_tokenizer, # base + per-language distributions
train_tokmix, # merge per-language vocabularies
save_unilid, unpack_unilid, # pack/unpack model files
Calibration, estimate_tau, # calibration artifact + threshold recipe
)Running the training helpers needs the [train] extra; they are imported lazily
so a prediction-only install, and the [dev] test suite, work without it.
CorpusTokenizer
(python -m unilid.corpus_tokenizer) batch-tokenizes corpora with existing
tokenizers. Lower-level trainer classes (StandardUnigramLMTokenizer,
LanguageSpecificUnigramLMTokenizer, EMUnigramTrainer) expose encode_lang,
best_language_encode, and the EM internals.
REPRODUCING.md maps the released files and modes to the paper's reported rows, describes the calibrated-inference mechanism in detail, lists the measured effects on GlotLID-C, UDHR, and CommonLID, and states the evaluation conventions. The paper is the specification of record.
predict and predict_batch accept forward=True in base mode, which scores
each language by marginalizing over all segmentations (log-sum-exp) instead of
taking the single best segmentation:
lang, tokens, score = base_model.predict(text, forward=True) # ~2x slowerThis mode exists for experimentation; the two decodings give nearly identical
accuracy in practice and there is no established use case where forward
scoring is preferable, so Viterbi (the default) is recommended. forward=True
is defined for the base model only and raises under calibrated inference: the
calibration thresholds are percentiles of Viterbi margins, so marginalized
scores would not match them.
UNILID/
train.py # Training entry point (CLI)
eval.py # Bulk prediction / evaluation (CLI, base inference)
convert.py # Pack/unpack .unilid files (CLI)
doctor.py # Setup checker (submodules, Rust, extension)
REPRODUCING.md # Paper-results reproduction guide
sentencepiece/ # Forked SentencePiece (git submodule)
tokenizers/ # Forked HF tokenizers with fast inference (git submodule)
examples/add_language/ # Runnable add-language walkthrough on toy data
tests/ # Unit + integration tests (pytest)
unilid/
model_io.py # .unilid format I/O, UnilidModel
calibration.py # Calibration artifact, clamp, gate + walk, thresholds
add_language.py # add_language() and unilid-add-language
calibrate_cli.py # unilid-calibrate (export/bundle/estimate/subset)
api.py # Training convenience functions
trainers/ # Base + per-language trainers (EM, SentencePiece)
algorithms/ # Viterbi, forward-backward, EM accumulation
constants.py, encoding.py, pruning.py, corpus_tokenizer.py, ...
If you use UNILID in your work, please cite the ICML 2026 paper:
@inproceedings{meister2026unilid,
title = {What Language is This? Ask Your Tokenizer},
author = {Meister, Clara and Yavuz, Ahmetcan and Lesci, Pietro and Pimentel, Tiago},
booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
year = {2026},
address = {Seoul, South Korea},
url = {https://arxiv.org/abs/2602.17655}
}