This module is still experimental, expect the API to change or break. If used in a production setting, please pin the version number.
pip install --pre tokenizers
from tokenizers import Padding, Tokenizer
# A `Tokenizer` represents a tokenizer's spec (json) and can be edited.
# You can load it from the JSON file
tokenizer = Tokenizer.from_file("tokenizer.json")
# Or load the `tokenizer.json` of a model repo on the Hub.
# Downloads go through the `huggingface_hub` package
tokenizer = Tokenizer.from_pretrained("openai-community/gpt2", revision="main")
# `compile()` compiles the specification to an optimized tokenizer in rust.
# The resulting `CompiledTokenizer` object can encode text to token ids and decode ids back to text.
# It's immutable and can be shared among threads.
compiled = tokenizer.compile()
encoding = compiled.encode("Hello there, how are you?")
encoding.ids # [101, 7592, 2045, 1010, 2129, 2024, 2017, 1029, 102]
encoding.type_ids # [0, 0, 0, 0, 0, 0, 0, 0, 0]
encoding.attention_mask # [1, 1, 1, 1, 1, 1, 1, 1, 1]
# The same fields as numpy arrays. Each read returns a *new copy* that the caller owns.
encoding.ids_array # array([101, 7592, 2045, 1010, 2129, 2024, 2017, 1029, 102], dtype=uint32)
encoding.type_ids_array # array([0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=uint32)
encoding.attention_mask_array # array([1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=uint32)
torch.from_numpy(encoding.ids_array) # no-copy translation
compiled.encode("Hello", add_special_tokens=False).ids # no [CLS]/[SEP]
compiled.decode(encoding.ids) # "hello there, how are you?"
compiled.decode(encoding.ids_array) # a numpy array works too
compiled.decode(encoding.ids, skip_special_tokens=False)
# Pad one call. `length=None` pads each batch to its longest item.
short, long = compiled.encode_batch(
["Hello", "Hello there, how are you?"],
padding=Padding(direction="left", pad_id=0, pad_token="[PAD]"),
)
len(short) == len(long) # True
short.attention_mask # [0, 0, 0, 0, 0, 0, 1, 1, 1]
compiled.encode("Hello", padding=Padding(length=16)) # exactly 16 ids
# Padding and truncation can also be set on the spec.
# `padding=None` switches the default off for one call.
tokenizer.padding = Padding(pad_to_multiple_of=8)
padded = tokenizer.compile()
padded.encode("Hello", padding=None)examples/ holds a runnable script for each of these; make examples runs them all.
Every class here can be pickled, so a tokenizer can be sent to a worker process and its encodings
sent back: multiprocessing, a PyTorch DataLoader, datasets.map(num_proc=...).
A pickled CompiledTokenizer carries its whole tokenizer.json, and unpickling compiles it again.
With the "spawn" start method, we recommend compiling the tokenizer once per worker to avoid unpickle compilation overhead.
def compile_tokenizer(tokenizer):
global compiled
compiled = tokenizer.compile()
def encode(text):
return compiled.encode(text)
with multiprocessing.get_context("spawn").Pool(4, initializer=compile_tokenizer, initargs=(tokenizer,)) as pool:
encoded = list(pool.imap(encode, texts, chunksize=32))With the "fork" start method, compiling before starting the pool is recommended: the workers inherit the compiled tokenizer
and nothing is pickled.
make developregenerates the.pyistub, rebuilds the extension (debug) and reinstalls it in editable mode.make testdoes the above, then runstests/with pytest.make examplesdoes the above, then runs every script inexamples/.make stubsregeneratespython/tokenizers/tokenizers.pyi, including the memberspython/tokenizers/__init__.pyattaches to the Rust classes.make style/make check-styleregenerate the stub, then format/lint the Rust and Python sides and type-check withty. CI runsmake check-styleand fails if the regenerated stub differs from the committed one.