WebAssembly bindings over the Rust tokenizers library,
for fast tokenization in the Browser.
This package requires somewhat recent features of WebAssembly, including SIMD instructions.
| Browser | Minimum version |
|---|---|
| Chrome | 96 |
| Edge | 96 |
| Firefox | 89 |
| Safari (macOS and iOS) | 16.4 |
In older browsers, importing the package throws a WebAssembly.CompileError.
npm install tokenizers-wasm@latestimport { Tokenizer } from "tokenizers-wasm";
const tokenizer = await Tokenizer.from_pretrained("openai-community/gpt2");
const ids = tokenizer.encode("Hello world!");
console.log(ids); // Uint32Array [15496, 995, 0]
console.log(tokenizer.decode_tokens(ids)); // ["Hello", "Ġworld", "!"]
console.log(tokenizer.decode(ids)); // "Hello world!"The package loads its WebAssembly module on import, and the garbage collector frees the tokenizer object once it is unreachable.
