Skip to content

Latest commit

 

History

History

wasm

README.md



Build GitHub


WebAssembly bindings over the Rust tokenizers library, for fast tokenization in the Browser.

Requirements

This package requires somewhat recent features of WebAssembly, including SIMD instructions.

Browser Minimum version
Chrome 96
Edge 96
Firefox 89
Safari (macOS and iOS) 16.4

In older browsers, importing the package throws a WebAssembly.CompileError.

Installation

npm install tokenizers-wasm@latest

Usage

import { Tokenizer } from "tokenizers-wasm";

const tokenizer = await Tokenizer.from_pretrained("openai-community/gpt2");
const ids = tokenizer.encode("Hello world!");
console.log(ids); // Uint32Array [15496, 995, 0]
console.log(tokenizer.decode_tokens(ids)); // ["Hello", "Ġworld", "!"]
console.log(tokenizer.decode(ids)); // "Hello world!"

The package loads its WebAssembly module on import, and the garbage collector frees the tokenizer object once it is unreachable.