Skip to content

export-llm-optimum.md

Latest commit

 

History

History
187 lines (133 loc) · 7.71 KB

export-llm-optimum.md

File metadata and controls

187 lines (133 loc) · 7.71 KB
Raw

Exporting LLMs with Hugging Face's Optimum ExecuTorch

Optimum ExecuTorch provides a streamlined way to export Hugging Face transformer models to ExecuTorch format. It offers seamless integration with the Hugging Face ecosystem, making it easy to export models directly from the Hugging Face Hub.

Transformers also includes an experimental ExecuTorch exporter for broad programmatic export of PreTrainedModel architectures to XNNPACK and CUDA. Its generative path returns independent graph components that the caller must orchestrate. Use Optimum ExecuTorch when you want its CLI, quantization, model wrappers, and tested task-level workflows.

Overview

Optimum ExecuTorch supports a much wider variety of model architectures compared to ExecuTorch's native export_llm API. While export_llm focuses on a limited set of highly optimized models (Llama, Qwen, Phi, and SmolLM) with advanced features like SpinQuant and attention sink, Optimum ExecuTorch can export diverse architectures including Gemma, Mistral, GPT-2, BERT, T5, Whisper, Voxtral, and many others.

Use Optimum ExecuTorch when:

  • You need to export models beyond the limited set supported by export_llm
  • Exporting directly from Hugging Face Hub model IDs, including model variants such as finetunes
  • You want a simpler interface with Hugging Face ecosystem integration

Use export_llm when:

  • Working with one of the highly optimized supported models (Llama, Qwen, Phi, SmolLM)
  • You need advanced optimizations like SpinQuant or attention sink
  • You need pt2e quantization for QNN/CoreML/Vulkan backends
  • Working with Llama models requiring custom checkpoints

See Exporting LLMs for details on using the native export_llm API.

Prerequisites

Installation

Install the released package from PyPI:

pip install optimum-executorch

To work from the latest source instead, use an isolated environment and run:

git clone https://github.com/huggingface/optimum-executorch.git
cd optimum-executorch
pip install '.[dev]'

For nightly or source builds of the underlying projects, run:

python install_dev.py

This may replace executorch, torch, torchao, transformers, and other dependencies with nightly or source builds, which is why a separate environment is recommended. It is not required for the released package workflow below.

Supported Models

Optimum ExecuTorch supports a wide range of model architectures including decoder-only LLMs (Llama, Qwen, Gemma, Mistral, etc.), multimodal models, vision models, audio models (Whisper), encoder models (BERT, RoBERTa), and seq2seq models (T5).

For the complete list of supported models, see the Optimum ExecuTorch documentation.

CLI Export

The optimum-cli command is the recommended way to export Hugging Face models. It provides a single invocation that downloads the model from the Hub, applies the configured optimizations, and writes the resulting .pte file.

Basic Export

optimum-cli export executorch \
    --model "HuggingFaceTB/SmolLM2-135M-Instruct" \
    --task "text-generation" \
    --recipe "xnnpack" \
    --output_dir="./smollm2_exported"

With Optimizations

Add custom SDPA, KV cache optimization, and quantization:

optimum-cli export executorch \
    --model "HuggingFaceTB/SmolLM2-135M-Instruct" \
    --task "text-generation" \
    --recipe "xnnpack" \
    --use_custom_sdpa \
    --use_custom_kv_cache \
    --qlinear 8da4w \
    --qembedding 8w \
    --output_dir="./smollm2_exported"

Available CLI Arguments

Key arguments for LLM export include --model, --task, --recipe (backend), --use_custom_sdpa, --use_custom_kv_cache, --qlinear (linear quantization), --qembedding (embedding quantization), and --max_seq_len.

For the complete list of arguments, run:

optimum-cli export executorch --help

Optimization Options

Custom Operators

Optimum ExecuTorch includes custom SDPA (~3x speedup) and custom KV cache (~2.5x speedup) operators. Enable with --use_custom_sdpa and --use_custom_kv_cache.

Quantization

Optimum ExecuTorch uses TorchAO for quantization. Common options:

  • --qlinear 8da4w: int8 dynamic activation + int4 weight (recommended)
  • --qembedding 4w or --qembedding 8w: int4/int8 embedding quantization

Example:

optimum-cli export executorch \
    --model "meta-llama/Llama-3.2-1B" \
    --task "text-generation" \
    --recipe "xnnpack" \
    --use_custom_sdpa \
    --use_custom_kv_cache \
    --qlinear 8da4w \
    --qembedding 4w \
    --output_dir="./llama32_1b"

Backend Support

The xnnpack recipe is the CPU path used in this guide. The current package also registers portable, cuda, cuda-windows, metal, and precision/compute-unit-specific Core ML recipes such as coreml_fp16. Applicability depends on the model, task, host, and installed dependencies; use the exact recipe documented by Optimum ExecuTorch for your target. A bare coreml recipe is not registered.

Exporting Different Model Types

Optimum ExecuTorch supports various model architectures with different tasks:

  • Decoder-only LLMs: Use --task text-generation
  • Multimodal LLMs: Use --task multimodal-text-to-text
  • Seq2Seq models (T5): Use --task text2text-generation
  • ASR models (Whisper): Use --task automatic-speech-recognition

For detailed examples of exporting each model type, see the Optimum ExecuTorch export guide.

Running Exported Models

Verifying Output with Python

After exporting, you can verify the model output in Python before deploying to device using classes from modeling.py, such as the ExecuTorchModelForCausalLM class for LLMs:

from optimum.executorch import ExecuTorchModelForCausalLM
from transformers import AutoTokenizer

# Load the exported model
model = ExecuTorchModelForCausalLM.from_pretrained("./smollm2_exported")
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M-Instruct")

# Generate text
generated_text = model.text_generation(
    tokenizer=tokenizer,
    prompt="Once upon a time",
    max_seq_len=128,
)
print(generated_text)

Running on Device

After verifying your model works correctly, deploy it to device:

Performance

For performance benchmarks and on-device metrics, see the Optimum ExecuTorch benchmarks and the ExecuTorch Benchmark Dashboard.

Additional Resources