Run commands from the repository root after installation and checkpoint setup. Measurements below use NVIDIA B200 GPUs with PyTorch 2.9.1 / CUDA 12.8. For camera-controlled generation and Fig. 5, see the LingBot guide.
These commands profile each component and sum their latencies. Baselines require their upstream clones; eight-GPU runs require one node with NVSwitch (NVLink multicast). Comments give the paper's B200 measurements:
uv run python evaluation/latency/benchmark.py ff-s # 36 ms
uv run python evaluation/latency/benchmark.py ff-s --gpus 8 # 14 ms
uv run python evaluation/latency/benchmark.py ff-l # 158 ms
uv run python evaluation/latency/benchmark.py ff-l --gpus 8 # 45 ms
uv run --extra baselines python evaluation/latency/benchmark.py cfpp # 272 ms
uv run --extra baselines python evaluation/latency/benchmark.py self-forcing # 717 msWeights default to checkpoints/ (--checkpoint-dir); records and latency.json
go to outputs/latency/<method>_<gpus>gpu/ (--output-dir). The first run compiles
and autotunes; TORCHINDUCTOR_CACHE_DIR selects the compiler cache location.
Measurements use batch 1, BF16/TF32, and a full 21-latent-frame cache. After warmup, each component is timed over a sustained window (default 60 s), taking the median time per call and the slowest rank. The Wan VAE uses CUDA-event timing over streamed decodes. Step latency sums component medians and excludes text encoding, host transfer and display. Each latent frame produces four RGB frames.
| Method | Generation unit | Step latency |
|---|---|---|
| FastFrames | 1 latent frame | 2 DiT evaluations + KV refinement + pixel decoder |
| CF++ | 1 latent frame | 3 DiT evaluations + VAE decode |
| Self-Forcing | 3 latent frames | 5 DiT evaluations + 3 single-latent VAE decodes |
| CausVid | 3 latent frames | 4 DiT evaluations + 3 single-latent VAE decodes |
Baseline DiT counts include the clean KV-cache write. Timings vary with hardware conditions and compiler autotuning. Fig. 1 plots these latencies against VBench Total.
For the Fig. 2–3 components and Table 2 profiles:
CKPT=checkpoints
OUT=outputs/latency/components
P="uv run python evaluation/latency/profile_components.py"
$P dit --model 1.3b --dit $CKPT/causal-forcing++/framewise-2step.pt \
--timestep 1000 --output $OUT/dit_first.json
$P dit --model 1.3b --dit $CKPT/causal-forcing++/framewise-2step.pt --output $OUT/dit_final.json
$P kv-refine --model 1.3b --dit $CKPT/causal-forcing++/framewise-2step.pt \
--refiner $CKPT/fastframes-s/kv_refiner.safetensors --streams 6 --output $OUT/kv.json
$P pixel-decoder --model 1.3b --decoder $CKPT/fastframes-s/pixel_decoder.safetensors \
--output $OUT/decoder.json
$P vae --vae $CKPT/Wan2.1-T2V-1.3B/Wan2.1_VAE.pth --output $OUT/vae.json
uv run python evaluation/latency/summarize.py outputs/latency --output outputs/latency/summary.jsonUse --model 14b and FastFrames-L checkpoints for the larger model. For eight-GPU
DiT, KV-refinement and pixel-decoder profiles, replace uv run python with
uv run torchrun --standalone --nproc_per_node 8. KV refinement uses
--shard tokens for 1.3B and --shard layers for 14B. Set --streams 1 for
14B or multi-GPU KV profiles. Add
--resolution 720p for Table 2, using separate output filenames. Omit checkpoint
flags to profile random weights; full-step benchmarks require checkpoints.
DiT options include --mode baseline, --split-kv 1, and --chunk 3; VAE also
accepts --chunk 3. --dry-run prints a component's configuration without running it.
check_dit.py compares optimized and reference outputs with released weights:
uv run python evaluation/latency/check_dit.py --model 1.3b \
--dit $CKPT/causal-forcing++/framewise-2step.pt --fp32-reference --output $OUT/check.jsonThe summarizer selects the fastest matching component record and lists its candidates. Use separate result directories when comparing runs or environments.
uv run python scripts/generate.py --model ff-s --prompt "A corgi surfing a wave at sunset" \
--seed 0 --output outputs/corgi.mp4Use --model ff-l for FastFrames-L or --checkpoint-dir to change the weights
directory. Both produce 832 × 480 H.264 MP4s, with 81 frames at 16 fps.
For a file with one prompt per line, start one process per GPU:
for i in 0 1 2 3 4 5 6 7; do
CUDA_VISIBLE_DEVICES=$i uv run --no-sync python scripts/generate.py --model ff-l \
--prompt-file prompts.txt --output-dir outputs/ff-l --shard $i --num-shards 8 &
done; waitPrompt index i uses seed --seed + i and writes 0000.mp4, 0001.mp4, etc.
uv run python scripts/generate_project_page.py --model ff-s --output-dir outputs/project_page/ff-sUse --model ff-l for FastFrames-L. The script selects the project-page prompts
and model-specific noise convention. For Figs. 3–4, generate all 109
demo prompts, with stills at 0 and 5 seconds:
Q="uv run python scripts/generate_project_page.py --indices all --stills 0,5"
$Q --model ff-s --context dit --decoder vae --output-dir outputs/figures/cfpp
$Q --model ff-s --context dit --decoder pixel --output-dir outputs/figures/pixel
$Q --model ff-s --output-dir outputs/figures/ff-s
$Q --model ff-l --output-dir outputs/figures/ff-lThe first two commands compare decoders on the same latents. Zero-based prompt
index i uses seed 1000003 * i; outputs include videos, stills and records of
the prompt, seed and latent hash.
Complete the VBench environment and weight setup, which
sets VBENCH_PYTHON and VBENCH_CACHE_DIR. Generation uses the FastFrames
environment; scoring uses its separate pinned environment.
- The 946 prompt rows contain 944 unique prompts. Extended prompts pair by row
with originals; the last occurrence supplies duplicate prompts' extensions.
FastFrames, Self-Forcing and CF++ use Self-Forcing extended prompts; CausVid uses
VBench's GPT-enhanced
all_dimension_longer.txt. - Generate five samples per prompt, except the 75 temporal-flickering prompts, which get 25 candidates. RAFT visits these in numeric order and selects the first five static clips. This yields 6,220 raw videos and up to 375 filtered clips.
- Sample
kof prompti, inVBench_full_info.jsonorder, uses seed25 * i + k. Save as<original_prompt>-<sample_index>.mp4: 832 × 480, 81 frames, 16 fps, H.264 with the MP4 index first. - Score temporal flickering on the filtered clips and the other 15 dimensions on
raw videos in
vbench_standardmode, using eight B200 GPUs and metric RNG seed 0.
Prompt files, pairings and seeds are pinned by hash. Inspect them with
uv run python evaluation/vbench/protocol.py summary.
RUN=outputs/vbench/ff-s
for i in 0 1 2 3 4 5 6 7; do
CUDA_VISIBLE_DEVICES=$i uv run python evaluation/vbench/generate_corpus.py --model ff-s \
--checkpoint-dir checkpoints --output-dir $RUN/raw_videos --shard $i --num-shards 8 &
done; wait
GPUS="0 1 2 3 4 5 6 7" uv run bash evaluation/vbench/score.sh $RUN ff-sUse ff-l for FastFrames-L. After baseline setup,
generate and score CF++ with:
RUN=outputs/vbench/cfpp
for i in 0 1 2 3 4 5 6 7; do
CUDA_VISIBLE_DEVICES=$i uv run --extra baselines python evaluation/vbench/generate_baselines.py --model cfpp \
--upstream upstream/Causal-Forcing --checkpoint checkpoints/causal-forcing++/framewise-2step.pt \
--output-dir $RUN/raw_videos --shard $i --num-shards 8 &
done; wait
GPUS="0 1 2 3 4 5 6 7" uv run bash evaluation/vbench/score.sh $RUN cfpp| Baseline | --upstream |
--checkpoint |
Denoising steps / latent frames per chunk |
|---|---|---|---|
self-forcing |
upstream/Self-Forcing |
checkpoints/self-forcing/checkpoints/self_forcing_dmd.pt |
4 / 3 |
cfpp |
upstream/Causal-Forcing |
checkpoints/causal-forcing++/framewise-2step.pt |
2 (4 on the first frame) / 1 |
causvid |
upstream/CausVid |
checkpoints/causvid/autoregressive_checkpoint/model.pt |
3 / 3 |
Baselines generate one video per call, without classifier-free guidance. To use
another generator, run uv run python evaluation/vbench/protocol.py jobs --output jobs.jsonl;
condition on each row's conditioning_prompt, use its seed, and save as filename.
protocol.write_faststart_video writes the required format with OpenH264.
Generation reruns skip valid existing videos. Remove leftover .vbench-*.mp4
temporary files from interrupted writes before scoring. score.sh verifies,
filters and scores the corpus, writing score_summary.json, per-dimension results
in vbench_results/, media checks, filter provenance and logs/ under $RUN.
Restart the same scoring command to reuse completed work. After changing raw videos,
delete temporal_filter_contract.json, filtered_temporal/ and vbench_results/
before scoring again. Recompute aggregates with
uv run python evaluation/vbench/summarize.py --from-summary "$RUN/score_summary.json".
| Model | Total | Quality | Semantic |
|---|---|---|---|
| CausVid | 83.18 | 84.15 | 79.31 |
| Self-Forcing | 84.10 | 84.84 | 81.13 |
| Causal Forcing++ | 84.11 | 85.09 | 80.20 |
| FastFrames-S | 84.23 | 85.30 | 79.92 |
| FastFrames-L | 84.44 | 85.48 | 80.29 |
Raw dimension scores back these rows. Keep the scorer, metric weights, dependencies and GPU count fixed when comparing runs.
Scoring uses vbench==0.1.5 with
upstream updates followed by
local changes.
Score-affecting changes disable MUSIQ/ViCLIP dropout and average appearance-style
frame scores across GPUs. Script patches
document the evaluator and static filter changes. Other patches handle validation,
distributed output and numeric static-filter ordering.
scorer_manifest.json records file and tree hashes; filtering and scoring verify them. VBench is Apache-2.0. Verify the built tree or run CPU checks:
uv run python evaluation/vbench/setup_scorer.py --verify
uv run --extra dev python -m pytest evaluation/vbench/tests