perf(llama32-1b): default to compiled-in ngram-cache lookup decoding - #226
Open
noor-malaika wants to merge 5 commits into
Open
perf(llama32-1b): default to compiled-in ngram-cache lookup decoding#226noor-malaika wants to merge 5 commits into
noor-malaika wants to merge 5 commits into
Conversation
Repins llama.cpp-et to noor-malaika/llama.cpp@5c2b11aae (branch perf/launch-geometry). Targets per-launch overhead rather than the mul_mat inner loop. Fitting decode latency across the nine transformer models on the board leaderboard to T_token = c*n_layers + d*weight_GB gives 2.80 ms/layer and 22.7 ms/GB, reproducing all nine within 5.5%. For llama32_1b that splits the 73.9 ms token into ~44.7 ms of fixed per-layer cost (~307 kernel launches at ~146 us) and ~30 ms of weight streaming. The earlier C1 and E experiments both fought over the 30 ms half; this goes after the 44.7 ms half, which nothing landed so far has touched. See the submodule commit for the three changes and for GGML_ET_LAUNCH_GEOMETRY=0, which restores the previous behaviour so a single build can measure both settings. Not yet run on hardware.
Repins llama.cpp-et to noor-malaika/llama.cpp@a835b8003 (branch perf/launch-fusion), which stacks op fusion on top of the launch-geometry commit. Takes a decode layer from ~19 kernel launches to ~15 by folding both residual adds into the projections that feed them and collapsing ffn_gate + ffn_up + GLU into a single kernel. Unlike the launch-geometry change, this does not depend on per-launch cost scaling with shire count -- fewer launches is fewer launches either way. Sweepable from one build via GGML_ET_LAUNCH_GEOMETRY, GGML_ET_FUSE_MM_ADD and GGML_ET_FUSE_FFN (each =0 to disable), so all eight combinations can be measured in one board session without rebuilding. Not yet run on hardware.
…gative)
Repins llama.cpp-et to perf/launch-probes. Both knobs default off, so decode
behaviour is unchanged from the launch-fusion result (14.93 tok/s reproduced
on the probes build).
Board 2026-07-25, measured against 14.93 baseline:
GGML_ET_MAX_SHIRES=16/8/4 -> 12.52 / 8.94 / 5.61 tok/s, and all three FAIL
validation (no perplexity). Capping shires loses more to lost parallelism
than it recovers from a cheaper barrier, and some kernel on the prefill
path depends on the full 32-shire mask for correctness.
GGML_ET_PREFETCH_ROWS=1/2 -> 13.57 / 13.53 tok/s. Row-ahead L2 prefetch is a
~9% regression; the dependent-load stall it targets is not what decode is
waiting on.
Kept rather than reverted: they are the measurement that rules out barrier
count and per-hart memory latency as the missing ~45 ms of the token.
Repins llama.cpp-et to cafc4cd3a and claims the track. Sibling of perf/llama32-1b-attention-fusion, both cut from the launch-probes lineage rather than stacked, so each is measurable on its own (an unmeasured change stacked under another can mask or cancel it). Adds, both env-gated and default-safe: - GGML_ET_FUSE_SET_ROWS (default on): the K-cache and V-cache SET_ROWS of a decode layer are independent, so they run in one launch instead of two. - GGML_ET_PREFETCH_DEST (default 1 = L2, i.e. unchanged): the prefetch CSR destination was hardcoded to L2, and both measured prefetch regressions (ours -9%, DarthCeltic's -3.3% in aifoundry-org#170) were to L2. Each hart owns whole rows, so nothing shared justifies stopping short of L1. Neither is measured on hardware yet.
Repins llama.cpp-et to a build with n-gram lookup decoding on by default for every llama-server caller, not a per-benchmark override. Strategy: amortize this backend's fixed per-launch overhead by verifying several drafted tokens per kernel launch instead of one. Helps most on repetitive completions, least on high-entropy output. Still needs the shared-runtime regression sweep before it can claim the track.
github-actions
Bot
added
track: week-2-challenge
Contributor
Author
|
Retriggering CI (unrelated sim-emulator link failure in shared infra). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
llama.cpp-eton top of the existing fusion/launch-geometry stack to add a compiled-in runtime default that turns on n-gram lookup decoding for everyllama-servercaller on this runtime, rather than a per-benchmark CLI override.submissions/llama32_1b.track.jsonupdated to the exact shippedllama.cpp-etrevision.Test plan
llama-perplexitynever reads the speculative-decode default).🤖 Generated with Claude Code