perf(llama32-1b): SET_ROWS pair fusion + packed-params CONT kernel - #227
Open
noor-malaika wants to merge 5 commits into
Open
perf(llama32-1b): SET_ROWS pair fusion + packed-params CONT kernel#227noor-malaika wants to merge 5 commits into
noor-malaika wants to merge 5 commits into
Conversation
Repins llama.cpp-et to noor-malaika/llama.cpp@5c2b11aae (branch perf/launch-geometry). Targets per-launch overhead rather than the mul_mat inner loop. Fitting decode latency across the nine transformer models on the board leaderboard to T_token = c*n_layers + d*weight_GB gives 2.80 ms/layer and 22.7 ms/GB, reproducing all nine within 5.5%. For llama32_1b that splits the 73.9 ms token into ~44.7 ms of fixed per-layer cost (~307 kernel launches at ~146 us) and ~30 ms of weight streaming. The earlier C1 and E experiments both fought over the 30 ms half; this goes after the 44.7 ms half, which nothing landed so far has touched. See the submodule commit for the three changes and for GGML_ET_LAUNCH_GEOMETRY=0, which restores the previous behaviour so a single build can measure both settings. Not yet run on hardware.
Repins llama.cpp-et to noor-malaika/llama.cpp@a835b8003 (branch perf/launch-fusion), which stacks op fusion on top of the launch-geometry commit. Takes a decode layer from ~19 kernel launches to ~15 by folding both residual adds into the projections that feed them and collapsing ffn_gate + ffn_up + GLU into a single kernel. Unlike the launch-geometry change, this does not depend on per-launch cost scaling with shire count -- fewer launches is fewer launches either way. Sweepable from one build via GGML_ET_LAUNCH_GEOMETRY, GGML_ET_FUSE_MM_ADD and GGML_ET_FUSE_FFN (each =0 to disable), so all eight combinations can be measured in one board session without rebuilding. Not yet run on hardware.
…gative)
Repins llama.cpp-et to perf/launch-probes. Both knobs default off, so decode
behaviour is unchanged from the launch-fusion result (14.93 tok/s reproduced
on the probes build).
Board 2026-07-25, measured against 14.93 baseline:
GGML_ET_MAX_SHIRES=16/8/4 -> 12.52 / 8.94 / 5.61 tok/s, and all three FAIL
validation (no perplexity). Capping shires loses more to lost parallelism
than it recovers from a cheaper barrier, and some kernel on the prefill
path depends on the full 32-shire mask for correctness.
GGML_ET_PREFETCH_ROWS=1/2 -> 13.57 / 13.53 tok/s. Row-ahead L2 prefetch is a
~9% regression; the dependent-load stall it targets is not what decode is
waiting on.
Kept rather than reverted: they are the measurement that rules out barrier
count and per-hart memory latency as the missing ~45 ms of the token.
Repins llama.cpp-et to cafc4cd3a and claims the track. Sibling of perf/llama32-1b-attention-fusion, both cut from the launch-probes lineage rather than stacked, so each is measurable on its own (an unmeasured change stacked under another can mask or cancel it). Adds, both env-gated and default-safe: - GGML_ET_FUSE_SET_ROWS (default on): the K-cache and V-cache SET_ROWS of a decode layer are independent, so they run in one launch instead of two. - GGML_ET_PREFETCH_DEST (default 1 = L2, i.e. unchanged): the prefetch CSR destination was hardcoded to L2, and both measured prefetch regressions (ours -9%, DarthCeltic's -3.3% in aifoundry-org#170) were to L2. Each hart owns whole rows, so nothing shared justifies stopping short of L1. Neither is measured on hardware yet.
Repins llama.cpp-et past the setrows-pair fusion to add a packed kernel-arguments variant of the CONT op, kept under its own gate alongside the existing ones. Strategy: launches with parameter payloads over the runtime's inline argument limit take a slower dispatch path (extra staging buffer, forced barrier). Packing CONT's arguments under that limit avoids the slow path for that op; the other fused ops still carry their existing gates so every combination stays independently switchable from one build.
github-actions
Bot
added
track: week-2-challenge
Contributor
Author
|
Retriggering CI (unrelated sim-emulator link failure in shared infra). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
llama.cpp-etpast the existing launch-geometry / MUL_MAT+ADD / gate-up-GLU fusion stack to add two further per-launch-overhead reductions, each independently gated:SET_ROWSpair fusion: the two independent cache writes run in one launch instead of two.CONTkernel: launches whose parameter payload exceeds the runtime's inline-argument limit take a slower dispatch path (extra staging buffer, forced barrier); packingCONT's arguments under that limit avoids it for that op.GGML_ET_*env gate, so one build sweeps every combination.submissions/llama32_1b.track.jsonupdated to the exact shippedllama.cpp-etrevision.Test plan
🤖 Generated with Claude Code