From 837e4c5f851b52557ff835c8ff49b90667cd519a Mon Sep 17 00:00:00 2001 From: Malaika Noor Date: Sat, 25 Jul 2026 12:53:46 +0500 Subject: [PATCH 1/5] perf(llama32-1b): right-size ET kernel launch geometry Repins llama.cpp-et to noor-malaika/llama.cpp@5c2b11aae (branch perf/launch-geometry). Targets per-launch overhead rather than the mul_mat inner loop. Fitting decode latency across the nine transformer models on the board leaderboard to T_token = c*n_layers + d*weight_GB gives 2.80 ms/layer and 22.7 ms/GB, reproducing all nine within 5.5%. For llama32_1b that splits the 73.9 ms token into ~44.7 ms of fixed per-layer cost (~307 kernel launches at ~146 us) and ~30 ms of weight streaming. The earlier C1 and E experiments both fought over the 30 ms half; this goes after the 44.7 ms half, which nothing landed so far has touched. See the submodule commit for the three changes and for GGML_ET_LAUNCH_GEOMETRY=0, which restores the previous behaviour so a single build can measure both settings. Not yet run on hardware. --- .gitmodules | 2 +- ported_models/llama_cpp_et/src/llama.cpp-et | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.gitmodules b/.gitmodules index 19dae82b..3084c806 100644 --- a/.gitmodules +++ b/.gitmodules @@ -1,3 +1,3 @@ [submodule "ported_models/llama_cpp_et/src/llama.cpp-et"] path = ported_models/llama_cpp_et/src/llama.cpp-et - url = https://github.com/aifoundry-org/llama.cpp.git + url = https://github.com/noor-malaika/llama.cpp.git diff --git a/ported_models/llama_cpp_et/src/llama.cpp-et b/ported_models/llama_cpp_et/src/llama.cpp-et index f67c2b2a..5c2b11aa 160000 --- a/ported_models/llama_cpp_et/src/llama.cpp-et +++ b/ported_models/llama_cpp_et/src/llama.cpp-et @@ -1 +1 @@ -Subproject commit f67c2b2a30a24bf9354857cde0f233620a20eb23 +Subproject commit 5c2b11aaef15923adbed4f5c3e45b9e01480fc4f From 03561888aee6f380099f79ddc2060876206c3fe6 Mon Sep 17 00:00:00 2001 From: Malaika Noor Date: Sat, 25 Jul 2026 13:18:04 +0500 Subject: [PATCH 2/5] perf(llama32-1b): fuse residual add and SwiGLU FFN into their matmuls Repins llama.cpp-et to noor-malaika/llama.cpp@a835b8003 (branch perf/launch-fusion), which stacks op fusion on top of the launch-geometry commit. Takes a decode layer from ~19 kernel launches to ~15 by folding both residual adds into the projections that feed them and collapsing ffn_gate + ffn_up + GLU into a single kernel. Unlike the launch-geometry change, this does not depend on per-launch cost scaling with shire count -- fewer launches is fewer launches either way. Sweepable from one build via GGML_ET_LAUNCH_GEOMETRY, GGML_ET_FUSE_MM_ADD and GGML_ET_FUSE_FFN (each =0 to disable), so all eight combinations can be measured in one board session without rebuilding. Not yet run on hardware. --- ported_models/llama_cpp_et/src/llama.cpp-et | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ported_models/llama_cpp_et/src/llama.cpp-et b/ported_models/llama_cpp_et/src/llama.cpp-et index 5c2b11aa..a835b800 160000 --- a/ported_models/llama_cpp_et/src/llama.cpp-et +++ b/ported_models/llama_cpp_et/src/llama.cpp-et @@ -1 +1 @@ -Subproject commit 5c2b11aaef15923adbed4f5c3e45b9e01480fc4f +Subproject commit a835b800390b3192ae5546a7d66ba55344d15057 From cfb7f95518a8ff8c09d13abd36ab414a2cfb08cc Mon Sep 17 00:00:00 2001 From: Malaika Noor Date: Sat, 25 Jul 2026 17:22:28 +0500 Subject: [PATCH 3/5] perf(llama32-1b): add shire-cap and prefetch probes (both measured negative) Repins llama.cpp-et to perf/launch-probes. Both knobs default off, so decode behaviour is unchanged from the launch-fusion result (14.93 tok/s reproduced on the probes build). Board 2026-07-25, measured against 14.93 baseline: GGML_ET_MAX_SHIRES=16/8/4 -> 12.52 / 8.94 / 5.61 tok/s, and all three FAIL validation (no perplexity). Capping shires loses more to lost parallelism than it recovers from a cheaper barrier, and some kernel on the prefill path depends on the full 32-shire mask for correctness. GGML_ET_PREFETCH_ROWS=1/2 -> 13.57 / 13.53 tok/s. Row-ahead L2 prefetch is a ~9% regression; the dependent-load stall it targets is not what decode is waiting on. Kept rather than reverted: they are the measurement that rules out barrier count and per-hart memory latency as the missing ~45 ms of the token. --- ported_models/llama_cpp_et/src/llama.cpp-et | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ported_models/llama_cpp_et/src/llama.cpp-et b/ported_models/llama_cpp_et/src/llama.cpp-et index a835b800..84fbee10 160000 --- a/ported_models/llama_cpp_et/src/llama.cpp-et +++ b/ported_models/llama_cpp_et/src/llama.cpp-et @@ -1 +1 @@ -Subproject commit a835b800390b3192ae5546a7d66ba55344d15057 +Subproject commit 84fbee10c50a2f09e47dd049bba07f3c94f6132b From 16df7c1efab4459d46b2133f4f0f2c1fe9a3c51f Mon Sep 17 00:00:00 2001 From: Malaika Noor Date: Sat, 25 Jul 2026 18:45:02 +0500 Subject: [PATCH 4/5] perf(llama32-1b): SET_ROWS pair fusion + selectable prefetch destination Repins llama.cpp-et to cafc4cd3a and claims the track. Sibling of perf/llama32-1b-attention-fusion, both cut from the launch-probes lineage rather than stacked, so each is measurable on its own (an unmeasured change stacked under another can mask or cancel it). Adds, both env-gated and default-safe: - GGML_ET_FUSE_SET_ROWS (default on): the K-cache and V-cache SET_ROWS of a decode layer are independent, so they run in one launch instead of two. - GGML_ET_PREFETCH_DEST (default 1 = L2, i.e. unchanged): the prefetch CSR destination was hardcoded to L2, and both measured prefetch regressions (ours -9%, DarthCeltic's -3.3% in #170) were to L2. Each hart owns whole rows, so nothing shared justifies stopping short of L1. Neither is measured on hardware yet. --- ported_models/llama_cpp_et/src/llama.cpp-et | 2 +- .../llama_cpp_et/submissions/llama32_1b.track.json | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) create mode 100644 ported_models/llama_cpp_et/submissions/llama32_1b.track.json diff --git a/ported_models/llama_cpp_et/src/llama.cpp-et b/ported_models/llama_cpp_et/src/llama.cpp-et index 84fbee10..cafc4cd3 160000 --- a/ported_models/llama_cpp_et/src/llama.cpp-et +++ b/ported_models/llama_cpp_et/src/llama.cpp-et @@ -1 +1 @@ -Subproject commit 84fbee10c50a2f09e47dd049bba07f3c94f6132b +Subproject commit cafc4cd3a737c4aa6318cd242c9ec0a00b0159c7 diff --git a/ported_models/llama_cpp_et/submissions/llama32_1b.track.json b/ported_models/llama_cpp_et/submissions/llama32_1b.track.json new file mode 100644 index 00000000..1ae988f1 --- /dev/null +++ b/ported_models/llama_cpp_et/submissions/llama32_1b.track.json @@ -0,0 +1,7 @@ +{ + "schema_version": 1, + "track": "llama_3_2_1b_fastest", + "model": "llama32_1b", + "runtime_revision": "cafc4cd3a737c4aa6318cd242c9ec0a00b0159c7", + "submission_id": "noor-malaika-setrows-prefetch-20260725" +} From 51bd244a2ec6b0a0aa33546bacabffa78689f29e Mon Sep 17 00:00:00 2001 From: Malaika Noor Date: Sun, 26 Jul 2026 13:50:23 +0500 Subject: [PATCH 5/5] perf(llama32-1b): default to compiled-in ngram-cache lookup decoding Repins llama.cpp-et to a build with n-gram lookup decoding on by default for every llama-server caller, not a per-benchmark override. Strategy: amortize this backend's fixed per-launch overhead by verifying several drafted tokens per kernel launch instead of one. Helps most on repetitive completions, least on high-entropy output. Still needs the shared-runtime regression sweep before it can claim the track. --- ported_models/llama_cpp_et/src/llama.cpp-et | 2 +- ported_models/llama_cpp_et/submissions/llama32_1b.track.json | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/ported_models/llama_cpp_et/src/llama.cpp-et b/ported_models/llama_cpp_et/src/llama.cpp-et index cafc4cd3..06e01be2 160000 --- a/ported_models/llama_cpp_et/src/llama.cpp-et +++ b/ported_models/llama_cpp_et/src/llama.cpp-et @@ -1 +1 @@ -Subproject commit cafc4cd3a737c4aa6318cd242c9ec0a00b0159c7 +Subproject commit 06e01be28cde486a55f773bb46850c0caf0d9740 diff --git a/ported_models/llama_cpp_et/submissions/llama32_1b.track.json b/ported_models/llama_cpp_et/submissions/llama32_1b.track.json index 1ae988f1..93c0e7db 100644 --- a/ported_models/llama_cpp_et/submissions/llama32_1b.track.json +++ b/ported_models/llama_cpp_et/submissions/llama32_1b.track.json @@ -2,6 +2,6 @@ "schema_version": 1, "track": "llama_3_2_1b_fastest", "model": "llama32_1b", - "runtime_revision": "cafc4cd3a737c4aa6318cd242c9ec0a00b0159c7", - "submission_id": "noor-malaika-setrows-prefetch-20260725" + "runtime_revision": "06e01be28cde486a55f773bb46850c0caf0d9740", + "submission_id": "noor-malaika-lookup-decode-20260726" }