feat(sdk)!: generalize llama_cpp speculative decoding to all types#1195
Draft
Mengsheng Wu (mengshengwu) wants to merge 5 commits into
Draft
feat(sdk)!: generalize llama_cpp speculative decoding to all types#1195Mengsheng Wu (mengshengwu) wants to merge 5 commits into
Mengsheng Wu (mengshengwu) wants to merge 5 commits into
Conversation
3 tasks
b10019 rolls a hexagon backend session-lifecycle rework that fails to open FastRPC session 0 on Windows-ARM64 after any prior HTP init in the same process (ADSP_AEE_QURT_ERR_DUPLICATE 0x80000406). b9775 is the last commit before the rework and runs clean on cpu / gpu (Adreno) / npu (HTP0). Signed-off-by: Mengsheng Wu <mengshen@qti.qualcomm.com>
Mengsheng Wu (mengshengwu)
force-pushed
the
feat/llama-cpp-speculative
branch
2 times, most recently
from
July 17, 2026 10:21
0b036f5 to
bc2497a
Compare
Replace the MTP-only spec_draft_model/spec_n_draft fields on geniex_ModelConfig with a generalized speculative config: spec_type (comma-separated llama.cpp type names) + spec_draft_model + spec_n_max/spec_n_min/spec_p_min. The plugin's setup_speculative now parses spec_type via common_speculative_types_from_names and covers all nine types on b9775 — draft-model (mtp / eagle3 / draft-simple, needs a draft GGUF) and self-speculative (ngram-simple / ngram-map-k / ngram-map-k4v / ngram-mod / ngram-cache, no draft model). The draft context path keeps our device-selection pinning so a Hexagon target target does not lose the drafter to default placement. geniex-bench mirrors the flags. Signed-off-by: Mengsheng Wu <mengshen@qti.qualcomm.com>
Wire the generalized speculative fields through geniex infer. --spec-type accepts one or a comma-separated set of llama.cpp type names; --draft-model resolves either a local GGUF path or a catalogue name (auto-pulling via the model manager, matching the main-model argument's behaviour). Non-llama_cpp runtimes warn and drop the spec flags rather than erroring. Signed-off-by: Mengsheng Wu <mengshen@qti.qualcomm.com>
Add SpecType / SpecDraftModel / SpecNMax / SpecNMin / SpecPMin to bindings/go/common.go's ModelConfig and its cgo fill/free helpers. Python and Android bindings will follow in separate PRs to keep this change focused on the SDK + Go CLI path. Signed-off-by: Mengsheng Wu <mengshen@qti.qualcomm.com>
reacquire_before_load() calls the HTP FastRPC session-reacquire proc unconditionally, which can crash on CPU/GPU inference paths when the ADSP domain is in a bad state. Only reacquire when the target device is actually HTP (npu). cpu / gpu targets no longer depend on the ADSP domain's health. Signed-off-by: Mengsheng Wu <mengshen@qti.qualcomm.com>
Mengsheng Wu (mengshengwu)
force-pushed
the
feat/llama-cpp-speculative
branch
from
July 17, 2026 14:04
bc2497a to
4d02d86
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Generalize
llama_cppspeculative decoding from the MTP-only shape of #1185 to the full set of speculative types llama.cpp exposes at b9775 (nine types, five needing no draft model at all).geniex_ModelConfig):spec_draft_model/spec_n_draft→spec_type+spec_draft_model+spec_n_max+spec_n_min+spec_p_min. Public API break, mirrored across Go / Python / Android per CONTRIBUTING §4.setup_speculativenow parsesspec_type(comma-separated) viacommon_speculative_types_from_names. Draft-model types (mtp/eagle3/draft-simple) load a GGUF into a shared-KV context, pinned to the target's device selection to keep the Hexagon path working. Self-speculative types (ngram-simple/ngram-map-k/ngram-map-k4v/ngram-mod/ngram-cache) skip the draft model entirely.--spec-type,--draft-min,--draft-p-min.--draft-modelaccepts a catalogue name (auto-pull) or a local GGUF path, matching the main-model argument.chore(sdk): pin llama.cpp back to b9775. b10019 rolls a hexagon backend session-lifecycle rework that failshtp_iface_openwithADSP_AEE_QURT_ERR_DUPLICATEon Windows-ARM64 after any prior HTP init in the same process; b9775 runs clean on cpu / gpu (Adreno) / npu (HTP0).Test plan
geniex-benchclean against pinned b9775 (GENIEX_PLUGIN_QAIRT=OFF).geniex-bench --spec-type ngram-mod(no draft model) on--device gpuand--device npu: draft/accept counters populate, no crashes.geniex-bench --spec-type mtp --draft-model <mtp-head.gguf>on--device gpu: same MTP acceptance behaviour as feat(sdk): add MTP speculative decoding to the llama_cpp backend #1185's evidence, unchanged plumbing.--draft-model, or on a non-llama_cpp runtime.Supersedes #1185 (MTP-only) — that PR can be closed once this lands.