fix: prevent KV cache corruption on SWA/ISWA models + hot-path perf#2180

avion23 wants to merge 1 commit intoabetlen:mainabetlen/llama-cpp-python:mainfrom

avion23:fix/perf-and-iswaavion23/llama-cpp-python:fix/perf-and-iswaCopy head branch name to clipboard