Repository navigation
Conversation
send() checked `generating` but only set it after awaiting cancelBackgroundTask(). A conversation summary still reading its prompt can take tens of seconds to stop, and every tap on Send in that window passed the check and queued the question again. When the summary stopped, the queued sends resumed in the same millisecond with the same Date.now() message ids, and saving them failed with "UNIQUE constraint failed: chat_messages.id" (11 times on a Pixel 6a). A ref now marks the send as started before any await, the input clears and the busy state shows at once, and message ids get a random suffix. A failure before generation puts the question back in the input box. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Android chat sent the last 6 messages verbatim, counted in messages.
With the detailed personality each answer is ~500 tokens, so history
grew to ~1,700 tokens and the first word of a later question took up
to 48 s on a Pixel 6a (Qwen2.5-1.5B), against 1.6 s in a new chat. The
background conversation summary read the whole older history the same
way, and the next question waited for it.
budgetHistory (src/routing/historyBudget.ts, pure) keeps the newest turns
within 400 tokens: questions whole, earlier answers cut to their opening
sentences (~80 tokens), the latest exchange always. Used for the
answer's history and for the summary's input.
Measured on the Pixel 6a, same six questions in one chat, detailed:
first word 1.6-5.4 s (was 1.6-48 s), prompt under ~530 tokens (was up
to 1,916), decode 13-16 tok/s (was halving to 7 as the phone heated).
Follow-ups still resolve their topic ("What are the side effects?"
after a vaccines answer, "What did he publish?" after Darwin).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The warm-up that keeps the answer prompt's fixed start (persona, source rules, grounding) in the KV cache asked llama.rn for n_predict 0. On llama.rn 0.13 rc.6 that returns before the prompt is decoded: the warm-up logged "1 tokens in 0 ms" every time, even right after the session title had replaced the cache, and every answer re-read its whole prompt although 229 of its first tokens matched the prefix (checked by tokenizing both with the model's own chat template, Pixel 6a, Qwen3-4B). WARM_N_PREDICT = 1 makes the decode loop run; the one token is discarded. Measured on the same phone: the warm-up now processes the 240-token prefix, and answers read only what follows it (116 of 345 tokens, 374 of 603). On the 4B that is ~11 s less to the first word when the user pauses before asking; on the 1.5B a few seconds. The engine is shared, so iOS gets it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Retrieval saw only the question's own words, so follow-ups found nothing:
"What did he publish?" after a Darwin answer, "What are the side
effects?" after a vaccines answer (Pixel 6a, Qwen2.5-1.5B and Qwen3-4B:
no sources). The model resolved the reference from the history and
answered from memory, uncited.
followUpSearch (src/routing/followUp.ts, pure), from the previous
question in the history:
- a question that points back ("he", "it", "that"; Portuguese "ele",
"isso"...) is searched together with the previous question, then with
the previous question alone if that finds nothing;
- a short question with no name of its own is searched as written first,
and with the previous question only when that finds nothing;
- a question that names its own subject never borrows another topic.
Used by the router's retrieve step and the fixed-model path. On the
Pixel 6a, "What are the side effects?" now gets the Vaccine article, and
"What is the capital of Australia?" right after it doesn't pull vaccine
sources.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two hard stops cut answers mid-sentence: Max Output Tokens (512 by default; detailed answers from Qwen2.5-1.5B hit it) and the 120 s step timeout (Qwen3-4B on a Pixel 6a writes ~4 tok/s, so every detailed answer stopped at ~450 tokens). Raising them would trade a cut answer for minutes of waiting, so both stay. answerLength (src/routing/answerLength.ts, pure) sets the length to min(max tokens, measured decode speed x 90 s), from the per-model speed the chat already records, and the style reminder asks for "about N words, and finish it". A model never measured on the phone gets the max-tokens target only. Pixel 6a, Qwen3-4B, detailed: the target came out at ~210 words; the vaccines and French/Industrial Revolution answers ended on their own (247 tokens each, 68 s and 85 s total) instead of being cut at 120 s. Read as complete, shorter than before. Qwen2.5-1.5B (~15 tok/s) gets ~310 words, the 512-token setting being its limit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The answer length used the model's median decode speed and a fixed 90 s of writing. Over one conversation on a Pixel 6a (Qwen3-4B, detailed) the phone heated and decode fell from 4.9 to 2.5 tok/s, while the first word took up to 32 s of the 120 s step timeout: two answers were still cut. The length now comes from the model's latest successful answer when it is from the last 15 minutes (its speed and its time to the first word), else the median as before. Writing time = 120 s - time to the first word - 15 s of margin, between 30 and 90 s. Same conversation, same phone: the targets went from ~220 words to ~160 as the phone warmed, and every answer they sized finished within the limit (63-101 s). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
send() adds the new question to the chat before building the history, and messagesRef can already hold it when the history is read. The question then went into the prompt twice (as the last history turn and as the question), and the follow-up search took it as the previous question: "What did he publish? What did he publish?" found nothing (Pixel 6a). The race exists on main too. The history now leaves out this send's own messages, and the follow-up search never takes the current question as the previous one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The KV cache is the one allocation mmap can't page out. llama.rn takes cache_type_k / cache_type_v; a quantized V cache needs flash attention, so Android gets K and V at q8_0 with flash attention on, and iOS (flash attention off, see initWithCpuFallback) and the CPU fallback get K only, keeping V at f16. memoryFit sizes the cache with the matching bytes per element. Pixel 6a, Qwen3-4B, n_ctx 3072: the app's resident anonymous memory after a fresh start went from ~1.27-1.31 GB to ~1.18 GB. Smaller than the ~200 MB the cache sizes predict; part of the app was in swap in both measurements. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On ARM, llama.cpp repacks the weights at load for its fast CPU kernels. The repacked copies live in the app's own (anonymous) memory, not the mmap'd file, so Android can't drop and re-read them; it compresses them into swap. On a Pixel 6a (6 GB) with Qwen3-4B that meant ~300-700 MB free and 1.9 GB of the app in swap; about one answer in six stalled at 0.1 tok/s while weights came back from swap, and Android killed the app twice while it loaded (2.8 GB anonymous at the load peak). shouldRepack (memoryFit.ts) repacks only when the whole file plus the buffers fits the memory budget with 0.75 GB to spare; otherwise no_extra_bufts keeps the weights as the mapped file. The 1.5B on the same phone and the 4B on 8-12 GB phones still repack. No RAM readouts: repack, as before. Pixel 6a, Qwen3-4B kept as the mapped file: ~2.7 GB free instead of ~0.3-0.7 GB, no stalls (slowest answer 3.6 tok/s), no kills. Decode as fast (3.6-5.3 tok/s); prompt reading ~2x slower, so the first word comes later (median 23-26 s vs 13 s in a six-question conversation). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The answer length used the previous answer's time to the first word, set before retrieval. A question with more sources or history reads longer before its first word: on a Pixel 6a (Qwen3-4B), a comparison question took 56 s against the previous answer's 11.5 s, and was cut at the 120 s step timeout. The engine now records the last answer's prompt reading speed (tokens evaluated / time to the first token) and how many tokens the warm prefix covers. The router step, once the sources are known, tokenizes the prompt it is about to send, subtracts the cached prefix, estimates the first word from that speed, and sets the length target from it (the previous answer's first word remains the fallback). The fixed-model fallback keeps the up-front target. Same six-question conversation on the Pixel 6a: estimated vs actual first word 19/23, 27/30, 54/57, 47/45 s; all six answers finished (longest 98 s), the first run where none was cut. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Raw results behind this PR: seven conversation runs driven through the real chat screen over adb (Qwen3-4B: main, three intermediate states, final; Qwen2.5-1.5B: main, final) and the eval:device standard (17) and Vitalik (6) sets on main and the branch, with JSONL, reports and answers. The README gives the method, the code each run used, the results and the caveats, including that single questions on the 4B are slower on a 6 GB phone with the final code (the repack rule keeps it as a mapped file there). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stricter rule (file + buffers + 0.75 GB) kept Qwen3-4B as a mapped file on a 6 GB Pixel 6a: stable in long conversations, but prompt reading ~2x slower, so single questions on the eval:device standard set took 23.0 s to the first word instead of 9.0 s on main. shouldRepack now keeps the mapped file only when the model file itself doesn't fit the memory budget, the case where repacking can't work at all (an 11 GB mixture of experts on 12 GB must stream its experts). The 4B on 6 GB repacks again: standard set, first word 11.6 s median, total 26.0 s average (main: 9.0 s, 25.3 s). Switching to the mapped file under memory pressure is left for a follow-up. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 8-bit KV cache (with the flash attention it needs) slowed prompt reading: on a Pixel 6a the 4B's first word on the eval:device standard set was 11.6 s median with it and 9.5 s without (main: 9.0 s; slower than main on 16 of 17 questions with it, 9 of 17 without), for ~100 MB. It now applies only when the weights are kept as the mapped file, i.e. a model too big for memory that streams from storage (the 35B deep tier on 12 GB), where a half-size cache matters. Every other model keeps the default f16 cache and llama.cpp's default flash attention, as on main. The pre-load fit check sizes the cache as f16, the larger case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…V when repacked Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The router can answer with another model than the chat's base model, but the length target used the base model's speed. The executor now asks for the picked model's latest speed. The engine also forgets the last prompt-reading rate when it unloads a model, so the first-word estimate never uses another model's rate. (CodeRabbit review on rferrari#72) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The length target used the max-tokens setting, but generation stops at the step's own cap (step.maxTokens ?? 512); when that is lower, the model was asked for more words than it can write. The conversation driver also stops when the database pull fails instead of exporting an empty file. (CodeRabbit review on rferrari#72) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… stalls Repacked weights live in the app's own memory, which Android can only swap. On a 6 GB phone that occasionally stalls an answer at 0.1 tok/s, or gets the app killed while loading; the crash guard then blocked the model and fell back to another one. - A repacked load that killed the app is no longer a CR-2 crash: the marker records that the load repacked, and the next load of that model keeps the mapped file, once. If that load dies too, it is a crash as before. - On Android, an answer that writes under 1 tok/s for 10 s or more with repacked weights reloads the model as the mapped file before the next generation (questions sent meanwhile wait for it), and it stays mapped until the app restarts. Nothing changes until one of those happens: shouldRepack still repacks every model whose file fits the budget. Choosing from the memory free at load was tried and dropped: on a Pixel 6a with 1.8-2.1 GB free the mapped file took 25 s to the first word against 10 s repacked, and the repacked load was not killed in three tries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xel 6a Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 5 minutes. View limit detailsLimit details: You’ve used the included review currently available. Review configuration: ⚙️ Run configuration
📒 Files selected for processing (72)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Owner
|
Hey @bgrana! Yeah, fully on board with this as we discussed in today's team meeting. I’m holding off on the final review until the current batch of in-progress tests finishes up. Once those results land, we'll dive back in and review again. Awesome work as always - let’s keep building BOAR together! 🐗 |
rferrari
added a commit
that referenced
this pull request
Oct 10, 2026
…t to RAM, thinking off by default, refill the prompt cache - LoadGuard (CR-2 + #73): a marker around each model load. A killed repacked load keeps the mapped file once (no_extra_bufts through JNI); a killed mapped load blocks the model and the chat answers with the previous or smallest model, saying so once. Picking the model again unblocks it. - Auto context from RAM (contextSizeForRam): 2048 up to 4.5 GB, 3072 up to 6.5 GB. - Thinking: an Advanced tweak, Auto off, on every answer path. - Weight repacking: an Advanced tweak, Auto on. - After a title or summary, the system prompt is prefilled again so the next question doesn't read it.
rferrari
added a commit
that referenced
this pull request
Oct 10, 2026
…d from storage (#73) An answer that writes under 1 tok/s for at least 10 s with repacked weights (swapped out on a 6 GB phone) marks the model: before the next generation it reloads with the weights mapped from storage, and stays mapped until the app restarts. A user who set repacking on in Advanced tweaks keeps it.
Owner
|
Reviewed against today's
Good to merge after #72 and a green CI run. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Keep the mapped file after a repacked load is killed or stalls
Depends on #72 (it's built on that branch; the last two commits are this PR's).
On ARM, llama.cpp repacks the weights into the app's own memory for its fast CPU kernels. Android can only swap that memory, not drop it and re-read it from the model file. On 6 GB phones that occasionally:
The mapped file (
no_extra_bufts) avoids both, but reads prompts about 2× slower. This PR keeps repacking by default and switches a model to the mapped file only once one of those actually happens:shouldRepackstill repacks every model whose file fits the budget.iOS is left out of the stall rule: it has no swap (jetsam ends the app instead), so a slow answer there is a slow model.
Why not choose from the memory free at load time
That was the first version. On the Pixel 6a with background apps open (1.8–2.1 GB free), it loaded the 4B as the mapped file, and single questions got much slower:
The repacked load wasn't killed in three tries with 1.6–2.4 GB free: Android stopped the background apps instead. Raw results:
docs/evidence/2026-10-06-repack-under-pressure/.Tests
loadMarker.test.ts: a repacked load that killed the app isn't a CR-2 crash, keeps the mapped file once, and a second death is a crash.LlamaEngine.test.ts: the marker carriesrepacked; the mapped file is used after a crash; a stalled answer reloads the model as the mapped file, a question sent meanwhile waits for it, and a re-request doesn't load a third time; no reload on iOS.memoryFit.test.ts:isDecodeStall.npm run typecheckis clean;npm testshows the same 9 failures asmainon my machine (SQLite-backedrag/import tests, Node 23 here), none new. No local CodeRabbit review (crisn't installed here).On the phone, the killed-load path was checked by writing the marker a killed repacked load leaves and restarting: the 4B loaded as the mapped file ("the last repacked load closed the app") and stayed the active model. The stall path is covered by unit tests only; no stall happened during these runs.
🤖 Generated with Claude Code