diff --git a/docs/evidence/2026-10-05-android-conversations/README.md b/docs/evidence/2026-10-05-android-conversations/README.md new file mode 100644 index 000000000..c0d27ff4d --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/README.md @@ -0,0 +1,90 @@ +# Android chat before/after, Pixel 6a (4–5 Oct 2026) + +Raw results behind PR #72 (`perf/android-chat-conversations`). Every number here comes from the phone: the app's +`execution_telemetry` table for conversations, and `npm run eval:device` JSONL for the question sets. + +## Setup + +- **Phone:** Google Pixel 6a, Tensor G1, 6 GB RAM (5.46 GiB usable), Android 16, dev build (`expo-dev-client`, arm64) served by + Metro over `adb reverse`, Wi-Fi adb. +- **Models:** Qwen3-4B-Instruct-2507 Q4_K_M (catalog default) and Qwen2.5-1.5B-Instruct Q4_K_M (default on 6–8 GB phones). +- **Library:** bundled corpus + Standard set. No knowledge packs. +- **Conditions:** on battery; runs started from a fresh app launch (force-stop, relaunch, model loaded) and waited until the + battery was at or below 33.0 °C with thermal status 0, for at most 30 minutes. Exceptions are marked in the tables. +- **Baseline:** `main` at a6633f8d (the branch's base before the rebase), served from a clean worktree to the same app. + +## Conversation protocol (`conversation/`) + +The harness runs single questions, so it can't see history growth, follow-ups or answers cut by the step timeout. These runs +drive the real chat screen over adb (`driver.py`: `uiautomator dump` to find the controls, `input text`, Send tapped once): +one new chat, six questions in order, a 30 s pause after each answer, personality **Thorough & Detailed**: + +1. How do vaccines work? 2. What are the side effects? 3. Who proposed the theory of evolution by natural selection? +4. What did he publish? 5. Compare the French Revolution and the Industrial Revolution. 6. What is the capital of Australia? + +Each `*.json` (the driver's `bench/LABEL.results.json`, renamed) holds the telemetry rows (`ttft_ms`, `tok_per_sec`, `tokens_generated`, `total_latency_ms`, …) and the +chat messages of the run. "Finished" means the answer ends with sentence punctuation instead of mid-sentence. + +| Run | Model | What changed | Code | First word, median | Slowest decode | Longest answer | Tokens | Finished | +|---|---|---|---|---|---|---|---|---| +| `base-4b` | Qwen3-4B | baseline | main a6633f8d | 45.4 s | 0.1 tok/s | 129 s | 1134 | 0 of 6 | +| `fix-4b` | Qwen3-4B | history budget, warm-up fix, follow-up search, length target | branch at 98fd3ed2 (first 5 changes) | 15.3 s | 2.5 tok/s | 122 s | 1644 | 4 of 6 | +| `fix2-4b` | Qwen3-4B | + length from the current speed, history race fixed | 98fd3ed2 + e334679c + 2c92437e (working copy) | 13.4 s | 0.1 tok/s | 127 s | 1187 | 5 of 6 | +| `norepack-4b` | Qwen3-4B | + weights kept as the mapped file, by hand | fix2 + 8-bit KV + no_extra_bufts forced on (working copy) | 22.9 s | 3.6 tok/s | 122 s | 1359 | 5 of 6 | +| `auto-4b` | Qwen3-4B | strict repack rule, 8-bit KV, length from each prompt | b19e7037 (working copy) | 26.3 s | 3.7 tok/s | 98 s | 1306 | 6 of 6 | +| `narrow-4b` | Qwen3-4B | final: narrowed repack rule, 8-bit KV only when streaming | PR head 4622454c; app started by hand, background apps stopped | 9.1 s | 3.7 tok/s | 85 s | 1580 | 6 of 6 | +| `base-15b` | Qwen2.5-1.5B | baseline | main a6633f8d | 20.0 s | 6.4 tok/s | 112 s | 2849 | 3 of 6 | +| `fix-15b` | Qwen2.5-1.5B | branch | b19e7037 (the 1.5B repacks under every rule) | 4.7 s | 8.0 tok/s | 48 s | 2478 | 4 of 6 | + +## Question sets (`harness/`) + +`npm run eval:device -- --models qwen3-4b` (standard set v1, 17 questions) and `--queries vitalik` (6), Succinct, no +history, questions back to back. Each folder has the JSONL (authoritative), the report from `npm run eval:summary`, and the +answers for grading where the run produced them. + +| Run | Question set | Code | OK | First word, median | Decode, median | Total, median | +|---|---|---|---|---|---|---| +| `base-4b-std` | standard (17) | main a6633f8d | 17/17 | 9.0 s | 5.5 tok/s | 21.8 s | +| `fix-4b-std` | standard (17) | 98fd3ed2; ran right after a low-memory kill and cold restart | 17/17 | 9.2 s | 4.5 tok/s | 28.6 s | +| `fix-4b-std-rerun` | standard (17) | 98fd3ed2, fresh start | 17/17 | 9.1 s | 5.0 tok/s | 21.4 s | +| `final-4b-std` | standard (17) | 429917b5: strict repack rule | 17/17 | 23.0 s | 5.0 tok/s | 34.3 s | +| `narrow-4b-std` | standard (17) | 568ad1c7: narrowed rule, 8-bit KV always | 17/17 | 11.6 s | 5.9 tok/s | 27.6 s | +| `narrow-f16kv-4b-std` | standard (17) | narrowed rule, 16-bit KV (what 4622454c runs for the 4B on this phone) | 17/17 | 9.5 s | 5.3 tok/s | 19.9 s | +| `base-4b-vitalik` | vitalik (6) | main a6633f8d | 6/6 | 1.0 s | 6.1 tok/s | 18.6 s | +| `fix-4b-vitalik` | vitalik (6) | 98fd3ed2 | 6/6 | 0.9 s | 6.3 tok/s | 13.9 s | +| `final-4b-vitalik` | vitalik (6) | 429917b5: strict repack rule | 6/6 | 1.7 s | 6.2 tok/s | 15.1 s | + +## Reading the results + +- **Final code, single questions (the harness):** same as `main`. Standard set first word 9.5 s vs 9.0 s median, total 25.9 s vs + 25.3 s average; per question the first word is slower than `main` on 9 of 17 (median ratio 1.00×). Not rerun for the Vitalik set + on the final code; on the earlier branch it was 13.9 s vs 18.6 s total. +- **Final code, conversations:** the 4B finished 6 of 6 answers (0 of 6 on `main`) with a 9.1 s median first word (45 s on `main`) + and no stalls; the 1.5B's first word went from a 20 s median to 4.7 s. +- **How the repack rule got here.** A strict rule (repack only with 0.75 GB to spare) kept the 4B as a mapped file on this phone: + stable, but prompt reading ~2× slower (`final-4b-std`, 23 s first word). The final rule keeps the mapped file only when the model + file itself doesn't fit the memory budget (a 35B mixture of experts on 12 GB), so the 4B repacks here (`narrow-*`). The 8-bit KV + cache cost ~19% of prompt reading (`narrow-4b-std` vs `narrow-f16kv-4b-std`), so it now applies only to streamed models. + +## Known risk on 6 GB phones (not fixed by this PR, same as `main`) + +With repacked weights the 4B's load peaks at ~2.8 GB of anonymous memory. When the phone has little free memory, Android's +low-memory killer stops the app during the load: three times in a row on 5 Oct (18:38, 18:49, 18:50; `am_low_memory` followed by +`am_proc_died team.sopa.aoair`, ~0.8 GB free before the load), after which the app's crash guard falls back to the 1.5B. Earlier +repacked runs also had occasional 0.1 tok/s stalls (`base-4b` answer 2, `fix2-4b` answer 3) while weights came back from swap. +A follow-up can decide at load time from the memory actually free, and switch to the mapped file after a stall. + +## Caveats + +- One phone, one room; n = 6 per conversation and 17 + 6 per question set. +- Runs marked "working copy" had dev-only console logging on top of the listed commits (timing lines, and a tokenizer check + before each answer that adds a few tens of milliseconds). No behavior change otherwise. +- `narrow-4b` started from the app as the user opened it (after the load-time kills above) with cached background apps stopped + (`am kill-all`), not from a scripted restart, since another restart risked another kill. It began with 2.2 GB of the app in swap. +- `narrow-f16kv-4b-std` ran the narrowed rule with the 8-bit cache disabled by hand, which is what the final code does for a + repacked model. +- `fix-4b-std` ran right after Android killed the app for low memory and is kept for completeness; `fix-4b-std-rerun` is the + comparable run. +- Answer quality was not scored. The answers are here for the team's judge. +- Qwen3-4B on a 6 GB phone is not the setup wizard's recommendation (it recommends the 4B from ~10 GB); it is used here because + it's the catalog default and shows the memory behavior most clearly. diff --git a/docs/evidence/2026-10-05-android-conversations/conversation/auto-4b.json b/docs/evidence/2026-10-05-android-conversations/conversation/auto-4b.json new file mode 100644 index 000000000..6f2de0cb4 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/conversation/auto-4b.json @@ -0,0 +1,134 @@ +{ + "label": "auto-4b", + "since_ms": 1791228483265, + "telemetry": [ + { + "created_at": 1791228554016, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "research", + "ttft_ms": 12383.707525968552, + "tok_per_sec": 5.4827027993605455, + "tokens_generated": 277, + "generation_latency_ms": 50522.526961028576, + "total_latency_ms": 63417.44566899538, + "outcome": "success" + }, + { + "created_at": 1791228719249, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 23189.493501007557, + "tok_per_sec": 4.843178910252969, + "tokens_generated": 314, + "generation_latency_ms": 64833.45047098398, + "total_latency_ms": 88836.05558598042, + "outcome": "success" + }, + { + "created_at": 1791228882876, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 29502.60438501835, + "tok_per_sec": 4.513697984150574, + "tokens_generated": 255, + "generation_latency_ms": 56494.69700795412, + "total_latency_ms": 86732.36727100611, + "outcome": "success" + }, + { + "created_at": 1791228986293, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 14253.910447955132, + "tok_per_sec": 4.640694837290158, + "tokens_generated": 184, + "generation_latency_ms": 39649.2349640131, + "total_latency_ms": 55022.6159529686, + "outcome": "success" + }, + { + "created_at": 1791229163581, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "compare", + "ttft_ms": 57122.61554700136, + "tok_per_sec": 3.7118641886819623, + "tokens_generated": 149, + "generation_latency_ms": 40141.55487000942, + "total_latency_ms": 98131.43177002668, + "outcome": "success" + }, + { + "created_at": 1791229292803, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 45009.53802597523, + "tok_per_sec": 3.873281368336207, + "tokens_generated": 127, + "generation_latency_ms": 32788.7359380126, + "total_latency_ms": 78621.58723700047, + "outcome": "success" + } + ], + "messages": [ + { + "created_at": 1791228490449, + "role": "user", + "text": "How do vaccines work?" + }, + { + "created_at": 1791228554021, + "role": "assistant", + "text": "Vaccines work by training the immune system to recognize and fight specific pathogens\u2014like viruses or bacteria\u2014without causing the full disease. They introduce a harmless version of the pathogen, such as a weakened or inactivated form, or a piece of it (like a surface protein), so the body can learn to identify it [1]. This exposure prompts the immune system to produce antibodies and memory cells, which remain active long after vaccination.\n\nUnlike natural infection, vaccines do not cause illness. For example, the measles vaccine uses a weakened virus that stimulates immunity without leading to the actual disease. This makes vaccination safer and more reliable. In contrast, natural infection carries risks like severe illness or complications, especially in vulnerable populations.\n\nThe immune response created by vaccines is highly specific and durable. Memory B and T cells \"remember\" the pathogen, allowing a faster, stronger response upon future exposure. This is like training a soldier to recognize an enemy\u2014once trained, they respond quickly when the threat reappears.\n\nComparatively, vaccines offer consistent, controlled immunity, while natural exposure varies in severity and outcome. Vaccines also allow for mass immunization, protecting entire communities through herd immunity. This prevents outbreaks and reduces the spread of disease, making public health interventions far more effective than relying on individual recoveries.\n\nIn essence, vaccines are a scientifically proven method to build immunity safely and efficiently." + }, + { + "created_at": 1791228630380, + "role": "user", + "text": "What are the side effects?" + }, + { + "created_at": 1791228719260, + "role": "assistant", + "text": "Vaccine side effects are typically mild and short-lived, often resembling common illnesses like fever or soreness at the injection site. These reactions occur because the immune system is actively responding to the vaccine, as it would to a real infection. For example, soreness at the injection site or fatigue may happen within hours to days after vaccination, and usually resolve within 24\u201348 hours [1].\n\nMore serious side effects are extremely rare. The U.S. Centers for Disease Control and Prevention (CDC) reports that adverse events from vaccines are overwhelmingly mild and often not different from those seen with natural infections. For instance, fever after a flu shot is more common and less severe than the fever from actual influenza.\n\nSome people may experience allergic reactions\u2014such as hives or anaphylaxis\u2014but these are rare and typically occur within minutes of vaccination. Such reactions are closely monitored, and immediate medical care is available. In contrast, natural infection carries a much higher risk of severe complications, including hospitalization or death.\n\nVaccines are rigorously tested and continuously monitored for safety. The benefits of vaccination\u2014such as preventing millions of cases of disease annually\u2014far outweigh the risks. For example, the measles vaccine prevents serious complications like pneumonia or encephalitis, which can occur in 1 in 1,000 cases of natural infection.\n\nIn summary, vaccine side effects are generally mild, transient, and well-managed, while the risks of contracting the actual disease are significantly greater. This makes vaccination one of the most effective public health tools available." + }, + { + "created_at": 1791228796130, + "role": "user", + "text": "Who proposed the theory of evolution by natural selection?" + }, + { + "created_at": 1791228882883, + "role": "assistant", + "text": "The theory of evolution by natural selection was primarily proposed by Charles Darwin, published in his 1859 work *On the Origin of Species*. Darwin synthesized observations from natural history, including those of Alfred Russel Wallace, who independently developed a similar idea. Darwin\u2019s key insight was that individuals with traits better suited to their environment are more likely to survive and reproduce, passing those advantageous traits to offspring over generations [1].\n\nUnlike Lamarckism, which suggested traits were passed on due to use or disuse, Darwin emphasized variation and differential survival. Natural selection acts on random genetic variation, favoring traits that enhance reproductive success\u2014such as camouflage in birds or drought-resistant plants in arid regions.\n\nDarwin\u2019s theory explained biodiversity through gradual, observable changes over time, supported by fossil records and biogeography. For example, the adaptive differences in finch beaks on the Gal\u00e1pagos Islands illustrate how environmental pressures shape evolution.\n\nWhile Wallace contributed significantly, Darwin was the first to present a comprehensive, testable mechanism. His theory laid the foundation for modern evolutionary biology, integrating genetics in the 20th century through the modern synthesis.\n\nThus, Darwin is credited as the originator of the theory, though it was refined through later scientific advances. [1]" + }, + { + "created_at": 1791228931238, + "role": "user", + "text": "What did he publish?" + }, + { + "created_at": 1791228986295, + "role": "assistant", + "text": "Charles Darwin published *On the Origin of Species* in 1859, which introduced the theory of evolution by natural selection as a comprehensive explanation for biodiversity. In this work, he demonstrated how small, inherited variations in organisms lead to differential survival and reproduction in changing environments. For instance, in the case of finches on the Gal\u00e1pagos Islands, those with beaks better suited to available food sources survived and passed on those traits [1].\n\nHe contrasted natural selection with Lamarckism, which claimed traits were passed based on use or disuse. Darwin\u2019s mechanism offered a testable, observable process\u2014unlike speculative theories. His examples, like the peppered moth during industrialization, showed how environmental pressure shifts allele frequencies over time.\n\nThe book was revolutionary because it unified biology, showing species are not fixed but descended from common ancestors. This framework laid the foundation for modern genetics and evolutionary biology." + }, + { + "created_at": 1791229065436, + "role": "user", + "text": "Compare the French Revolution and the Industrial Revolution." + }, + { + "created_at": 1791229163582, + "role": "assistant", + "text": "The French Revolution (1789\u20131799) was a political and social upheaval that dismantled monarchy and aristocracy, promoting ideals of liberty, equality, and fraternity [1]. In contrast, the Industrial Revolution (starting c. 1760) was an economic transformation marked by mechanization, urbanization, and rising productivity [2]. While the French Revolution changed societal structures and governance, the Industrial Revolution reshaped economies and labor systems. The former disrupted power hierarchies; the latter redefined production and class relations. Both were catalysts of modernity, but one focused on political change, the other on technological and economic advancement. Neither was a single event but a complex process with lasting global impacts." + }, + { + "created_at": 1791229214162, + "role": "user", + "text": "What is the capital of Australia?" + }, + { + "created_at": 1791229292804, + "role": "assistant", + "text": "The capital of Australia is Canberra. Unlike most countries, Australia does not have a capital city at a single geographic location, as it was deliberately chosen to be a compromise between Sydney and Melbourne. Canberra was established in 1913 as a neutral site, reflecting the nation\u2019s commitment to federal balance. It is located in the Australian Capital Territory, a non-contiguous territory designed to serve as a national center. This contrasts with capitals like London or Washington, D.C., which are central to historical or political developments. Canberra\u2019s design, with its planned layout and government institutions, symbolizes Australia\u2019s modern federalism and democratic governance." + } + ] +} \ No newline at end of file diff --git a/docs/evidence/2026-10-05-android-conversations/conversation/base-15b.json b/docs/evidence/2026-10-05-android-conversations/conversation/base-15b.json new file mode 100644 index 000000000..2e8b08fbb --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/conversation/base-15b.json @@ -0,0 +1,134 @@ +{ + "label": "base-15b", + "since_ms": 1791230985563, + "telemetry": [ + { + "created_at": 1791231021696, + "model_id": "qwen2.5-1.5b-instruct-q4km", + "task_type": "research", + "ttft_ms": 2090.1255300045013, + "tok_per_sec": 15.61549749061318, + "tokens_generated": 414, + "generation_latency_ms": 26512.123629033566, + "total_latency_ms": 29006.711073994637, + "outcome": "success" + }, + { + "created_at": 1791231139932, + "model_id": "qwen2.5-1.5b-instruct-q4km", + "task_type": "lookup", + "ttft_ms": 8227.47851896286, + "tok_per_sec": 12.419020163945847, + "tokens_generated": 416, + "generation_latency_ms": 33497.00656801462, + "total_latency_ms": 42119.32358801365, + "outcome": "success" + }, + { + "created_at": 1791231276609, + "model_id": "qwen2.5-1.5b-instruct-q4km", + "task_type": "lookup", + "ttft_ms": 14867.467537045479, + "tok_per_sec": 10.331995532565314, + "tokens_generated": 483, + "generation_latency_ms": 46747.987692952156, + "total_latency_ms": 62027.88487702608, + "outcome": "success" + }, + { + "created_at": 1791231441507, + "model_id": "qwen2.5-1.5b-instruct-q4km", + "task_type": "lookup", + "ttft_ms": 25162.57622396946, + "tok_per_sec": 8.066302871782652, + "tokens_generated": 512, + "generation_latency_ms": 63473.936962008476, + "total_latency_ms": 89089.24227201939, + "outcome": "success" + }, + { + "created_at": 1791231605900, + "model_id": "qwen2.5-1.5b-instruct-q4km", + "task_type": "compare", + "ttft_ms": 31463.43283599615, + "tok_per_sec": 6.3522168279783395, + "tokens_generated": 512, + "generation_latency_ms": 80601.7826319933, + "total_latency_ms": 112495.12664097548, + "outcome": "success" + }, + { + "created_at": 1791231760193, + "model_id": "qwen2.5-1.5b-instruct-q4km", + "task_type": "lookup", + "ttft_ms": 30791.74501001835, + "tok_per_sec": 7.260327007837067, + "tokens_generated": 512, + "generation_latency_ms": 70520.23957699537, + "total_latency_ms": 102065.68388801813, + "outcome": "success" + } + ], + "messages": [ + { + "created_at": 1791230992533, + "role": "user", + "text": "How do vaccines work?" + }, + { + "created_at": 1791231021698, + "role": "assistant", + "text": "Vaccines work by stimulating the immune system to recognize and respond to pathogens. They do this by introducing a weakened or inactive form of the pathogen into the body, which the immune system then identifies as foreign and mounts an immune response. This response includes the production of antibodies, which are proteins that specifically bind to the pathogen and mark it for destruction by other immune cells. \n\nThe immune response also includes the activation of helper T cells, which release cytokines to attract additional immune cells and initiate inflammation. This creates a localized immune response against the pathogen. \n\nIf the pathogen invades the body later, the immune system is primed and can quickly recognize and eliminate it, preventing the disease. This is because the immune system remembers the specific pathogen, so it can recognize and neutralize it without the need for a full-scale attack. This is why vaccines are so effective - they allow the immune system to gain immunity without the risk of the disease itself.\n\nVaccines are compared to natural infection because they provide an equivalent level of immunity without the risk of serious illness or death. For example, the flu vaccine provides protection against the flu virus, which can cause serious illness and death. In contrast, the natural infection with the flu virus can result in flu symptoms, mild illness, or severe complications, such as pneumonia. \n\nThe effectiveness of vaccines is supported by rigorous scientific studies. For instance, the CDC recommends that all children receive the measles, mumps, and rubella (MMR) vaccine to protect against these three diseases. This vaccine has been shown to be 97% effective in preventing measles in children who receive it. \n\nIn conclusion, vaccines work by stimulating the immune system to recognize and respond to pathogens, allowing it to produce antibodies and activate helper T cells. This prepares the immune system to quickly neutralize the pathogen if it invades the body later. Vaccines are a safe and effective way to prevent serious illnesses and diseases, with the added benefit of the immune system remembering the specific pathogen to prevent future infections." + }, + { + "created_at": 1791231097576, + "role": "user", + "text": "What are the side effects?" + }, + { + "created_at": 1791231139932, + "role": "assistant", + "text": "The side effects of vaccines are generally minor and short-lived, with the most common being pain, swelling, and redness at the injection site, fatigue, and a low-grade fever. These symptoms are a sign that the immune system is working to fight the vaccine and do not indicate a serious side effect. \n\nSome individuals may experience more severe side effects, such as anaphylaxis or anaphylactic shock, which is an allergic reaction to the vaccine that can cause swelling, difficulty breathing, and low blood pressure. While rare, anaphylaxis is a serious condition that can be life-threatening. \n\nIn rare cases, vaccines can cause more severe side effects, such as severe allergic reactions, such as anaphylactic shock or anaphylaxis. However, these side effects are much less common than the side effects of natural infection. \n\nFor example, the MMR vaccine, which is used to prevent measles, mumps, and rubella, has been shown to be 97% effective in preventing measles in children who receive it. The vaccine is made up of a weakened form of the virus that can't cause disease, but is enough to trigger the immune system to produce antibodies. \n\nThe side effects of vaccines are compared to the risks of natural infection. For instance, natural measles can cause severe complications, such as pneumonia, ear infections, and brain inflammation. Natural mumps can cause meningitis, and natural rubella can cause heart damage. The risk of these complications is much higher with natural infection than with the vaccine, making the vaccine much safer and more effective. \n\nIn conclusion, the side effects of vaccines are generally minor and short-lived, with the most common being pain, swelling, and redness at the injection site, fatigue, and a low-grade fever. These symptoms are a sign that the immune system is working to fight the vaccine and do not indicate a serious side effect. The side effects of vaccines are compared to the risks of natural infection, with the vaccine being much safer and more effective in preventing serious illnesses and diseases." + }, + { + "created_at": 1791231214405, + "role": "user", + "text": "Who proposed the theory of evolution by natural selection?" + }, + { + "created_at": 1791231276610, + "role": "assistant", + "text": "The theory of evolution by natural selection was proposed by the Victorian naturalist Charles Darwin in the mid-19th century. Darwin was a member of the Cambridge University Natural History and Philosophical Museum, and his work was initially influenced by the work of earlier naturalists such as Linnaeus, Buffon, and Lamarck. \n\nDarwin's theory was based on his observations and research on the diversity of life on earth, including his experiences on the Beagle, a six-year-long voyage around the world. On the Beagle, Darwin collected thousands of specimens of plants and animals, and during his time on the ship, he developed his ideas about evolution and natural selection. \n\nDarwin's theory of natural selection posits that the survival of a species depends on the ability to adapt to its environment. Individuals within a species that are better adapted to their environment tend to survive and reproduce more successfully than those that are not. Over time, this leads to the evolution of new species and the extinction of others. \n\nDarwin's theory was not immediately accepted by the scientific community, and it faced intense criticism from many prominent scientists who believed in the theory of creationism or the doctrine of catastrophism. However, Darwin's theory was later supported by the work of other scientists, including Alfred Russel Wallace, who independently developed the theory in 1858, and the geneticist Gregor Mendel, who developed the theory of genetics.\n\nThe theory of evolution by natural selection has been supported by an enormous amount of evidence. For example, the fossil record shows that many species have evolved from earlier forms, and the genetic evidence shows that humans share a common ancestor with chimpanzees. The theory of natural selection also explains the diversity of life on earth, from the smallest bacteria to the largest mammals. \n\nIn conclusion, the theory of evolution by natural selection was proposed by Charles Darwin in the mid-19th century, based on his observations and research on the diversity of life on earth. Darwin's theory was initially met with criticism from many scientists, but it has been supported by an enormous amount of evidence, including the fossil record, genetic evidence, and the diversity of life on earth. The theory of natural selection has been a cornerstone of modern biology, and it has been supported by an enormous amount of evidence, including the fossil record, genetic evidence, and the diversity of life on earth." + }, + { + "created_at": 1791231352255, + "role": "user", + "text": "What did he publish?" + }, + { + "created_at": 1791231441508, + "role": "assistant", + "text": "Charles Darwin published his theory of evolution by natural selection in his groundbreaking work \"On the Origin of Species\" (1859), which is considered one of the most important scientific texts in the history of science. Darwin's theory proposed that species evolve over time through the process of natural selection, where the fittest individuals in a population are more likely to survive and reproduce, passing on their advantageous traits to future generations.\n\nDarwin's theory was revolutionary because it challenged the prevailing belief of the time, that species were fixed and immutable. Darwin's research was based on his observations and investigations, and he traveled the world to collect evidence, including the fossil record, the anatomy of animals, and the distribution of species. He also observed the diversity of life on earth and the adaptability of organisms to their environments.\n\nDarwin's theory was based on the principle of natural selection, which states that individuals with traits that are better suited to their environment are more likely to survive and reproduce. Over time, these traits become more common in the population, leading to the evolution of new species. Darwin also proposed the concept of the \"descent of man,\" suggesting that humans evolved from apes, which was a radical idea at the time.\n\nDarwin's theory was not immediately accepted, and it faced intense criticism from many prominent scientists, including the geologist Charles Lyell, who rejected the theory of natural selection. However, Darwin's theory was later supported by the work of other scientists, including Alfred Russel Wallace, who independently developed the theory in 1858. Darwin's theory was further supported by the work of the geneticist Gregor Mendel, who developed the theory of genetics, and it has been a cornerstone of modern biology.\n\nDarwin's theory of natural selection has been supported by an enormous amount of evidence, including the fossil record, genetic evidence, and the diversity of life on earth. For example, the fossil record shows that many species have evolved from earlier forms, and the genetic evidence shows that humans share a common ancestor with chimpanzees. The theory of natural selection also explains the diversity of life on earth, from the smallest bacteria to the largest mammals.\n\nIn conclusion, Charles Darwin published his theory of evolution by natural selection in his groundbreaking work \"On the Origin of Species\" (1859), which challenged the prevailing belief of the time that species were fixed and immutable. Darwin's theory was based on his observations and research, and it proposed that individuals with traits that are better suited to their environment are more likely to survive" + }, + { + "created_at": 1791231493017, + "role": "user", + "text": "Compare the French Revolution and the Industrial Revolution." + }, + { + "created_at": 1791231605900, + "role": "assistant", + "text": "The French Revolution and the Industrial Revolution were two of the most transformative events in modern history, each having a profound impact on society, culture, and the economy. The French Revolution, which began in 1789 and lasted until 1799, was a period of political and social upheaval that sought to establish a new system of government and a more equitable society. The Industrial Revolution, on the other hand, began in the late 18th century and was characterized by a rapid expansion of manufacturing and a shift from agrarian to industrial economies.\n\nThe French Revolution was marked by radical changes in government, society, and culture. It led to the rise of the bourgeoisie, the establishment of the First French Republic, and the rise of Napoleon Bonaparte. The revolution also saw the rise of the Enlightenment and the emergence of new ideas about human rights, democracy, and equality. The revolution was characterized by violence, as the royalists and the revolutionaries battled it out, and the French Revolution is often seen as a precursor to the French Empire, which lasted until Napoleon's defeat in 1815.\n\nThe Industrial Revolution, on the other hand, was characterized by a rapid expansion of manufacturing and a shift from agrarian to industrial economies. The Industrial Revolution began in Great Britain and spread to other parts of the world, including the United States and Europe. The Industrial Revolution led to the creation of new industries, such as textiles, iron, and coal mining, and it led to the rise of the industrial proletariat and the working class. The Industrial Revolution also led to the development of new technologies, such as the steam engine and the spinning jenny, and it led to the creation of new social classes, such as the industrial bourgeoisie.\n\nOne of the key differences between the French Revolution and the Industrial Revolution is the pace of change. The French Revolution was marked by violent upheaval and radical change, while the Industrial Revolution was characterized by gradual progress and steady change. The French Revolution was a period of political upheaval and social change, while the Industrial Revolution was a period of economic and technological progress.\n\nIn conclusion, the French Revolution and the Industrial Revolution were two of the most transformative events in modern history, each having a profound impact on society, culture, and the economy. The French Revolution was marked by radical changes in government, society, and culture, while the Industrial Revolution was characterized by a rapid expansion of manufacturing and a shift from agrarian to industrial economies. The pace of change was also different, with the French Revolution being a" + }, + { + "created_at": 1791231657859, + "role": "user", + "text": "What is the capital of Australia?" + }, + { + "created_at": 1791231760194, + "role": "assistant", + "text": "The capital of Australia is Canberra, located in the Australian Capital Territory. Canberra is the seat of the Australian government and is home to the Australian Parliament and the High Court of Australia. It was established in 1913 as a planned city and has since become a vibrant and diverse city.\n\nCanberra is situated at the intersection of two major waterways\u2014the Australian Alps and the Murray River\u2014and is surrounded by the Australian Capital Territory. The city is known for its unique architecture, which blends traditional Australian design with modern and international styles. The design of Canberra was influenced by the city of Bras\u00edlia in Brazil, which was also planned and built in a planned, futuristic manner.\n\nCanberra's location is significant for several reasons. It is located in a region known as the Great Dividing Range, which runs along the eastern coast of Australia and provides a natural barrier against the more extreme weather conditions found in the north and south. This natural protection has made Canberra a relatively mild climate for a tropical country like Australia.\n\nThe choice of Canberra as the capital city was influenced by the desire to create a central location for the government and administration of the newly federated nation. It was also intended to be a symbol of national unity and a place where the government could be more closely tied to the people. The city's layout was designed to be efficient and functional, with wide avenues and planned parks, which have since become iconic features of the city.\n\nCanberra has a diverse population, including many expatriates and tourists. It is known for its extensive parks and gardens, including the Australian War Memorial and the National Gallery of Australia. The city offers a range of cultural and recreational activities, including museums, art galleries, and theaters. Canberra is also a hub for politics and government, with many international conferences and diplomatic missions based in the city.\n\nCompared to other capital cities, Canberra is notable for its modern and futuristic design, which contrasts with the more traditional architecture of some other capitals. For instance, Paris is known for its iconic landmarks such as the Eiffel Tower and Notre-Dame Cathedral, while Bras\u00edlia is famous for its futuristic buildings and planned layout. Canberra, however, has been recognized as one of the world's most livable cities, and it continues to be a significant center for politics, culture, and industry in Australia.\n\nIn summary, Canberra's status as the capital of Australia is a testament to its strategic location, planned architecture, and modernity, making it a unique and important city in the heart of the nation. Its design and location have" + } + ] +} \ No newline at end of file diff --git a/docs/evidence/2026-10-05-android-conversations/conversation/base-4b.json b/docs/evidence/2026-10-05-android-conversations/conversation/base-4b.json new file mode 100644 index 000000000..a8e510462 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/conversation/base-4b.json @@ -0,0 +1,134 @@ +{ + "label": "base-4b", + "since_ms": 1791165452027, + "telemetry": [ + { + "created_at": 1791165584224, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "research", + "ttft_ms": 8045.093753993511, + "tok_per_sec": 3.9197071740767004, + "tokens_generated": 441, + "generation_latency_ms": 112508.40443301201, + "total_latency_ms": 121059.91062301397, + "outcome": "success" + }, + { + "created_at": 1791165795824, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 41053.99896198511, + "tok_per_sec": 0.10319181362007682, + "tokens_generated": 9, + "generation_latency_ms": 87216.22078603506, + "total_latency_ms": 128882.20481598377, + "outcome": "success" + }, + { + "created_at": 1791165975783, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 51977.43858295679, + "tok_per_sec": 2.1604258178954305, + "tokens_generated": 148, + "generation_latency_ms": 68505.01358300447, + "total_latency_ms": 121289.39381802082, + "outcome": "success" + }, + { + "created_at": 1791166145553, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 41134.18524098396, + "tok_per_sec": 2.735073584492747, + "tokens_generated": 217, + "generation_latency_ms": 79339.73010098934, + "total_latency_ms": 121129.125385046, + "outcome": "success" + }, + { + "created_at": 1791166321776, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "compare", + "ttft_ms": 49757.559675991535, + "tok_per_sec": 2.370846649928109, + "tokens_generated": 168, + "generation_latency_ms": 70860.76191604137, + "total_latency_ms": 121394.8571959734, + "outcome": "success" + }, + { + "created_at": 1791166492772, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 54643.299059033394, + "tok_per_sec": 2.289957559237557, + "tokens_generated": 151, + "generation_latency_ms": 65940.08670198917, + "total_latency_ms": 121452.3463320136, + "outcome": "success" + } + ], + "messages": [ + { + "created_at": 1791165463034, + "role": "user", + "text": "How do vaccines work?" + }, + { + "created_at": 1791165584257, + "role": "assistant", + "text": "Vaccines work by training the body\u2019s immune system to recognize and defend against specific pathogens\u2014such as viruses or bacteria\u2014without causing the full-blown disease. This process involves introducing a harmless or modified version of the pathogen (like a weakened or inactivated form, or a piece of it such as a surface protein) into the body [1]. When the immune system encounters this foreign agent, it mounts a response, producing specialized cells (like B cells and T cells) and antibodies that target the specific pathogen. These cells and antibodies are then stored in the body\u2019s memory, allowing a faster and stronger response if the actual pathogen ever enters the body in the future.\n\nOne key mechanism is the development of *immunological memory*. Unlike the initial exposure to a pathogen, which can take days or weeks to mount a defense, a vaccinated individual can respond in minutes. This is analogous to how a soldier learns a battlefield after a simulation\u2014once trained, the body can react quickly and effectively. For instance, the measles vaccine triggers memory B cells that can detect measles virus within hours of exposure, preventing the infection from taking hold. This is why vaccines are so effective at preventing diseases, even after years.\n\nAnother important concept is *antigenic specificity*. Vaccines are designed to introduce antigens\u2014molecules on the surface of pathogens that the immune system recognizes as foreign. For example, the influenza vaccine includes surface proteins like hemagglutinin, which are key in viral entry into cells. By targeting these proteins, the immune system learns to neutralize the virus before it can replicate. This specificity ensures that the immune response is focused and efficient, minimizing collateral damage and reducing the risk of autoimmune reactions.\n\nIt\u2019s also worth noting that vaccines don\u2019t always contain the whole pathogen. In contrast to a live virus, which might still cause illness, vaccines often use inactivated (killed) viruses, recombinant proteins, or viral vectors (like a harmless virus carrying the antigen). For example, the HPV vaccine uses virus-like particles that mimic the virus structure but don\u2019t cause disease. This approach enhances safety while maintaining immunogenicity. The difference between a vaccine and" + }, + { + "created_at": 1791165666781, + "role": "user", + "text": "What are the side effects?" + }, + { + "created_at": 1791165795833, + "role": "assistant", + "text": "Vaccines are generally safe, with side" + }, + { + "created_at": 1791165854274, + "role": "user", + "text": "Who proposed the theory of evolution by natural selection?" + }, + { + "created_at": 1791165975794, + "role": "assistant", + "text": "The theory of evolution by natural selection was primarily proposed by the English naturalist Charles Darwin, in collaboration with his cousin, Alfred Russel Wallace. Although Darwin developed the idea over several decades of observation, reading, and scientific inquiry\u2014particularly through his extensive work in the Gal\u00e1pagos Islands\u2014his pivotal contribution was published in 1859 in the book *On the Origin of Species*. In this work, Darwin presented a comprehensive and logically structured argument that species are not fixed but instead change over time through a process of natural selection, where individuals with advantageous traits are more likely to survive and reproduce, passing those traits to offspring [1].\n\nDarwin\u2019s theory was revolutionary because it offered a natural, non-theological explanation for the" + }, + { + "created_at": 1791166024130, + "role": "user", + "text": "What did he publish?" + }, + { + "created_at": 1791166145562, + "role": "assistant", + "text": "Charles Darwin\u2019s most influential work, *On the Origin of Species*, was first published in 1859. This landmark book systematically presented the theory of evolution by natural selection, arguing that species evolve over time through a process in which individuals with heritable traits better suited to their environment are more likely to survive and reproduce. Darwin built upon earlier ideas from scientists like Jean-Baptiste Lamarck and the concept of \"transformism,\" but he introduced a mechanism\u2014natural selection\u2014that was both observable and logically consistent with biological evidence. Unlike Lamarck\u2019s theory of inheritance of acquired characteristics, Darwin\u2019s framework relied on variation, inheritance, and differential survival, which could be tested and observed in nature.\n\nA central strength of *On the Origin of Species* was its use of empirical evidence from across the natural world. Darwin drew on observations of finch beak variation in the Gal\u00e1pagos Islands, the adaptation of species to different environments, and the geographical distribution of organisms. He showed that species, especially those in isolated regions, exhibit adaptations that match their ecological niches\u2014such" + }, + { + "created_at": 1791166200146, + "role": "user", + "text": "Compare the French Revolution and the Industrial Revolution." + }, + { + "created_at": 1791166321794, + "role": "assistant", + "text": "The French Revolution (1789\u20131799) and the Industrial Revolution (beginning c. 1760) were two of the most transformative events in modern history, but they differed fundamentally in nature, scope, and impact. The French Revolution was a political and social upheaval driven by demands for equality, liberty, and representation, rooted in the collapse of absolute monarchy and aristocratic privilege. It led to the abolition of feudalism, the rise of democratic ideals, and the establishment of a republic. In contrast, the Industrial Revolution was an economic and technological transformation that shifted human societies from agrarian, handcraft-based economies to industrialized, machine-driven systems. While both revolutions reshaped political and economic structures, the French Revolution was primarily ideological and institutional, whereas the Industrial Revolution was technological and economic in character" + }, + { + "created_at": 1791166371086, + "role": "user", + "text": "What is the capital of Australia?" + }, + { + "created_at": 1791166492785, + "role": "assistant", + "text": "The capital of Australia is Canberra. Unlike most countries, which designate a single city as their capital and often choose a major urban center, Australia selected Canberra as a purpose-built capital city to resolve a long-standing political dispute between the two largest cities\u2014Sydney and Melbourne\u2014each of which had strong claims to be the nation\u2019s capital. This compromise, reached in the 1910s, reflects a deliberate effort to create a neutral, centrally located capital that would not favor either the eastern or western parts of the country.\n\nCanberra was officially established in 1913 as a planned city, designed by architects and planners to be both visually appealing and functionally balanced. Its location, approximately 450 kilometers (280 miles" + } + ] +} \ No newline at end of file diff --git a/docs/evidence/2026-10-05-android-conversations/conversation/driver.py b/docs/evidence/2026-10-05-android-conversations/conversation/driver.py new file mode 100644 index 000000000..d8de9b508 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/conversation/driver.py @@ -0,0 +1,271 @@ +#!/usr/bin/env python3 +"""Before/after measurement driver for BOAR over adb (Pixel 6a). + + ANDROID_SERIAL= driver.py COMMAND + + driver.py cool wait until battery <= BENCH_MAX_C (33.0) C and thermal status 0 + driver.py chat bring the app back to the chat screen + driver.py convo LABEL run the 6-question conversation in the real chat UI, then pull + driver.py pull LABEL SINCE_MS export telemetry + answers created after SINCE_MS (phone clock) + driver.py now print the phone clock in ms + +Writes bench/LABEL.convo.json (timing of each send) and bench/LABEL.results.json (telemetry rows and chat +messages) next to this script. The conversation/*.json files in this folder are the results.json files, +renamed to LABEL.json. +""" +import json, os, re, sqlite3, subprocess, sys, time + +USAGE = __doc__.split("\n\n", 1)[1].rstrip() +NARGS = {"cool": 0, "chat": 0, "now": 0, "convo": 1, "pull": 2} +if "ANDROID_SERIAL" not in os.environ or len(sys.argv) < 2 or len(sys.argv) - 2 < NARGS.get(sys.argv[1], 99): + sys.exit(USAGE) +D = os.environ["ANDROID_SERIAL"] # adb serial of the phone +PKG = "team.sopa.aoair" +HERE = os.path.dirname(os.path.abspath(__file__)) +OUT = os.path.join(HERE, "bench") +os.makedirs(OUT, exist_ok=True) + +QUESTIONS = [ + "How do vaccines work?", + "What are the side effects?", + "Who proposed the theory of evolution by natural selection?", + "What did he publish?", + "Compare the French Revolution and the Industrial Revolution.", + "What is the capital of Australia?", +] +PAUSE_S = 30 + + +def adb(*args, check=True, binary=False): + r = subprocess.run(["adb", "-s", D, *args], capture_output=True, timeout=120) + if check and r.returncode != 0: + raise RuntimeError(f"adb {' '.join(args)}: {r.stderr.decode(errors='replace')}") + return r.stdout if binary else r.stdout.decode(errors="replace") + + +def log(msg): + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) + + +def phone_ms(): + return int(adb("shell", "date +%s%3N").strip()) + + +def thermal(): + temp = int(re.search(r"temperature: (\d+)", adb("shell", "dumpsys battery")).group(1)) / 10 + status = int(re.search(r"Thermal Status: (\d+)", adb("shell", "dumpsys thermalservice")).group(1)) + return temp, status + + +def cool(max_c=float(os.environ.get("BENCH_MAX_C", "33.0")), timeout_s=1800): + t0 = time.time() + while True: + temp, status = thermal() + log(f"battery {temp:.1f} C, thermal status {status}") + if temp <= max_c and status == 0: + return temp, status + if time.time() - t0 > timeout_s: + log("still warm after the timeout; continuing anyway") + return temp, status + time.sleep(30) + + +def wake(): + adb("shell", "input keyevent KEYCODE_WAKEUP", check=False) + + +def screen(): + adb("shell", "uiautomator dump /sdcard/bench-ui.xml", check=False) + xml = adb("exec-out", "cat /sdcard/bench-ui.xml", check=False) + nodes = [] + for n in re.findall(r"]*>", xml): + get = lambda k: (re.search(rf' {k}="([^"]*)"', n) or [None, ""])[1] + b = [int(v) for v in re.findall(r"\d+", get("bounds"))] + if len(b) == 4: + nodes.append({"cls": get("class"), "text": get("text"), "desc": get("content-desc"), "x": (b[0] + b[2]) // 2, "y": (b[1] + b[3]) // 2, "b": b}) + global TOOLS + t = next((n for n in nodes if n["desc"] == "Tools"), None) + if t: + TOOLS = t["b"] + return nodes + + +def near_tools(x, y, margin=100): + if not TOOLS: + return False + l, t, r, b = TOOLS + return l - margin <= x <= r + margin and t - margin <= y <= b + margin + + +TOOLS = None # bounds of the Expo dev "Tools" floating button, refreshed by screen() + + +def find(nodes, desc=None, cls=None): + for n in nodes: + if desc and n["desc"] == desc: + return n + if cls and cls in n["cls"]: + return n + return None + + +def tap(n): + if near_tools(n["x"], n["y"]): + raise RuntimeError(f"refusing to tap {n['desc'] or n['text']!r}: the Expo Tools button {TOOLS} is too close") + adb("shell", f"input tap {n['x']} {n['y']}") + + +def type_text(text): + # `input text`: spaces as %s; the whole argument single-quoted for the device shell. + adb("shell", "input text '" + text.replace(" ", "%s").replace("'", "") + "'") + + +def busy(nodes): + """Generating: no Send button (it turns into Stop while an answer streams).""" + return find(nodes, desc="Send message") is None + + +def wait_answer(timeout_s=300): + t0 = time.time() + # First the answer has to start (the send button turns into stop), then end. + started = False + while time.time() - t0 < 25: + wake() + if busy(screen()): + started = True + break + time.sleep(1) + calm = 0 + while time.time() - t0 < timeout_s: + wake() + if not busy(screen()): + calm += 1 + if calm >= 2: + return time.time() - t0, started + else: + calm = 0 + time.sleep(4) + return time.time() - t0, started + + +def to_chat(timeout_s=60): + """Leave the Evaluation screen (its Done button) and wait for the chat's New Chat button.""" + t0 = time.time() + while time.time() - t0 < timeout_s: + wake() + nodes = screen() + if find(nodes, desc="New Chat Session"): + log("on the chat screen") + return True + done = find(nodes, desc="Done") + if done: + # The Expo dev "Tools" button floats over most of Done: tap a point of Done outside it. + tools = find(nodes, desc="Tools") + x, y = done["x"], done["y"] + if tools: + l, t, r, btm = done["b"] + tl, tt, tr, tb = tools["b"] + if r - 4 > tr: + x, y = (tr + r) // 2, (t + btm) // 2 + elif tt - 4 > t: + x, y = (l + r) // 2, (t + tt) // 2 + if near_tools(x, y): + raise RuntimeError(f"Done is covered by the Expo Tools button {TOOLS}; move the button") + adb("shell", f"input tap {x} {y}") + log(f"closed the Evaluation screen (tap {x},{y})") + time.sleep(3) + raise RuntimeError("could not get back to the chat screen") + + +def convo(label): + adb("shell", f"monkey -p {PKG} -c android.intent.category.LAUNCHER 1", check=False) + time.sleep(3) + to_chat() + nodes = screen() + new_chat = find(nodes, desc="New Chat Session") + if not new_chat: + raise RuntimeError("New Chat button not found: is the chat screen open?") + tap(new_chat) + time.sleep(2) + since = phone_ms() + record = {"label": label, "since_ms": since, "questions": []} + for i, q in enumerate(QUESTIONS): + wake() + nodes = screen() + box, send = find(nodes, cls="EditText"), find(nodes, desc="Send message") + if not box or not send: + raise RuntimeError(f"input or send not found before question {i + 1}") + tap(box) + time.sleep(0.5) + type_text(q) + time.sleep(0.8) + # Tap Send once, then wait for the box to clear. On main the app clears it only after the + # background summary stops (the freeze the branch fixes); tapping again would send duplicates. + send = find(screen(), desc="Send message") or send + tapped = time.time() + tap(send) + while True: + wake() + box = find(screen(), cls="EditText") + if box and q not in box["text"]: + break + if time.time() - tapped > 300: + raise RuntimeError(f"question {i + 1} never left the input box") + time.sleep(2) + sent_at = time.time() + held_s = round(sent_at - tapped, 1) + log(f"{label} Q{i + 1} sent after {held_s}s in the box: {q}") + took, started = wait_answer() + log(f"{label} Q{i + 1} done after ~{took:.0f}s (started={started})") + record["questions"].append({"q": q, "held_in_box_s": held_s, "wall_s": round(took, 1), "started": started, "sent_at": sent_at}) + if i < len(QUESTIONS) - 1: + for _ in range(PAUSE_S // 10): + wake() + time.sleep(10) + record["temp_after"] = thermal() + path = os.path.join(OUT, f"{label}.convo.json") + with open(path, "w") as fh: + json.dump(record, fh, indent=2) + log(f"wrote {path}") + return since + + +def pull(label, since_ms): + dbdir = os.path.join(OUT, f"{label}.db") + os.makedirs(dbdir, exist_ok=True) + for f in ["aoair_knowledge.db", "aoair_knowledge.db-wal", "aoair_knowledge.db-shm"]: + data = adb("exec-out", f"run-as {PKG} cat files/SQLite/{f}", binary=True, check=False) + # The -wal/-shm files may not exist; the database itself must come back. + if f == "aoair_knowledge.db" and not data.startswith(b"SQLite format 3"): + sys.exit(f"could not pull files/SQLite/{f} from {PKG} (adb run-as failed?)") + with open(os.path.join(dbdir, f), "wb") as fh: + fh.write(data) + c = sqlite3.connect(os.path.join(dbdir, "aoair_knowledge.db")) + c.row_factory = sqlite3.Row + rows = [dict(r) for r in c.execute( + "SELECT created_at, model_id, task_type, ttft_ms, tok_per_sec, tokens_generated, generation_latency_ms, total_latency_ms, outcome " + "FROM execution_telemetry WHERE created_at >= ? ORDER BY created_at", (int(since_ms),))] + msgs = [dict(r) for r in c.execute( + "SELECT created_at, role, text FROM chat_messages WHERE created_at >= ? ORDER BY created_at", (int(since_ms),))] + out = {"label": label, "since_ms": int(since_ms), "telemetry": rows, "messages": msgs} + path = os.path.join(OUT, f"{label}.results.json") + with open(path, "w") as fh: + json.dump(out, fh, indent=2) + log(f"{label}: {len(rows)} telemetry rows, {len(msgs)} messages -> {path}") + for r in rows: + print(f" {r['model_id']:<30} ttft {r['ttft_ms'] or 0:>8.0f} ms tok/s {r['tok_per_sec'] or 0:>5.1f} tokens {r['tokens_generated'] or 0:>4} total {r['total_latency_ms'] or 0:>8.0f} ms {r['outcome']}") + + +if __name__ == "__main__": + cmd = sys.argv[1] + if cmd == "cool": + cool() + elif cmd == "chat": + to_chat() + elif cmd == "now": + print(phone_ms()) + elif cmd == "convo": + s = convo(sys.argv[2]) + pull(sys.argv[2], s) + elif cmd == "pull": + pull(sys.argv[2], int(sys.argv[3])) diff --git a/docs/evidence/2026-10-05-android-conversations/conversation/fix-15b.json b/docs/evidence/2026-10-05-android-conversations/conversation/fix-15b.json new file mode 100644 index 000000000..a5fa7024c --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/conversation/fix-15b.json @@ -0,0 +1,134 @@ +{ + "label": "fix-15b", + "since_ms": 1791233673154, + "telemetry": [ + { + "created_at": 1791233715623, + "model_id": "qwen2.5-1.5b-instruct-q4km", + "task_type": "research", + "ttft_ms": 2434.0511490106583, + "tok_per_sec": 15.754916968524805, + "tokens_generated": 512, + "generation_latency_ms": 32497.791072010994, + "total_latency_ms": 35422.74594801664, + "outcome": "success" + }, + { + "created_at": 1791233838365, + "model_id": "qwen2.5-1.5b-instruct-q4km", + "task_type": "lookup", + "ttft_ms": 4991.568035960197, + "tok_per_sec": 8.046489728177743, + "tokens_generated": 292, + "generation_latency_ms": 36289.116107046604, + "total_latency_ms": 42128.81764703989, + "outcome": "success" + }, + { + "created_at": 1791233955542, + "model_id": "qwen2.5-1.5b-instruct-q4km", + "task_type": "lookup", + "ttft_ms": 4346.926068007946, + "tok_per_sec": 13.659783869020076, + "tokens_generated": 512, + "generation_latency_ms": 37482.29144102335, + "total_latency_ms": 42475.43284100294, + "outcome": "success" + }, + { + "created_at": 1791234037770, + "model_id": "qwen2.5-1.5b-instruct-q4km", + "task_type": "lookup", + "ttft_ms": 1795.2140300273895, + "tok_per_sec": 14.064202533000605, + "tokens_generated": 463, + "generation_latency_ms": 32920.458796978, + "total_latency_ms": 35551.58473402262, + "outcome": "success" + }, + { + "created_at": 1791234132449, + "model_id": "qwen2.5-1.5b-instruct-q4km", + "task_type": "compare", + "ttft_ms": 8330.17013001442, + "tok_per_sec": 13.266973785750876, + "tokens_generated": 512, + "generation_latency_ms": 38592.071430027485, + "total_latency_ms": 47606.38804399967, + "outcome": "success" + }, + { + "created_at": 1791234201738, + "model_id": "qwen2.5-1.5b-instruct-q4km", + "task_type": "lookup", + "ttft_ms": 5676.43229496479, + "tok_per_sec": 14.919973618620363, + "tokens_generated": 187, + "generation_latency_ms": 12533.534226000309, + "total_latency_ms": 19094.62578302622, + "outcome": "success" + } + ], + "messages": [ + { + "created_at": 1791233680050, + "role": "user", + "text": "How do vaccines work?" + }, + { + "created_at": 1791233715625, + "role": "assistant", + "text": "Vaccines work by stimulating the immune system to recognize and protect against specific diseases. The process involves using weakened or weakened forms of the disease-causing microorganisms, known as antigens, to trigger an immune response. When the immune system encounters these antigens, it produces antibodies and memory cells that can quickly identify and neutralize the pathogen if it ever enters the body.\n\nKey points about vaccine effectiveness and safety include:\n\n1. **Effectiveness**: Vaccines are highly effective when administered correctly. They work by mimicking the pathogen to train the immune system to recognize and fight it. For example, the measles, mumps, and rubella (MMR) vaccine has an efficacy rate of approximately 97% in preventing these diseases.\n\n2. **Safety**: Vaccines are rigorously tested and approved by regulatory agencies before being administered to the public. Common side effects include mild symptoms like fever, swelling, or redness at the injection site. These are generally short-lived and minor. In rare cases, severe reactions can occur, but they are extremely rare. For instance, the MMR vaccine is very safe, with a reported 0.5 to 1 case of autism in every 100,000 children vaccinated.\n\n3. **Comparisons**: The effectiveness of vaccines compared to natural infection can be significant. Natural infections can sometimes lead to severe or life-threatening complications. For example, the diphtheria toxin, which is part of the DTP vaccine, is derived from a killed form of the Corynebacterium diphtheriae bacteria, which is a harmless organism. This toxin is used to prevent diphtheria, a severe bacterial infection.\n\n4. **Long-term benefits**: Vaccines have been instrumental in eradicating diseases that once were common and deadly. For example, the global eradication of smallpox is a testament to the power of vaccines. It is a disease that can be fatal, but it is now preventable through the smallpox vaccine.\n\n5. **Future outlook**: Despite their safety and efficacy, vaccines remain vital tools in public health. They are crucial in the fight against emerging diseases and in protecting against the resurgence of diseases that were once eradicated. For instance, the global response to the COVID-19 pandemic has been significantly aided by the development and distribution of vaccines, which have been shown to be highly effective at reducing the severity and spread of the virus.\n\nIn summary, vaccines work by using harmless or weakened antigens to stimulate the immune system" + }, + { + "created_at": 1791233796218, + "role": "user", + "text": "What are the side effects?" + }, + { + "created_at": 1791233838376, + "role": "assistant", + "text": "Vaccines have a long history of use and have been proven to be safe and effective in preventing diseases. However, it is important to understand that, like any medication, vaccines can have side effects, although they are generally mild and temporary. [2] [3]\n\nOne of the main side effects of vaccines is pain and swelling at the injection site. This is because the immune system is being stimulated to react to the vaccine, and the body responds by sending white blood cells to the site of injection, which can cause some discomfort. [1]\n\nAnother potential side effect is fever, which is also a result of the immune system's response to the vaccine. This is typically mild and lasts for a short period of time. [1]\n\nOther side effects of vaccines can include headache, fatigue, and general feelings of discomfort. [1]\n\nIt is important to note that the side effects of vaccines are usually mild and temporary, and most people do not experience any side effects at all. [2]\n\nVaccines are considered safe because they have been extensively tested and monitored. [2]\n\n[1] Vaccine, Wikipedia, (accessed 2021-02-23)\n\n[2] CDC, Vaccine Safety, (accessed 2021-02-23)\n\n[3] NHS, What are the side effects of vaccines? (accessed 2021-02-23)" + }, + { + "created_at": 1791233913047, + "role": "user", + "text": "Who proposed the theory of evolution by natural selection?" + }, + { + "created_at": 1791233955543, + "role": "assistant", + "text": "The theory of evolution by natural selection was proposed by Charles Darwin in his landmark book \"On the Origin of Species\" published in 1859. Darwin's theory explains how species evolve over time through the process of natural selection, in which organisms with advantageous traits are more likely to survive and reproduce, passing on those traits to their offspring. This leads to the gradual accumulation of beneficial traits over successive generations, resulting in the development of new species. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] [35] [36] [37] [38] [39] [40] [41] [42] [43] [44] [45] [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] [57] [58] [59] [60] [61] [62] [63] [64] [65] [66] [67] [68] [69] [70] [71] [72] [73] [74] [75] [76] [77] [78] [79] [80] [81] [82] [83] [84] [85] [86] [87] [88] [89] [90] [91] [92] [93] [94] [95] [96] [97] [98] [99] [100] [101] [102] [103] [104] [105] [106]" + }, + { + "created_at": 1791234002203, + "role": "user", + "text": "What did he publish?" + }, + { + "created_at": 1791234037771, + "role": "assistant", + "text": "Charles Darwin, an English naturalist and geologist, published his theory of evolution by natural selection in \"On the Origin of Species\" in 1859. The book not only introduced the theory of evolution but also introduced the concept of natural selection as the mechanism driving evolution. Darwin's work built upon the earlier theories of blending inheritance and the gradual accumulation of small variations over generations. [1]\n\nDarwin's theory was met with skepticism and controversy at the time, with many dismissing it as a flawed concept. However, the evidence from the fossil record and the diversity of life, as Darwin saw it, was compelling. His theory was bolstered by the work of Alfred Russel Wallace, who independently developed a similar theory based on observations of the natural world. [2]\n\nThe theory of natural selection is a key element in the evolutionary process, where individuals with advantageous traits are more likely to survive and reproduce, passing these traits on to their offspring. This leads to the gradual change in a population over generations, resulting in the diversity of life we see today. [3]\n\nDarwin's work revolutionized the field of biology and paved the way for future research in genetics, ecology, and conservation biology. His theory continues to be studied and refined, with ongoing research into the mechanisms of evolution and the role of natural selection in shaping the diversity of life on Earth. [4]\n\nIn conclusion, the publication of \"On the Origin of Species\" in 1859 was a seminal event in the history of evolutionary thought. It introduced the concept of natural selection as a mechanism driving evolution, which has been the cornerstone of scientific understanding of life's diversity and adaptability. [5] [6]\n\n[1] Darwin, C. (1859). On the Origin of Species.\n[2] Wallace, A. H. (1858). On the tendency of species to maintain their numbers.\n[3] Darwin, C. (1859). On the Origin of Species.\n[4] Darwin, C. (1859). On the Origin of Species.\n[5] Darwin, C. (1859). On the Origin of Species.\n[6] Darwin, C. (1859). On the Origin of Species." + }, + { + "created_at": 1791234084829, + "role": "user", + "text": "Compare the French Revolution and the Industrial Revolution." + }, + { + "created_at": 1791234132449, + "role": "assistant", + "text": "The French Revolution (1789\u20131799) and the Industrial Revolution (18th\u201319th centuries) were two pivotal historical events that had profound impacts on societies around the world. While the French Revolution was a transformative social and political upheaval, the Industrial Revolution was a technological and economic revolution that transformed production methods and led to significant improvements in living standards.\n\n### Social and Political Impact\n\nThe French Revolution was characterized by widespread social upheaval, revolution, and change. It led to the establishment of a republic, the end of the monarchy, and the beginning of the French Republic, marking the end of absolute monarchy. The revolution was also marked by the Reign of Terror, which saw the execution of thousands of perceived enemies of the Revolution. The Revolution's legacy includes the adoption of the French motto \"Libert\u00e9, \u00c9galit\u00e9, Fraternit\u00e9\" (Liberty, Equality, Fraternity), as well as the establishment of the First French Empire under Napoleon Bonaparte.\n\nThe Industrial Revolution, on the other hand, was a period of significant economic and social transformation. It began with the invention of the power loom in the 1730s, but it was the development of steam power and the invention of the steam engine that fundamentally changed the nature of production. The Industrial Revolution led to the rise of factory systems, urbanization, and the growth of the middle class. It also led to the creation of new industries, including textiles, mining, and steel manufacturing, which were central to the economic development of Europe. The Industrial Revolution was characterized by the growth of capitalist economies, the rise of the working class, and the development of new technologies and methods of production.\n\n### Economic Impact\n\nThe French Revolution had a profound impact on economic practices. The revolution led to the end of the feudal system, the redistribution of land, and the establishment of a new economic order. The revolution also had a significant impact on international trade, as France sought to gain a larger share of the global market. The revolution led to the decline of the monarchy and the rise of the Republic, which was characterized by its reliance on trade and industry to sustain its economy.\n\nThe Industrial Revolution, on the other hand, was a period of economic growth and development. The revolution led to the rapid expansion of industry, with the rise of factories and the growth of the urban population. The revolution also led to the development of new technologies, such as the steam engine, which transformed the nature of production. The revolution also led to the growth" + }, + { + "created_at": 1791234182628, + "role": "user", + "text": "What is the capital of Australia?" + }, + { + "created_at": 1791234201739, + "role": "assistant", + "text": "Australia, officially known as the Commonwealth of Australia, is a country located in the south of the Australian continent. Canberra is the capital and largest city of Australia. Canberra is located in the southeastern part of the continent, approximately in the center of the Australian Capital Territory. The city is built around a large green area and a large park and is surrounded by highlands and mountains. Canberra is the center of government and administrative work, and is also the national capital of Australia. It is located in the southeast of the continent, in the center of the Australian Capital Territory, on a hilltop. It is surrounded by highlands and mountains. It is located in the center of the continent. Canberra is the capital of Australia. Canberra is located in the southeast of the continent, in the center of the Australian Capital Territory, on a hilltop. It is surrounded by highlands and mountains. It is located in the center of the continent." + } + ] +} \ No newline at end of file diff --git a/docs/evidence/2026-10-05-android-conversations/conversation/fix-4b.json b/docs/evidence/2026-10-05-android-conversations/conversation/fix-4b.json new file mode 100644 index 000000000..1d4469087 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/conversation/fix-4b.json @@ -0,0 +1,134 @@ +{ + "label": "fix-4b", + "since_ms": 1791171986161, + "telemetry": [ + { + "created_at": 1791172063520, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "research", + "ttft_ms": 8521.85889095068, + "tok_per_sec": 4.859072998787893, + "tokens_generated": 285, + "generation_latency_ms": 58653.16287100315, + "total_latency_ms": 67734.7239099741, + "outcome": "success" + }, + { + "created_at": 1791172217789, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 9292.342858970165, + "tok_per_sec": 4.622324809101982, + "tokens_generated": 307, + "generation_latency_ms": 66416.7951580286, + "total_latency_ms": 76583.46496200562, + "outcome": "success" + }, + { + "created_at": 1791172344602, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 11021.531580984592, + "tok_per_sec": 3.9460855419668714, + "tokens_generated": 259, + "generation_latency_ms": 65634.66433900595, + "total_latency_ms": 77429.65404999256, + "outcome": "success" + }, + { + "created_at": 1791172517906, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 19781.24223703146, + "tok_per_sec": 2.5111464378010435, + "tokens_generated": 253, + "generation_latency_ms": 100750.79501199722, + "total_latency_ms": 121555.57517296076, + "outcome": "success" + }, + { + "created_at": 1791172694294, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "compare", + "ttft_ms": 31556.963476002216, + "tok_per_sec": 2.6705835598015057, + "tokens_generated": 239, + "generation_latency_ms": 89493.54874998331, + "total_latency_ms": 122076.84938400984, + "outcome": "success" + }, + { + "created_at": 1791172855503, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 19603.181976020336, + "tok_per_sec": 3.4531642734067285, + "tokens_generated": 301, + "generation_latency_ms": 87166.4294450283, + "total_latency_ms": 107854.29648000002, + "outcome": "success" + } + ], + "messages": [ + { + "created_at": 1791171995663, + "role": "user", + "text": "How do vaccines work?" + }, + { + "created_at": 1791172063528, + "role": "assistant", + "text": "Vaccines work by training the immune system to recognize and fight specific pathogens\u2014like viruses or bacteria\u2014without causing the full disease. They introduce a harmless version of the pathogen, such as a weakened or inactivated form, a piece of its surface protein (like the spike protein of SARS-CoV-2), or its toxin [1]. This triggers the body\u2019s immune system to produce antibodies and memory cells, which \"remember\" the pathogen.\n\nUnlike natural infection, vaccines don\u2019t cause illness. For example, the measles vaccine uses a weakened virus that replicates only slightly, triggering a strong immune response without causing measles. In contrast, natural infection carries risks like severe complications or death, especially in vulnerable populations.\n\nThe immune system\u2019s response is highly specific and long-lasting. After exposure, memory B cells and T cells remain, enabling a faster, stronger reaction if the real pathogen later enters. This is why people who are vaccinated can often recover quickly from diseases like influenza or polio.\n\nCompared to natural infection, vaccines are safer, more consistent, and can be administered widely. For instance, the HPV vaccine prevents cervical cancer by targeting the virus that causes it, reducing incidence by up to 90% in vaccinated populations.\n\nUltimately, vaccines are a powerful tool for population-level protection, preventing outbreaks and enabling herd immunity through widespread immunity. Their effectiveness and safety are backed by decades of clinical research." + }, + { + "created_at": 1791172141185, + "role": "user", + "text": "What are the side effects?" + }, + { + "created_at": 1791172217799, + "role": "assistant", + "text": "Vaccine side effects are typically mild and temporary, reflecting the immune system\u2019s active response. Common reactions\u2014like soreness at the injection site, low-grade fever, fatigue, or headache\u2014occur because the immune system is being stimulated. These symptoms usually resolve within 1\u20133 days and are more common after doses that introduce a weakened or inactivated pathogen (e.g., flu or MMR vaccines), where the immune response is robust but brief.\n\nCompared to natural infection, vaccine side effects are significantly milder and less likely to cause long-term harm. Natural infection often leads to severe symptoms, complications, or even death\u2014especially in vulnerable populations\u2014while vaccines mimic this response safely and predictably. For instance, the measles vaccine might cause a low fever, whereas actual measles infection leads to high fever, rash, and in rare cases, encephalitis.\n\nRare side effects\u2014such as allergic reactions or autoimmune phenomena\u2014are extremely uncommon. For example, the rare case of Guillain-Barr\u00e9 syndrome linked to the flu shot is far less frequent than the risk of developing the illness itself. Most reported adverse events are due to individual sensitivities, not the vaccine's mechanism.\n\nThe vast majority of side effects are temporary and outweighed by the protection vaccines provide against serious disease. Evidence from large-scale studies (e.g., CDC, WHO) shows that the benefits of vaccination far exceed risks.\n\nIn summary, while side effects exist, they are generally minor, predictable, and vastly less dangerous than the diseases they prevent." + }, + { + "created_at": 1791172267154, + "role": "user", + "text": "Who proposed the theory of evolution by natural selection?" + }, + { + "created_at": 1791172344605, + "role": "assistant", + "text": "The theory of evolution by natural selection was proposed by Charles Darwin, published in his 1859 work *On the Origin of Species*. Darwin observed patterns in nature, such as the variation among individuals in a species and the survival of those best adapted to their environment\u2014like finches with larger beaks surviving droughts. This idea was built upon earlier work, notably by Alfred Russel Wallace, who independently developed a similar theory and shared it with Darwin. Darwin\u2019s contribution lay in systematically presenting evidence from biogeography, anatomy, and embryology to support natural selection as the mechanism of evolution.\n\nUnlike Lamarckism, which proposed that organisms acquire traits during life and pass them to offspring, Darwin\u2019s theory emphasized that traits are inherited through variation and differential survival. Natural selection acts on random genetic variations, favoring those that enhance survival and reproduction.\n\nDarwin\u2019s theory revolutionized biology by providing a unifying explanation for biodiversity. It explains how species adapt over time, such as the evolution of antibiotic resistance in bacteria through selection pressure.\n\nThough Darwin didn\u2019t have modern genetics, his theory laid the groundwork for the modern synthesis, combining genetics with evolution. His work remains foundational, with evidence from fossils, molecular biology, and observed phenomena like industrial melanism in peppered moths [1]." + }, + { + "created_at": 1791172396332, + "role": "user", + "text": "What did he publish?" + }, + { + "created_at": 1791172517925, + "role": "assistant", + "text": "Charles Darwin published *On the Origin of Species* in 1859, which introduced the theory of evolution by natural selection as a comprehensive explanation for the diversity of life. In this work, he argued that species evolve over time through gradual changes, with individuals possessing advantageous traits more likely to survive and reproduce\u2014passing those traits to offspring. This mechanism, natural selection, acts as a filter, shaping populations over generations.\n\nDarwin supported his theory with extensive evidence from biology, including observations of finches in the Gal\u00e1pagos, the variation in plant and animal forms across regions, and the fossil record. He compared species to show how similarities in structure and function suggest common ancestry\u2014like the homologous limbs of mammals, birds, and reptiles.\n\nHe also addressed criticism by emphasizing that evolution is not goal-oriented or designed, but a process driven by environmental pressures. This contrasted with prevailing views of divine creation and challenged the idea that species were fixed and unchanging.\n\nWhile Alfred Russel Wallace independently developed a similar theory, Darwin's publication was more detailed and widely influential. His work laid the foundation for modern biology, shaping fields like genetics, ecology, and systematics.\n\nUltimately, *On the Origin of Species* revolutionized scientific thought by" + }, + { + "created_at": 1791172571927, + "role": "user", + "text": "Compare the French Revolution and the Industrial Revolution." + }, + { + "created_at": 1791172694304, + "role": "assistant", + "text": "The French Revolution (1789\u20131799) was a political and social upheaval driven by demands for equality, liberty, and fraternity, fueled by economic hardship and corruption in the monarchy. It led to the collapse of absolute monarchy, the rise of democratic ideals, and the establishment of a republic\u2014values still central to modern French politics [1]. In contrast, the Industrial Revolution (beginning c. 1760) was a transformative economic and technological shift, marked by mechanization, urbanization, and the rise of factories in Britain and later Europe and the U.S. [2]. While the French Revolution changed political structures, the Industrial Revolution transformed economic systems and daily life.\n\nThe French Revolution was rooted in social inequality and political discontent, resulting in radical changes to governance, law, and society. The Industrial Revolution, however, was driven by innovation\u2014such as steam power and the cotton mill\u2014leading to mass production and a shift from agrarian to industrial economies.\n\nWhile both revolutions were pivotal in shaping modern history, the French Revolution was a political revolution affecting institutions and ideologies, whereas the Industrial Revolution was an economic and technological transformation affecting labor, class structures," + }, + { + "created_at": 1791172747619, + "role": "user", + "text": "What is the capital of Australia?" + }, + { + "created_at": 1791172855520, + "role": "assistant", + "text": "The capital of Australia is Canberra. Unlike most countries, Australia does not have a capital city located at a natural geographic center; instead, it was deliberately chosen as a compromise between Sydney and Melbourne, the two largest cities. This decision emerged during the 19th century when the British colonial government sought to establish a neutral, central capital to avoid favoring either the eastern or southern economic centers.\n\nCanberra was officially founded in 1913 and designed by architects Walter Burley Griffin and Marion Mahony Griffin. Its layout is based on the principles of symmetry and balance, with the city divided into two halves\u2014north and south\u2014mirroring the Australian landscape. This unique design reflects both a political intent to foster national unity and a vision of harmonious urban planning.\n\nUnlike capitals such as London or Washington, D.C., Canberra was not built in a pre-existing city. This made it a symbol of Australia\u2019s modern, independent identity. It houses the Parliament House, the High Court, and other key government institutions, serving as the center of national governance.\n\nWhile Sydney and Melbourne remain Australia\u2019s largest and most vibrant cities, Canberra's role is primarily political and administrative. Its population is small (around 400,000), and it lacks the economic or cultural scale of the coastal cities, yet it remains vital to national cohesion.\n\nIn essence, Canberra stands as a deliberate, symbolic, and functional capital\u2014unlike traditional capitals\u2014reflecting Australia\u2019s unique political and geographical development." + } + ] +} \ No newline at end of file diff --git a/docs/evidence/2026-10-05-android-conversations/conversation/fix2-4b.json b/docs/evidence/2026-10-05-android-conversations/conversation/fix2-4b.json new file mode 100644 index 000000000..e1d8fb2e2 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/conversation/fix2-4b.json @@ -0,0 +1,134 @@ +{ + "label": "fix2-4b", + "since_ms": 1791217767738, + "telemetry": [ + { + "created_at": 1791217855521, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "research", + "ttft_ms": 8982.984094977379, + "tok_per_sec": 4.261186768591217, + "tokens_generated": 292, + "generation_latency_ms": 68525.51081597805, + "total_latency_ms": 78174.46886897087, + "outcome": "success" + }, + { + "created_at": 1791218012016, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 11281.54126495123, + "tok_per_sec": 3.865775175385102, + "tokens_generated": 259, + "generation_latency_ms": 66998.20559900999, + "total_latency_ms": 79002.72908300161, + "outcome": "success" + }, + { + "created_at": 1791218186707, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 15564.970752000809, + "tok_per_sec": 0.11727927670380106, + "tokens_generated": 13, + "generation_latency_ms": 110846.52263700962, + "total_latency_ms": 127208.39158201218, + "outcome": "success" + }, + { + "created_at": 1791218303614, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 6946.45072799921, + "tok_per_sec": 4.0297522779111485, + "tokens_generated": 220, + "generation_latency_ms": 54593.92658102512, + "total_latency_ms": 62533.03974401951, + "outcome": "success" + }, + { + "created_at": 1791218455190, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "compare", + "ttft_ms": 23000.740937948227, + "tok_per_sec": 3.2244981563636492, + "tokens_generated": 248, + "generation_latency_ms": 76911.19299000502, + "total_latency_ms": 100947.80009794235, + "outcome": "success" + }, + { + "created_at": 1791218572436, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 19708.721853017807, + "tok_per_sec": 3.31113053754737, + "tokens_generated": 155, + "generation_latency_ms": 46811.805889964104, + "total_latency_ms": 67653.14057701826, + "outcome": "success" + } + ], + "messages": [ + { + "created_at": 1791217777225, + "role": "user", + "text": "How do vaccines work?" + }, + { + "created_at": 1791217855531, + "role": "assistant", + "text": "Vaccines work by training the immune system to recognize and fight specific pathogens\u2014like viruses or bacteria\u2014without causing the full disease. They introduce a harmless version of the pathogen (such as a weakened or inactivated form, or a piece of it like a surface protein) so the body can learn to respond to it later [1]. This process triggers the production of antibodies and memory cells that remain in the body, allowing a faster and stronger response if the real pathogen is encountered.\n\nUnlike natural infections, vaccines do not cause illness. Instead, they mimic infection safely, enabling the immune system to build defenses without exposure to the full disease. For example, the measles vaccine uses a live-attenuated virus that replicates weakly, prompting immunity without causing measles.\n\nCompared to natural infection, vaccines are safer and more predictable. Natural infection can lead to severe illness, complications, or death, while vaccines offer consistent protection with minimal side effects. The HPV vaccine, for instance, targets specific virus proteins, teaching the immune system to destroy the virus before it causes cancer.\n\nThis \"training\" results in long-term immunity. After exposure, memory B and T cells quickly activate, reducing the risk of infection or minimizing disease severity. The effectiveness of vaccines is proven through extensive clinical trials and real-world data, showing they prevent millions of infections annually.\n\nIn short, vaccines are a safe, strategic way to build immunity through controlled exposure to pathogen components." + }, + { + "created_at": 1791217932995, + "role": "user", + "text": "What are the side effects?" + }, + { + "created_at": 1791218012026, + "role": "assistant", + "text": "Vaccines typically cause mild, temporary side effects such as soreness at the injection site, low-grade fever, fatigue, or headache. These are the body\u2019s natural immune response to the vaccine\u2019s components, signaling that the immune system is active [1]. For example, a fever after a flu shot reflects the immune system mobilizing defenses, much like it would after a real infection\u2014but without illness.\n\nMore serious side effects are extremely rare and well-documented in large-scale studies. For instance, the risk of a severe allergic reaction (anaphylaxis) is less than 1 in 1 million doses, and such cases are usually managed promptly. In comparison, the risk of severe complications from the actual disease\u2014like influenza or measles\u2014is far greater.\n\nSome people may experience behavioral or psychological effects, such as irritability, which are generally transient and not linked to long-term harm. These are often misattributed but are typically coincidental.\n\nVaccine side effects are typically less severe and less common than the risks posed by the diseases themselves. For instance, measles can lead to pneumonia or encephalitis, while the risk of a vaccine-related event is negligible in comparison.\n\nOverall, the benefits of vaccination in preventing disease and saving lives far outweigh any rare or mild side effects." + }, + { + "created_at": 1791218059471, + "role": "user", + "text": "Who proposed the theory of evolution by natural selection?" + }, + { + "created_at": 1791218186726, + "role": "assistant", + "text": "The theory of evolution by natural selection was primarily proposed by Charles Darwin" + }, + { + "created_at": 1791218241051, + "role": "user", + "text": "What did he publish?" + }, + { + "created_at": 1791218303622, + "role": "assistant", + "text": "Charles Darwin published *On the Origin of Species* in 1859, which presented the scientific foundation of evolution by natural selection. In this work, he argued that species evolve over time through the differential survival and reproduction of individuals with advantageous traits\u2014those better adapted to their environment are more likely to pass on their genes. This mechanism, natural selection, explains how small, inheritable variations accumulate over generations, leading to new species [1].\n\nDarwin contrasted this with the prevailing belief in divine design and gradualism, offering instead a natural, observable process. He used evidence from biogeography, embryology, and fossil records\u2014such as the similarities between finches on the Gal\u00e1pagos Islands\u2014to support his claims.\n\nThe book also introduced the concept of common descent, showing that all species share a common ancestor. This idea revolutionized biology by unifying diverse fields under a single explanatory framework.\n\nWhile Alfred Russel Wallace independently developed a similar theory, Darwin\u2019s work was the first comprehensive, evidence-based presentation. *On the Origin of Species* remains a cornerstone of modern biology." + }, + { + "created_at": 1791218354225, + "role": "user", + "text": "Compare the French Revolution and the Industrial Revolution." + }, + { + "created_at": 1791218455197, + "role": "assistant", + "text": "The French Revolution (1789\u20131799) was a political and social upheaval that dismantled the monarchy, aristocracy, and feudal structures, replacing them with ideals of liberty, equality, and fraternity [1]. It led to radical changes in governance, legal rights, and societal roles, fundamentally reshaping France\u2019s political identity. In contrast, the Industrial Revolution (beginning c. 1760) was an economic and technological transformation marked by mechanization, urbanization, and the rise of factories and mass production [2]. While both revolutions were transformative, one was ideological and political, the other economic and technological.\n\nThe French Revolution targeted institutions and power structures, resulting in violence, political instability, and the rise of radical factions. The Industrial Revolution, meanwhile, altered daily life through new technologies, changing labor patterns, and reshaping cities and economies. Unlike France's upheaval, industrial change was gradual and often driven by innovation rather than political rebellion.\n\nThough both revolutions had long-lasting impacts, the French Revolution influenced modern democratic governance, while the Industrial Revolution redefined global economic systems and living standards. Neither was a single event but a complex process with far-reaching consequences across society, politics, and technology." + }, + { + "created_at": 1791218504744, + "role": "user", + "text": "What is the capital of Australia?" + }, + { + "created_at": 1791218572445, + "role": "assistant", + "text": "The capital of Australia is Canberra. Unlike most countries, Australia does not have a capital city located at a geographical center or with a historical claim to prominence. Canberra was deliberately chosen as a compromise between Sydney and Melbourne, the two largest cities, to avoid favoring either economically or politically. \n\nBuilt in the 1910s, Canberra was designed as a planned city, with symmetry and modern architecture, reflecting a desire for neutrality and civic balance. \n\nIt is not the largest or most populous city, but its status as capital ensures administrative and governmental functions are spread equitably. \n\nIn contrast to Sydney or Melbourne\u2014major hubs for economy and culture\u2014Canberra prioritizes political neutrality. \n\nThis design reflects Australia\u2019s commitment to democratic balance and regional fairness in governance." + } + ] +} \ No newline at end of file diff --git a/docs/evidence/2026-10-05-android-conversations/conversation/narrow-4b.json b/docs/evidence/2026-10-05-android-conversations/conversation/narrow-4b.json new file mode 100644 index 000000000..9cacf1d9e --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/conversation/narrow-4b.json @@ -0,0 +1,134 @@ +{ + "label": "narrow-4b", + "since_ms": 1791251855002, + "telemetry": [ + { + "created_at": 1791251937662, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "research", + "ttft_ms": 8843.185672998428, + "tok_per_sec": 4.707114637083444, + "tokens_generated": 304, + "generation_latency_ms": 64583.088247954845, + "total_latency_ms": 74063.62202501297, + "outcome": "success" + }, + { + "created_at": 1791252084490, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 7446.732262969017, + "tok_per_sec": 4.833786209304961, + "tokens_generated": 293, + "generation_latency_ms": 60615.0101210475, + "total_latency_ms": 68786.9829840064, + "outcome": "success" + }, + { + "created_at": 1791252233683, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 9325.448083996773, + "tok_per_sec": 4.358414232590286, + "tokens_generated": 290, + "generation_latency_ms": 66537.96186500788, + "total_latency_ms": 76456.5895550251, + "outcome": "success" + }, + { + "created_at": 1791252350571, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 4157.569419980049, + "tok_per_sec": 4.123795635218978, + "tokens_generated": 252, + "generation_latency_ms": 61108.75084298849, + "total_latency_ms": 66091.21393996477, + "outcome": "success" + }, + { + "created_at": 1791252484099, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "compare", + "ttft_ms": 20098.92013502121, + "tok_per_sec": 3.6535803983934443, + "tokens_generated": 234, + "generation_latency_ms": 64046.76358097792, + "total_latency_ms": 85148.35107499361, + "outcome": "success" + }, + { + "created_at": 1791252594026, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 14359.996385991573, + "tok_per_sec": 4.424730917652381, + "tokens_generated": 207, + "generation_latency_ms": 46782.50584101677, + "total_latency_ms": 61983.29287695885, + "outcome": "success" + } + ], + "messages": [ + { + "created_at": 1791251863471, + "role": "user", + "text": "How do vaccines work?" + }, + { + "created_at": 1791251937669, + "role": "assistant", + "text": "Vaccines work by training the immune system to recognize and fight specific pathogens\u2014like viruses or bacteria\u2014without causing the full disease. They introduce a harmless version of the pathogen, such as a weakened or inactivated form, or a piece of it (like a surface protein), so the body can detect it and build a defense response [1]. This response creates memory cells that remain active, allowing the immune system to respond rapidly and effectively if the real pathogen later enters the body.\n\nUnlike natural infection, which can cause illness and long-term health risks, vaccines stimulate immunity with minimal risk. For example, the measles vaccine uses a weakened virus that triggers immunity without causing measles. In contrast, natural infection exposes the body to the full disease, potentially leading to severe complications or death.\n\nVaccines also offer population-level protection through herd immunity. When a high percentage of people are vaccinated, the spread of disease is blocked, protecting those who cannot be vaccinated\u2014such as infants or immunocompromised individuals. This is especially critical for highly contagious diseases like polio or influenza.\n\nComparatively, vaccines are more reliable and safer than relying on natural exposure. Unlike the unpredictable outcome of natural infection, vaccines provide consistent, measurable protection. For instance, the HPV vaccine prevents cervical cancer by targeting specific viruses, reducing disease incidence by over 90% in vaccinated populations.\n\nIn summary, vaccines are a proven method of immune education, offering both individual and community-level protection through safe, targeted stimulation of the body\u2019s defenses." + }, + { + "created_at": 1791252015601, + "role": "user", + "text": "What are the side effects?" + }, + { + "created_at": 1791252084498, + "role": "assistant", + "text": "Vaccines are generally safe, with side effects typically mild and short-lived. Common side effects include soreness at the injection site, low-grade fever, fatigue, or irritability\u2014similar to minor illness symptoms. These occur because the immune system is actively responding to the vaccine's components, which mimics a real infection but without causing disease [1]. For example, the measles, mumps, and rubella (MMR) vaccine may cause a fever or rash in about 1 in 100 children, but these are usually brief and self-limiting.\n\nSerious side effects are extremely rare. The misconception that vaccines cause autism, for instance, originated from a discredited study and has been thoroughly disproven by large-scale epidemiological research [1]. In contrast, untreated diseases like measles or polio carry far greater risks of severe complications, hospitalization, or death.\n\nSome people may experience allergic reactions, especially to components like egg protein (in some influenza vaccines), but these are uncommon and managed with medical supervision. For instance, individuals with a history of severe allergies should consult a healthcare provider before vaccination.\n\nCompared to the risks of natural infection\u2014such as pneumonia, encephalitis, or long-term disability\u2014vaccine side effects are vastly less dangerous. The benefits of immunity far outweigh the risks.\n\nIn summary, vaccine side effects are typically mild, temporary, and well-managed, while the risks of actual disease remain significantly higher." + }, + { + "created_at": 1791252157211, + "role": "user", + "text": "Who proposed the theory of evolution by natural selection?" + }, + { + "created_at": 1791252233684, + "role": "assistant", + "text": "The theory of evolution by natural selection was proposed by Charles Darwin, building on earlier ideas from scientists like Jean-Baptiste Lamarck and Alfred Russel Wallace. Darwin's key insight was that organisms with traits better suited to their environment are more likely to survive and reproduce, passing those advantageous traits to offspring over generations [1]. This process, natural selection, explains how species adapt and diversify over time.\n\nUnlike Lamarck\u2019s idea that organisms develop traits in response to environmental needs (e.g., giraffes stretching their necks), Darwin\u2019s theory is supported by observable evidence such as the fossil record, biogeography, and genetic variation. For example, the gradual shift in beak sizes among finches in the Gal\u00e1pagos Islands during droughts demonstrates natural selection in action.\n\nDarwin\u2019s theory was revolutionary because it provided a natural, non-supernatural explanation for biodiversity. It challenged the then-dominant belief in divine design and offered a mechanism\u2014selection pressure\u2014by which species change over time. This mechanism is now central to biology, explaining everything from antibiotic resistance in bacteria to the diversity of life.\n\nAlthough Wallace independently formulated a similar theory, Darwin was the first to publish it systematically (1859, *On the Origin of Species*). His work laid the foundation for modern evolutionary biology, influencing fields from medicine to ecology. Evolution is not just a theory\u2014it is a well-supported scientific principle backed by extensive evidence." + }, + { + "created_at": 1791252284346, + "role": "user", + "text": "What did he publish?" + }, + { + "created_at": 1791252350572, + "role": "assistant", + "text": "Charles Darwin published *On the Origin of Species* in 1859, which laid out his theory of evolution by natural selection in detail. This work presented a comprehensive argument that species change over time through the differential survival and reproduction of individuals with advantageous traits. Darwin used evidence from biogeography, fossil records, and observations of species variation in nature\u2014such as the differences in finch beaks on the Gal\u00e1pagos Islands\u2014to support his claims [1].\n\nHe contrasted natural selection with Lamarckism, which suggested traits were passed on based on use or disuse. Darwin\u2019s model emphasized that traits are inherited through genetic variation, and those better suited to the environment are more likely to survive and reproduce.\n\nDarwin did not provide a mechanism for inheritance at the molecular level (that came later with genetics), but his framework was revolutionary in explaining biodiversity. His ideas were initially met with resistance, partly due to their challenge to religious and scientific views of creation.\n\nThe book was foundational in biology, influencing fields from ecology to medicine. It showed that evolution is not random but driven by environmental pressures acting on heritable variation.\n\nIn essence, *On the Origin of Species* unified biological knowledge and established natural selection as the central mechanism of evolutionary change." + }, + { + "created_at": 1791252398936, + "role": "user", + "text": "Compare the French Revolution and the Industrial Revolution." + }, + { + "created_at": 1791252484100, + "role": "assistant", + "text": "The French Revolution (1789\u20131799) was a political and social upheaval that dismantled feudalism and established principles of liberty, equality, and democracy in France. It was driven by economic hardship, social inequality, and political discontent, culminating in the overthrow of the monarchy and the rise of radical republicanism [1]. In contrast, the Industrial Revolution (beginning c. 1760) was a transformative economic shift, marked by the mechanization of production, urbanization, and the rise of factories and steam power, primarily in Britain [2].\n\nWhile the French Revolution reshaped governance and societal structures, the Industrial Revolution transformed economies and daily life, altering labor patterns, class structures, and global trade. The former emphasized political ideology and societal reorganization; the latter focused on technological innovation and economic productivity.\n\nThe French Revolution created lasting democratic ideals that influence modern political systems, whereas the Industrial Revolution fundamentally redefined human interaction with technology and industry, laying foundations for modern capitalism and urban life.\n\nBoth events were pivotal in shaping modern history, but through different mechanisms: one through political change, the other through economic and technological advancement." + }, + { + "created_at": 1791252531824, + "role": "user", + "text": "What is the capital of Australia?" + }, + { + "created_at": 1791252594028, + "role": "assistant", + "text": "The capital of Australia is Canberra. Unlike most countries, Australia does not have a capital city located at a traditional geographic center. Instead, Canberra was deliberately created as a capital in 1913, chosen to serve as a compromise between Sydney and Melbourne, the two largest cities. \n\nIt is a planned city, designed by architects Walter Burley Griffin and Marion Mahony Griffin, and officially inaugurated in 1913. Its location in the Australian Capital Territory (ACT), in the inland region, avoids the political influence of coastal cities. \n\nCanberra is not only a political hub\u2014housing the Parliament House and the executive branch\u2014but also a symbol of national unity. Unlike Sydney or Melbourne, which host major economies and cultures, Canberra's purpose is administrative and symbolic. \n\nThis design reflects Australia\u2019s democratic values and commitment to equitable governance. It stands in contrast to capitals like London or Washington, which have long-standing historical roots. \n\nThus, Canberra's uniqueness lies in its deliberate, neutral establishment as a national center." + } + ] +} \ No newline at end of file diff --git a/docs/evidence/2026-10-05-android-conversations/conversation/norepack-4b.json b/docs/evidence/2026-10-05-android-conversations/conversation/norepack-4b.json new file mode 100644 index 000000000..54d03ad1f --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/conversation/norepack-4b.json @@ -0,0 +1,134 @@ +{ + "label": "norepack-4b", + "since_ms": 1791224120384, + "telemetry": [ + { + "created_at": 1791224193127, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "research", + "ttft_ms": 13350.508428990841, + "tok_per_sec": 5.2966262586050235, + "tokens_generated": 273, + "generation_latency_ms": 51542.243434011936, + "total_latency_ms": 65383.94408297539, + "outcome": "success" + }, + { + "created_at": 1791224357406, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 18148.76599997282, + "tok_per_sec": 4.636766264904051, + "tokens_generated": 314, + "generation_latency_ms": 67719.60932701826, + "total_latency_ms": 86659.5367039442, + "outcome": "success" + }, + { + "created_at": 1791224500179, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 27731.24858903885, + "tok_per_sec": 3.991452155729651, + "tokens_generated": 262, + "generation_latency_ms": 65640.2707029581, + "total_latency_ms": 94284.43677300215, + "outcome": "success" + }, + { + "created_at": 1791224598428, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 11474.15157699585, + "tok_per_sec": 4.622862504323237, + "tokens_generated": 163, + "generation_latency_ms": 35259.53883498907, + "total_latency_ms": 47728.817731022835, + "outcome": "success" + }, + { + "created_at": 1791224797295, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "compare", + "ttft_ms": 56049.445502996445, + "tok_per_sec": 3.5873790460138837, + "tokens_generated": 235, + "generation_latency_ms": 65507.4350900054, + "total_latency_ms": 122222.21347904205, + "outcome": "success" + }, + { + "created_at": 1791224926977, + "model_id": "qwen3-4b-instruct-2507-q4km", + "task_type": "lookup", + "ttft_ms": 51466.14138299227, + "tok_per_sec": 3.9742944285720827, + "tokens_generated": 112, + "generation_latency_ms": 28181.102837979794, + "total_latency_ms": 80808.56298798323, + "outcome": "success" + } + ], + "messages": [ + { + "created_at": 1791224127610, + "role": "user", + "text": "How do vaccines work?" + }, + { + "created_at": 1791224193131, + "role": "assistant", + "text": "Vaccines work by training the immune system to recognize and fight specific pathogens\u2014like viruses or bacteria\u2014without causing the full disease. They introduce a harmless component of the pathogen, such as a weakened or inactivated form, a piece of a virus (like a surface protein), or its toxin, which triggers an immune response [1]. This response creates memory cells that \"remember\" the pathogen, enabling a faster and stronger defense if the real pathogen later enters the body.\n\nUnlike natural infection, vaccines do not cause the disease. For example, the measles vaccine uses a weakened strain of the virus, which provokes immunity without leading to severe illness. This makes vaccines safer and more predictable than natural exposure, which carries risks like complications or death.\n\nThe immune response involves B cells producing antibodies and T cells targeting infected cells. Once activated, these cells remain dormant, allowing the body to respond rapidly upon future exposure. This is why vaccinated individuals often experience milder symptoms or no symptoms at all when infected.\n\nCompared to natural infection, vaccines provide immunity with far less risk and often without requiring actual disease exposure. For instance, the HPV vaccine prevents cervical cancer by targeting specific viral proteins, demonstrating how targeted immunity can prevent chronic illness.\n\nIn essence, vaccines are a safe, effective, and scientifically validated method of building immunity through controlled exposure to pathogen components." + }, + { + "created_at": 1791224270659, + "role": "user", + "text": "What are the side effects?" + }, + { + "created_at": 1791224357412, + "role": "assistant", + "text": "Vaccines are generally safe, with side effects typically mild and short-lived. Common side effects include soreness at the injection site, low-grade fever, fatigue, or headache\u2014similar to symptoms from a mild illness. These occur because the immune system is actively responding to the vaccine\u2019s components [1]. For example, the Pfizer-BioNTech COVID-19 vaccine may cause a sore arm or mild fever within 1\u20133 days, which typically resolves within a day or two.\n\nUnlike natural infections, vaccines do not cause the full disease. This means the risk of severe illness or complications is negligible. For instance, while a natural measles infection carries a 1 in 1,000 risk of death, vaccinated individuals face nearly zero such risk [1]. The immune response triggered by vaccines is controlled and targeted, reducing the chance of harmful side effects.\n\nSome people report rare adverse events, like allergic reactions (e.g., anaphylaxis), which are extremely uncommon\u2014less than 1 in a million doses. These are more likely in individuals with severe allergies to vaccine components (e.g., gelatin or polyethylene glycol). Such reactions are monitored closely and are preventable with proper screening.\n\nIn comparison, natural infection often leads to more severe outcomes, including hospitalization or long-term complications. Vaccines provide immunity without these risks, offering a safer, more predictable path to protection.\n\nOverall, the benefits of vaccination\u2014preventing disease, reducing transmission, and saving lives\u2014far outweigh the rare and mild side effects experienced by most people." + }, + { + "created_at": 1791224405880, + "role": "user", + "text": "Who proposed the theory of evolution by natural selection?" + }, + { + "created_at": 1791224500180, + "role": "assistant", + "text": "The theory of evolution by natural selection was primarily proposed by Charles Darwin, building on earlier ideas from scientists like Alfred Russel Wallace. Darwin\u2019s key insight was that individuals with heritable traits better suited to their environment are more likely to survive and reproduce, passing those advantageous traits to offspring. This process, over generations, leads to changes in populations\u2014what we now call evolution [1].\n\nDarwin\u2019s theory differed from earlier views, such as Lamarckism, which claimed organisms acquire traits during life and pass them on. Natural selection is a more evidence-based, observable process: it relies on variation, inheritance, and differential survival. For example, in finch populations on the Gal\u00e1pagos Islands, birds with larger beaks survived droughts better, leading to a shift in beak size over time.\n\nDarwin didn\u2019t work in isolation. He drew on data from geology, biogeography, and embryology, and his work was refined by later scientists like Gregor Mendel, whose principles of inheritance provided the genetic basis for natural selection. Together, these advances explain how evolution is both a process and a mechanism.\n\nIn contrast to theories of divine design or spontaneous generation, natural selection offers a testable, predictive framework. Its success in explaining biodiversity and adaptation underscores its central role in biology." + }, + { + "created_at": 1791224550681, + "role": "user", + "text": "What did he publish?" + }, + { + "created_at": 1791224598432, + "role": "assistant", + "text": "Charles Darwin published *On the Origin of Species* in 1859, which presented a comprehensive theory of evolution by natural selection. The book detailed how species adapt over time through heritable variations that enhance survival and reproduction in specific environments. Darwin used evidence from biogeography, fossils, and comparative anatomy, showing patterns of relatedness across species [1].\n\nHe contrasted natural selection with Lamarckism, which claimed traits were passed based on use or disuse. Darwin\u2019s mechanism\u2014where advantageous traits become more common through differential survival\u2014was grounded in observable natural processes.\n\nThe work influenced biology profoundly, shifting scientific thought from static species to dynamic, evolving life. Though Darwin didn't publish all data immediately, *On the Origin of Species* became the foundational text, establishing evolutionary biology as a scientific discipline." + }, + { + "created_at": 1791224674891, + "role": "user", + "text": "Compare the French Revolution and the Industrial Revolution." + }, + { + "created_at": 1791224797301, + "role": "assistant", + "text": "The French Revolution (1789\u20131799) was a political and social upheaval that transformed France\u2019s governance, ending absolute monarchy and establishing a republic. It was driven by economic hardship, social inequality, and Enlightenment ideals, leading to radical changes in law, citizenship, and governance [1]. In contrast, the Industrial Revolution (beginning c. 1760) was an economic transformation marked by mechanization, urbanization, and the rise of factories, fundamentally altering production methods and societal structures [2].\n\nWhile the French Revolution focused on political reorganization and human rights\u2014such as equality and liberty\u2014it did not significantly alter economic systems. The Industrial Revolution, however, redefined the economy, shifting from agrarian to industrial economies and creating new social classes, including the working class and industrial capitalists.\n\nThe French Revolution had profound ideological impacts, influencing liberal democracies worldwide. The Industrial Revolution, by contrast, reshaped global economies, leading to technological innovation, increased productivity, and long-term changes in labor, transport, and city life.\n\nBoth revolutions were transformative, but in different domains: one a political revolution of ideas and institutions, the other a" + }, + { + "created_at": 1791224846138, + "role": "user", + "text": "What is the capital of Australia?" + }, + { + "created_at": 1791224926987, + "role": "assistant", + "text": "The capital of Australia is Canberra. Unlike most countries, Australia does not have a capital city located at a geographic center. Instead, Canberra was deliberately chosen as a compromise between Sydney and Melbourne, the two largest cities. It was established in 1913 as a neutral seat of government to avoid political bias. Unlike capitals like London or Washington, Canberra is a planned city, designed for symmetry and civic balance. Its location in the Australian Capital Territory (ACT) ensures equitable representation of all states. This reflects Australia\u2019s commitment to federal governance and national unity." + } + ] +} \ No newline at end of file diff --git a/docs/evidence/2026-10-05-android-conversations/harness/base-4b-std/eval-2026-10-05T00-04-36-489Z.answers.md b/docs/evidence/2026-10-05-android-conversations/harness/base-4b-std/eval-2026-10-05T00-04-36-489Z.answers.md new file mode 100644 index 000000000..9a00e9ed8 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/base-4b-std/eval-2026-10-05T00-04-36-489Z.answers.md @@ -0,0 +1,139 @@ +## greeting-1 (greeting) + +> hey, what's up? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Hey! Not much, just here and ready to help. What’s on your mind? + +## factual-1 (factual) + +> What is the capital of Australia? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +The capital of Australia is Canberra. + +## factual-2 (factual) + +> Who proposed the theory of evolution by natural selection? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Evolution + +Charles Darwin proposed the theory of evolution by natural selection. [1] + +## explanation-1 (explanation) + +> How do vaccines work? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Vaccine + +Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed virus, or a surface protein—into the body, which triggers the immune system to produce antibodies and memory cells. This prepares the body to recognize and fight the actual disease if exposed later [1]. + +## explanation-2 (explanation) + +> Explain photosynthesis in simple terms. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Photosynthesis + +Photosynthesis is how plants use sunlight to turn carbon dioxide and water into food (like sugar) and release oxygen as a byproduct. This process happens in the leaves and helps plants grow while giving us the oxygen we need to breathe [1]. + +## comparison-1 (comparison) + +> Compare the French Revolution and the Industrial Revolution. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: French Revolution, Industrial Revolution + +The French Revolution (1789–1799) was a political and social upheaval that transformed France’s government and inspired democratic ideals, driven by economic hardship and demands for equality [1]. The Industrial Revolution (beginning c. 1760) was a shift in economic systems, introducing mechanized manufacturing and urbanization, fundamentally altering economies and societies worldwide [2]. While the French Revolution focused on political change and individual rights, the Industrial Revolution centered on technological advancement and economic transformation. + +## comparison-2 (comparison) + +> Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Supply and demand, Behavioral economics + +The supply and demand model assumes rational, informed actors who act to maximize utility, leading to equilibrium prices when quantity demanded equals quantity supplied [1]. Behavioral economics complicates this by showing that individuals often make irrational, emotionally driven, or bounded-rational decisions—such as herd behavior, loss aversion, or cognitive biases—that cause prices to deviate from equilibrium [2]. The simple model breaks down when explaining price bubbles, crashes, or persistent market anomalies where psychological and social factors dominate over pure supply and demand signals. + +## synthesis-1 (synthesis) + +> What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Industrial Revolution, Agricultural revolution + +Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society, transitioning from subsistence-based lifestyles to more productive, stable systems of food and manufacturing [1]. The Agricultural Revolution (starting ~10,000 BCE) changed daily life gradually over millennia, as people shifted from hunting/gathering to farming, enabling population growth and settlement. In contrast, the Industrial Revolution (beginning ~1760) transformed daily life more rapidly—within decades—through mechanization, urbanization, and mass production, drastically altering work, living conditions, and social structures. + +## synthesis-2 (synthesis) + +> How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Immune system, Vaccine + +The immune system detects pathogens and mounts a response by identifying foreign molecules, triggering both innate and adaptive responses. Vaccines introduce a harmless version of a pathogen—such as a weakened or killed microbe or its proteins—so the immune system learns to recognize and respond to it without causing disease. This priming allows the adaptive immune system to remember the pathogen, enabling a faster, stronger response if the real pathogen is encountered later [2]. + +## synthesis-3 (synthesis) + +> Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Biodiversity loss, Amazon rainforest + +The Amazon rainforest is crucial for global climate regulation because it absorbs large amounts of CO₂ and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens not just species survival, but also ecosystem resilience—disrupting climate regulation, reducing soil fertility, and weakening the forest’s ability to sequester carbon, which could accelerate global warming and reduce natural climate buffers. + +## reasoning-1 (reasoning) + +> A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. +The train arrives at 6:15 pm. + +## reasoning-2 (reasoning) + +> A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Total used: 2GB (OS/app) + 200MB (embedding) + 1.5GB (KV cache) = 3.2GB. +Remaining for LLM weights: 12GB - 3.2GB = 8.8GB. +A 9GB model would exceed the available 8.8GB, so it would not fit if loaded fully into RAM. + +## reasoning-3 (reasoning) + +> If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Minimum disk footprint: 8B per token × 1 byte/Q8 = 8 bytes per token. For a token stream, the total disk footprint is proportional to the number of tokens (e.g., 8 bytes per token), not the total 100B parameter count. RAM usage doesn't scale with the full 100B because only 8B per token is activated at any time (via expert selection), and the model uses parameter sparsity and activation masking — only active experts and their weights are loaded into memory. + +## grounded-1 (retrieval-grounded) + +> Why did the Western Roman Empire fall? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Fall of the Western Roman Empire, Byzantine Empire + +The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, political instability, and invasions by external groups like the Visigoths and Vandals. Internal issues such as population decline, disease, and religious changes also contributed [1]. The empire lost its ability to govern effectively as these pressures mounted over time. + +## grounded-2 (retrieval-grounded) + +> What is a black hole and how does one form? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Black hole + +A black hole is an astronomical object so dense that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, compressing its mass into an extremely small volume. + +## no-kb-1 (no-kb-content) + +> Who was Napoleon Bonaparte? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He expanded the French Empire across Europe, implemented major legal and administrative reforms, and was a key figure in shaping modern European politics and military strategy. + +## no-kb-2 (no-kb-content) + +> How do antibiotics work, and why does antibiotic resistance develop? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Antimicrobial resistance + +Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis or protein production, to kill or inhibit their growth [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to the antibiotic, often due to misuse or overuse in medicine or agriculture. + diff --git a/docs/evidence/2026-10-05-android-conversations/harness/base-4b-std/eval-2026-10-05T00-04-36-489Z.jsonl b/docs/evidence/2026-10-05-android-conversations/harness/base-4b-std/eval-2026-10-05T00-04-36-489Z.jsonl new file mode 100644 index 000000000..d90b07d61 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/base-4b-std/eval-2026-10-05T00-04-36-489Z.jsonl @@ -0,0 +1,17 @@ +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"greeting","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":8980.85356003046,"generationLatencyMs":2668.335653960705,"totalLatencyMs":11660.115118026733,"tokensGenerated":18,"tokPerSec":6.745778018324631,"peakRssBytes":2698452992,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"greeting-1","category":"greeting","query":"hey, what's up?","answer":"Hey! Not much, just here and ready to help. What’s on your mind?","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791158688239} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":503.81774896383286,"generationLatencyMs":887.9364430308342,"totalLatencyMs":1729.2150069475174,"tokensGenerated":7,"tokPerSec":7.883447126132788,"peakRssBytes":2823872512,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":336.84757500886917,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-1","category":"factual","query":"What is the capital of Australia?","answer":"The capital of Australia is Canberra.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791158690172} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9090.01192599535,"generationLatencyMs":2095.064209997654,"totalLatencyMs":11419.613043010235,"tokensGenerated":14,"tokPerSec":6.68237275649689,"peakRssBytes":2866528256,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":234.24336701631546,"contextTokensBefore":111,"contextTokensAfter":111,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-2","category":"factual","query":"Who proposed the theory of evolution by natural selection?","answer":"Charles Darwin proposed the theory of evolution by natural selection. [1]","promptFormat":"chat-template","retrievedTitles":["Evolution"],"expectedKbTitles":["Evolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791158701634} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":3251.991903960705,"generationLatencyMs":9495.239588022232,"totalLatencyMs":13012.361456036568,"tokensGenerated":60,"tokPerSec":6.318955877183656,"peakRssBytes":2888065024,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":264.83504301309586,"contextTokensBefore":154,"contextTokensAfter":102,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-1","category":"explanation","query":"How do vaccines work?","answer":"Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed virus, or a surface protein—into the body, which triggers the immune system to produce antibodies and memory cells. This prepares the body to recognize and fight the actual disease if exposed later [1].","promptFormat":"chat-template","retrievedTitles":["Vaccine"],"expectedKbTitles":["Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791158714710} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":6828.397992968559,"generationLatencyMs":8390.197677016258,"totalLatencyMs":15454.139614999294,"tokensGenerated":49,"tokPerSec":5.840148454931933,"peakRssBytes":2909069312,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":235.16935300827026,"contextTokensBefore":372,"contextTokensAfter":181,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-2","category":"explanation","query":"Explain photosynthesis in simple terms.","answer":"Photosynthesis is how plants use sunlight to turn carbon dioxide and water into food (like sugar) and release oxygen as a byproduct. This process happens in the leaves and helps plants grow while giving us the oxygen we need to breathe [1].","promptFormat":"chat-template","retrievedTitles":["Photosynthesis"],"expectedKbTitles":["Photosynthesis"],"expectedKbHit":true,"timedOut":false,"createdAt":1791158730247} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"compare","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9994.964605033398,"generationLatencyMs":18778.185880959034,"totalLatencyMs":29033.627659022808,"tokensGenerated":103,"tokPerSec":5.485087891500817,"peakRssBytes":2937335808,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":260.24043798446655,"contextTokensBefore":286,"contextTokensAfter":286,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-1","category":"comparison","query":"Compare the French Revolution and the Industrial Revolution.","answer":"The French Revolution (1789–1799) was a political and social upheaval that transformed France’s government and inspired democratic ideals, driven by economic hardship and demands for equality [1]. The Industrial Revolution (beginning c. 1760) was a shift in economic systems, introducing mechanized manufacturing and urbanization, fundamentally altering economies and societies worldwide [2]. While the French Revolution focused on political change and individual rights, the Industrial Revolution centered on technological advancement and economic transformation.","promptFormat":"chat-template","retrievedTitles":["French Revolution","Industrial Revolution"],"expectedKbTitles":["French Revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791158759356} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":7621.422245025635,"generationLatencyMs":18835.28805398941,"totalLatencyMs":26748.414359986782,"tokensGenerated":103,"tokPerSec":5.4684589747054115,"peakRssBytes":2946236416,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":288.1071780323982,"contextTokensBefore":188,"contextTokensAfter":188,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-2","category":"comparison","query":"Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down?","answer":"The supply and demand model assumes rational, informed actors who act to maximize utility, leading to equilibrium prices when quantity demanded equals quantity supplied [1]. Behavioral economics complicates this by showing that individuals often make irrational, emotionally driven, or bounded-rational decisions—such as herd behavior, loss aversion, or cognitive biases—that cause prices to deviate from equilibrium [2]. The simple model breaks down when explaining price bubbles, crashes, or persistent market anomalies where psychological and social factors dominate over pure supply and demand signals.","promptFormat":"chat-template","retrievedTitles":["Supply and demand","Behavioral economics"],"expectedKbTitles":["Supply and demand","Behavioral economics"],"expectedKbHit":true,"timedOut":false,"createdAt":1791158786185} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":12077.795497000217,"generationLatencyMs":25537.056735038757,"totalLatencyMs":37880.13038897514,"tokensGenerated":116,"tokPerSec":4.542418541164116,"peakRssBytes":2941894656,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":264.81571501493454,"contextTokensBefore":308,"contextTokensAfter":307,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-1","category":"synthesis","query":"What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life?","answer":"Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society, transitioning from subsistence-based lifestyles to more productive, stable systems of food and manufacturing [1]. The Agricultural Revolution (starting ~10,000 BCE) changed daily life gradually over millennia, as people shifted from hunting/gathering to farming, enabling population growth and settlement. In contrast, the Industrial Revolution (beginning ~1760) transformed daily life more rapidly—within decades—through mechanization, urbanization, and mass production, drastically altering work, living conditions, and social structures.","promptFormat":"chat-template","retrievedTitles":["Industrial Revolution","Agricultural revolution"],"expectedKbTitles":["Agricultural revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791158824147} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":14627.860237002373,"generationLatencyMs":18576.560677945614,"totalLatencyMs":33615.40846401453,"tokensGenerated":90,"tokPerSec":4.844815009640047,"peakRssBytes":2957234176,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":410.5345870256424,"contextTokensBefore":330,"contextTokensAfter":298,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-2","category":"synthesis","query":"How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism.","answer":"The immune system detects pathogens and mounts a response by identifying foreign molecules, triggering both innate and adaptive responses. Vaccines introduce a harmless version of a pathogen—such as a weakened or killed microbe or its proteins—so the immune system learns to recognize and respond to it without causing disease. This priming allows the adaptive immune system to remember the pathogen, enabling a faster, stronger response if the real pathogen is encountered later [2].","promptFormat":"chat-template","retrievedTitles":["Immune system","Vaccine"],"expectedKbTitles":["Immune system","Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791158857899} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":35089.6581799984,"generationLatencyMs":15656.766242980957,"totalLatencyMs":52352.829370975494,"tokensGenerated":78,"tokPerSec":4.981871657882609,"peakRssBytes":3034714112,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":1605.7679859995842,"contextTokensBefore":310,"contextTokensAfter":216,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-3","category":"synthesis","query":"Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species?","answer":"The Amazon rainforest is crucial for global climate regulation because it absorbs large amounts of CO₂ and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens not just species survival, but also ecosystem resilience—disrupting climate regulation, reducing soil fertility, and weakening the forest’s ability to sequester carbon, which could accelerate global warming and reduce natural climate buffers.","promptFormat":"chat-template","retrievedTitles":["Biodiversity loss","Amazon rainforest"],"expectedKbTitles":["Amazon rainforest"],"expectedKbHit":true,"timedOut":false,"createdAt":1791158911008} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"chat","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9026.062749028206,"generationLatencyMs":10378.618005990982,"totalLatencyMs":19857.043995976448,"tokensGenerated":58,"tokPerSec":5.5884126351427446,"peakRssBytes":3041538048,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":451.7218019962311,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-1","category":"reasoning","query":"A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive?","answer":"Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. \nThe train arrives at 6:15 pm.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791158931011} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"calculate","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":4560.299440979958,"generationLatencyMs":17258.238005042076,"totalLatencyMs":21819.053517997265,"tokensGenerated":87,"tokPerSec":5.041070819314378,"peakRssBytes":3036741632,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-2","category":"reasoning","query":"A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM?","answer":"Total used: 2GB (OS/app) + 200MB (embedding) + 1.5GB (KV cache) = 3.2GB. \nRemaining for LLM weights: 12GB - 3.2GB = 8.8GB. \nA 9GB model would exceed the available 8.8GB, so it would not fit if loaded fully into RAM.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791158952998} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":3695.775961995125,"generationLatencyMs":22499.707937002182,"totalLatencyMs":26771.43446099758,"tokensGenerated":111,"tokPerSec":4.933397371681147,"peakRssBytes":3041906688,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":575.175862967968,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-3","category":"reasoning","query":"If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure?","answer":"Minimum disk footprint: 8B per token × 1 byte/Q8 = 8 bytes per token. For a token stream, the total disk footprint is proportional to the number of tokens (e.g., 8 bytes per token), not the total 100B parameter count. RAM usage doesn't scale with the full 100B because only 8B per token is activated at any time (via expert selection), and the model uses parameter sparsity and activation masking — only active experts and their weights are loaded into memory.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791158979934} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":29527.223565995693,"generationLatencyMs":15667.439460992813,"totalLatencyMs":45720.57175898552,"tokensGenerated":68,"tokPerSec":4.340211441014304,"peakRssBytes":3024785408,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":525.5115560293198,"contextTokensBefore":396,"contextTokensAfter":359,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-1","category":"retrieval-grounded","query":"Why did the Western Roman Empire fall?","answer":"The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, political instability, and invasions by external groups like the Visigoths and Vandals. Internal issues such as population decline, disease, and religious changes also contributed [1]. The empire lost its ability to govern effectively as these pressures mounted over time.","promptFormat":"chat-template","retrievedTitles":["Fall of the Western Roman Empire","Byzantine Empire"],"expectedKbTitles":["Fall of the Western Roman Empire"],"expectedKbHit":true,"timedOut":false,"createdAt":1791159025692} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":6473.539553999901,"generationLatencyMs":10271.306321024895,"totalLatencyMs":17158.855558991432,"tokensGenerated":51,"tokPerSec":4.965288582193808,"peakRssBytes":3023155200,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":413.48079496622086,"contextTokensBefore":155,"contextTokensAfter":121,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-2","category":"retrieval-grounded","query":"What is a black hole and how does one form?","answer":"A black hole is an astronomical object so dense that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, compressing its mass into an extremely small volume.","promptFormat":"chat-template","retrievedTitles":["Black hole"],"expectedKbTitles":["Black hole"],"expectedKbHit":true,"timedOut":false,"createdAt":1791159043043} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":7974.090986967087,"generationLatencyMs":11406.527348995209,"totalLatencyMs":19776.522266983986,"tokensGenerated":65,"tokPerSec":5.69849157515287,"peakRssBytes":3023310848,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":395.4268389940262,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-1","category":"no-kb-content","query":"Who was Napoleon Bonaparte?","answer":"Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He expanded the French Empire across Europe, implemented major legal and administrative reforms, and was a key figure in shaping modern European politics and military strategy.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791159063006} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":31208.867161989212,"generationLatencyMs":13173.439662992954,"totalLatencyMs":45581.73228096962,"tokensGenerated":63,"tokPerSec":4.782350062829881,"peakRssBytes":3048304640,"outcome":"success","runId":"eval-2026-10-05T00-04-36-489Z","evalSetVersion":"1","retrievalMs":1198.947103023529,"contextTokensBefore":175,"contextTokensAfter":147,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-2","category":"no-kb-content","query":"How do antibiotics work, and why does antibiotic resistance develop?","answer":"Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis or protein production, to kill or inhibit their growth [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to the antibiotic, often due to misuse or overuse in medicine or agriculture.","promptFormat":"chat-template","retrievedTitles":["Antimicrobial resistance"],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791159109329} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/base-4b-std/eval-2026-10-05T00-04-36-489Z.report.md b/docs/evidence/2026-10-05-android-conversations/harness/base-4b-std/eval-2026-10-05T00-04-36-489Z.report.md new file mode 100644 index 000000000..83c48bf42 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/base-4b-std/eval-2026-10-05T00-04-36-489Z.report.md @@ -0,0 +1,17 @@ +BOAR Device Evaluation — eval-2026-10-05T00-04-36-489Z (set v1, 17 queries) +──────────────────────────────────────────────────────────── + +Qwen3-4B-Instruct-2507 (Q4_K_M) + Config: model:qwen3-4b-instruct-2507-q4km · prompt chat-template + Queries: 17 ok · 0 failed · 0 cancelled (of 17) + TTFT: avg 11.8s · p50 9.0s + Model load: none (already resident) + Generation: avg 13.0s · p50 13.2s + Tokens/sec: avg 5.5 · p50 5.5 + Search time: avg 0.5s · p50 0.3s (before the model starts) + Context tokens: 253 → 211 avg (trimmed to the answering sentences) + Total per query: avg 25.3s · p50 21.8s + Peak RSS: 2.84 GB + Residency: cold 0 · switched 0 · resident 17 + Model switches: 0 within plans · 0 across messages + Retrieval: used 11/17 · expected KB article found 10/10 diff --git a/docs/evidence/2026-10-05-android-conversations/harness/base-4b-std/eval-2026-10-05T00-04-36-489Z.status.json b/docs/evidence/2026-10-05-android-conversations/harness/base-4b-std/eval-2026-10-05T00-04-36-489Z.status.json new file mode 100644 index 000000000..3f3fee5aa --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/base-4b-std/eval-2026-10-05T00-04-36-489Z.status.json @@ -0,0 +1,17 @@ +{ + "requestId": "req-20261005t000431-3a58", + "state": "done", + "configs": [ + "model:qwen3-4b-instruct-2507-q4km" + ], + "total": 17, + "completed": 17, + "runId": "eval-2026-10-05T00-04-36-489Z", + "resultPath": "files/eval/eval-2026-10-05T00-04-36-489Z.jsonl", + "stopped": false, + "installedModels": [ + "qwen3-4b-instruct-2507-q4km", + "qwen2.5-1.5b-instruct-q4km" + ], + "updatedAt": 1791159109516 +} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/base-4b-vitalik/eval-2026-10-05T00-27-54-743Z.jsonl b/docs/evidence/2026-10-05-android-conversations/harness/base-4b-vitalik/eval-2026-10-05T00-27-54-743Z.jsonl new file mode 100644 index 000000000..9eb157bcb --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/base-4b-vitalik/eval-2026-10-05T00-27-54-743Z.jsonl @@ -0,0 +1,6 @@ +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"chat","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":10669.934168994427,"generationLatencyMs":15132.802781999111,"totalLatencyMs":26080.265718996525,"tokensGenerated":79,"tokPerSec":5.220447338015447,"peakRssBytes":2657308672,"outcome":"success","runId":"eval-2026-10-05T00-27-54-743Z","evalSetVersion":"vitalik-1","retrievalMs":276.75789403915405,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-about-x","category":"vitalik","query":"Tell me about X.com.","answer":"X.com was founded in 1999 by Elon Musk as a peer-to-peer payment platform. It was later rebranded as \"X\" in 2023, becoming a social media and messaging platform, distinct from the original financial service. The company is now primarily known as the platform formerly known as Twitter, which shifted its focus to real-time communication and public discourse.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["X (social network)"],"expectedKbHit":false,"timedOut":false,"createdAt":1791160100869} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":1034.3592119812965,"generationLatencyMs":17587.939625024796,"totalLatencyMs":18969.573495984077,"tokensGenerated":107,"tokPerSec":6.083714311126944,"peakRssBytes":2805817344,"outcome":"success","runId":"eval-2026-10-05T00-27-54-743Z","evalSetVersion":"vitalik-1","retrievalMs":346.57609099149704,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-zk-snark","category":"vitalik","query":"How does a zk-SNARK differ from a zk-STARK?","answer":"zk-SNARKs and zk-STARKs are both zero-knowledge proof systems that allow one party to prove knowledge of a secret without revealing it. zk-SNARKs (Zero-Knowledge Succinct Non-Interactive Arguments of Knowledge) are more compact and efficient but rely on a trusted setup, while zk-STARKs (Zero-Knowledge Scalable Transparent ARguments of Knowledge) are transparent, do not require a trusted setup, and offer stronger security guarantees, though they are typically larger in size and slower in computation.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Zero-knowledge proof","Non-interactive zero-knowledge proof"],"expectedKbHit":false,"timedOut":false,"createdAt":1791160120023} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":864.7390959858894,"generationLatencyMs":17149.71713000536,"totalLatencyMs":18309.575529038906,"tokensGenerated":104,"tokPerSec":6.064239964520482,"peakRssBytes":2819862528,"outcome":"success","runId":"eval-2026-10-05T00-27-54-743Z","evalSetVersion":"vitalik-1","retrievalMs":294.68168103694916,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-quadratic-funding","category":"vitalik","query":"What is quadratic funding and what are its weaknesses?","answer":"Quadratic funding is a mechanism used in decentralized communities to allocate funds to proposals based on participation and impact, where the funding increases with the square of the number of participants. This encourages broad participation and ensures that projects with wide support receive more funding relative to their cost.\n\nWeaknesses include potential inefficiency in funding highly popular but low-impact projects, difficulty in distinguishing between genuine community interest and noise, and the risk of over-funding projects with many small contributors while under-funding those with fewer but more dedicated supporters.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Quadratic funding","Quadratic voting"],"expectedKbHit":false,"timedOut":false,"createdAt":1791160138458} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":1061.2976489663124,"generationLatencyMs":14209.734016001225,"totalLatencyMs":15521.217658996582,"tokensGenerated":87,"tokPerSec":6.122563582262095,"peakRssBytes":2821152768,"outcome":"success","runId":"eval-2026-10-05T00-27-54-743Z","evalSetVersion":"vitalik-1","retrievalMs":249.74759995937347,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-far-uvc","category":"vitalik","query":"Does far-UVC light at 222 nm harm human skin?","answer":"Far-UVC light at 222 nm does not significantly harm human skin because it is largely absorbed by the outermost layer of the skin (stratum corneum) and does not penetrate deeply into living tissue. Studies suggest it is safe for human exposure at typical doses used for disinfection, as it primarily affects microbes without reaching deeper skin layers. However, long-term or high-dose exposure effects are still under investigation.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Far-UVC","Ultraviolet germicidal irradiation"],"expectedKbHit":false,"timedOut":false,"createdAt":1791160154069} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":673.0546879768372,"generationLatencyMs":9484.159062027931,"totalLatencyMs":10444.152348995209,"tokensGenerated":61,"tokPerSec":6.4317774091566955,"peakRssBytes":2826870784,"outcome":"success","runId":"eval-2026-10-05T00-27-54-743Z","evalSetVersion":"vitalik-1","retrievalMs":286.51879900693893,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-zuzalu","category":"vitalik","query":"What was Zuzalu?","answer":"Zuzalu was a short-lived online platform launched in the early 2000s that aimed to provide a social networking experience focused on user-generated content and community interaction. It quickly failed due to technical issues, lack of user engagement, and competition from emerging platforms like Friendster and MySpace.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Zuzalu","Network state"],"expectedKbHit":false,"timedOut":false,"createdAt":1791160164610} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":939.295979976654,"generationLatencyMs":55629.90827101469,"totalLatencyMs":56812.755479991436,"tokensGenerated":75,"tokPerSec":1.3481956438723424,"peakRssBytes":2829963264,"outcome":"success","runId":"eval-2026-10-05T00-27-54-743Z","evalSetVersion":"vitalik-1","retrievalMs":241.09590595960617,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-vegan-buenos-aires","category":"vitalik","query":"Where can I find vegan food in Buenos Aires?","answer":"In Buenos Aires, you can find vegan food at places like *El Vegetal*, *La Vida Saludable*, and *Veggie Kitchen* in the Recoleta and San Telmo neighborhoods. Many cafes and restaurants in the Puerto Madero and Microcentro areas also offer vegan options. Check local vegan food guides or apps like HappyCow for updated listings.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Buenos Aires","Veganism"],"expectedKbHit":false,"timedOut":false,"createdAt":1791160222284} \ No newline at end of file diff --git a/docs/evidence/2026-10-05-android-conversations/harness/base-4b-vitalik/eval-2026-10-05T00-27-54-743Z.report.md b/docs/evidence/2026-10-05-android-conversations/harness/base-4b-vitalik/eval-2026-10-05T00-27-54-743Z.report.md new file mode 100644 index 000000000..0f1c85d4c --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/base-4b-vitalik/eval-2026-10-05T00-27-54-743Z.report.md @@ -0,0 +1,15 @@ +BOAR Device Evaluation — eval-2026-10-05T00-27-54-743Z (set vvitalik-1, 6 queries) +──────────────────────────────────────────────────────────── + +Qwen3-4B-Instruct-2507 (Q4_K_M) + Config: model:qwen3-4b-instruct-2507-q4km · prompt chat-template + Queries: 6 ok · 0 failed · 0 cancelled (of 6) + TTFT: avg 2.5s · p50 1.0s + Model load: none (already resident) + Generation: avg 21.5s · p50 16.1s + Tokens/sec: avg 5.2 · p50 6.1 + Total per query: avg 24.4s · p50 18.6s + Peak RSS: 2.64 GB + Residency: cold 0 · switched 0 · resident 6 + Model switches: 0 within plans · 0 across messages + Retrieval: used 0/6 · expected KB article found 0/6 diff --git a/docs/evidence/2026-10-05-android-conversations/harness/final-4b-std/eval-2026-10-05T23-07-42-918Z.answers.md b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-std/eval-2026-10-05T23-07-42-918Z.answers.md new file mode 100644 index 000000000..63f08bc35 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-std/eval-2026-10-05T23-07-42-918Z.answers.md @@ -0,0 +1,141 @@ +## greeting-1 (greeting) + +> hey, what's up? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Hey! Just checking in. How can I help you today? + +## factual-1 (factual) + +> What is the capital of Australia? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +The capital of Australia is Canberra. + +## factual-2 (factual) + +> Who proposed the theory of evolution by natural selection? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Evolution + +Charles Darwin proposed the theory of evolution by natural selection. [1] + +## explanation-1 (explanation) + +> How do vaccines work? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Vaccine + +Vaccines work by introducing a harmless version of a pathogen—such as a weakened or killed microorganism, or its proteins—into the body, which triggers the immune system to produce antibodies. This prepares the body to recognize and fight the actual disease if exposed later [1]. + +## explanation-2 (explanation) + +> Explain photosynthesis in simple terms. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Photosynthesis + +Photosynthesis is how plants use sunlight to turn carbon dioxide and water into sugar (food) and oxygen as a byproduct. The sunlight powers the process, and the oxygen is released into the air [1]. + +## comparison-1 (comparison) + +> Compare the French Revolution and the Industrial Revolution. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: French Revolution, Industrial Revolution + +The French Revolution (1789–1799) was a political and social upheaval that transformed France’s government and inspired democratic ideals, while the Industrial Revolution (beginning c. 1760) was an economic transformation marked by mechanization, urbanization, and industrial growth [1][2]. The former changed political structures and values, the latter shifted production methods and societal organization. Both were pivotal in shaping modern history but operated in different domains: political/social vs. economic/technological. + +## comparison-2 (comparison) + +> Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Supply and demand, Behavioral economics + +The supply and demand model assumes rational, informed actors who act to maximize utility, leading to equilibrium prices [1]. Behavioral economics complicates this by showing that individuals often make irrational, emotionally driven, or biased decisions—such as herd behavior, loss aversion, or cognitive biases—that cause prices to deviate from equilibrium [2]. The simple model breaks down when real-world factors like emotions, social influences, or limited information override rational choice, leading to market anomalies like bubbles or crashes. + +## synthesis-1 (synthesis) + +> What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Industrial Revolution, Agricultural revolution + +Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society by transforming how people produced food and goods, leading to permanent settlements and population growth [1]. The Agricultural Revolution gradually changed daily life over centuries, transitioning societies from nomadic hunting to settled farming; in contrast, the Industrial Revolution brought rapid, dramatic changes in daily life within a few decades, especially through mechanization and urbanization [1]. + +## synthesis-2 (synthesis) + +> How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Immune system, Vaccine + +The immune system detects pathogens and mounts a response by identifying foreign molecules, activating innate and adaptive defenses, and creating memory cells for future responses. A vaccine mimics this by introducing a harmless version of a pathogen (like a weakened or inactivated form or a surface protein), which triggers the adaptive immune system to recognize and respond to that specific pathogen. This creates memory cells that enable a faster, stronger response if the real pathogen is encountered later [2]. + +## synthesis-3 (synthesis) + +> Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Biodiversity loss, Amazon rainforest + +The Amazon rainforest is crucial for global climate regulation because it absorbs significant amounts of CO₂ and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens ecosystem resilience, disrupts climate feedback loops (e.g., reduced carbon sequestration), and compromises vital services like water cycling and soil fertility, which affect both regional and global environmental stability. + +## reasoning-1 (reasoning) + +> A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. +Arrival time: 6:15 pm. + +## reasoning-2 (reasoning) + +> A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Total used: 2GB (OS/app) + 200MB (embedding) + 1.5GB (KV cache) = 3.7GB. +Remaining: 12GB - 3.7GB = 8.3GB. +A 9GB model would exceed the available RAM and not fit if loaded fully. +Answer: 8.3GB left; no, 9GB model does not fit. + +## reasoning-3 (reasoning) + +> If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Minimum disk footprint: 8B × 1 byte = 8 bytes per token. +RAM usage doesn’t scale with 100B because only 8B per token are activated at inference—only the selected expert parameters are computed, not the full model. The total parameter count (100B) is a design choice for capacity, not runtime usage. + +## grounded-1 (retrieval-grounded) + +> Why did the Western Roman Empire fall? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Fall of the Western Roman Empire, Byzantine Empire + +The Western Roman Empire fell due to a combination of factors including military weakening, economic decline, population decline, political instability, and external invasions by barbarian tribes. Climatic changes and widespread diseases also contributed to the collapse [1]. The Eastern Roman (Byzantine) Empire continued to thrive until 1453 [2]. + +## grounded-2 (retrieval-grounded) + +> What is a black hole and how does one form? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Black hole + +A black hole is an astronomical object so dense that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, compressing its mass into an extremely small volume, creating a region where spacetime curvature becomes extreme. + +## no-kb-1 (no-kb-content) + +> Who was Napoleon Bonaparte? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He led France through a series of military campaigns across Europe, significantly shaping the political landscape of the continent, and later ruled as a dominant figure in European history until his exile in 1815. + +## no-kb-2 (no-kb-content) + +> How do antibiotics work, and why does antibiotic resistance develop? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Antimicrobial resistance + +Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis or protein production, to kill or inhibit their growth [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to misuse or overuse in medicine and agriculture. + diff --git a/docs/evidence/2026-10-05-android-conversations/harness/final-4b-std/eval-2026-10-05T23-07-42-918Z.jsonl b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-std/eval-2026-10-05T23-07-42-918Z.jsonl new file mode 100644 index 000000000..53524c6f4 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-std/eval-2026-10-05T23-07-42-918Z.jsonl @@ -0,0 +1,17 @@ +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"greeting","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":true,"modelResidency":"switched","modelLoadMs":4263.414633989334,"ttftMs":15274.084562003613,"generationLatencyMs":1622.7388110160828,"totalLatencyMs":21167.57207196951,"tokensGenerated":13,"tokPerSec":8.01114751908843,"peakRssBytes":2964856832,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"greeting-1","category":"greeting","query":"hey, what's up?","answer":"Hey! Just checking in. How can I help you today?","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791241684153} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":1034.8756919503212,"generationLatencyMs":871.5878100395203,"totalLatencyMs":2254.09700602293,"tokensGenerated":7,"tokPerSec":8.031319299523705,"peakRssBytes":3047141376,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":347.029785990715,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-1","category":"factual","query":"What is the capital of Australia?","answer":"The capital of Australia is Canberra.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791241686565} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":23886.57854294777,"generationLatencyMs":2201.3081480264664,"totalLatencyMs":26326.72686100006,"tokensGenerated":14,"tokPerSec":6.359854713003896,"peakRssBytes":3068719104,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":238.47888201475143,"contextTokensBefore":111,"contextTokensAfter":111,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-2","category":"factual","query":"Who proposed the theory of evolution by natural selection?","answer":"Charles Darwin proposed the theory of evolution by natural selection. [1]","promptFormat":"chat-template","retrievedTitles":["Evolution"],"expectedKbTitles":["Evolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791241712950} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":8852.77547699213,"generationLatencyMs":9373.73494899273,"totalLatencyMs":18586.38880300522,"tokensGenerated":58,"tokPerSec":6.187501600547441,"peakRssBytes":2850410496,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":359.56087297201157,"contextTokensBefore":154,"contextTokensAfter":102,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-1","category":"explanation","query":"How do vaccines work?","answer":"Vaccines work by introducing a harmless version of a pathogen—such as a weakened or killed microorganism, or its proteins—into the body, which triggers the immune system to produce antibodies. This prepares the body to recognize and fight the actual disease if exposed later [1].","promptFormat":"chat-template","retrievedTitles":["Vaccine"],"expectedKbTitles":["Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791241731635} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":16703.719286978245,"generationLatencyMs":6737.738285005093,"totalLatencyMs":23719.112640976906,"tokensGenerated":42,"tokPerSec":6.233545772098545,"peakRssBytes":2860453888,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":277.3092039823532,"contextTokensBefore":372,"contextTokensAfter":181,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-2","category":"explanation","query":"Explain photosynthesis in simple terms.","answer":"Photosynthesis is how plants use sunlight to turn carbon dioxide and water into sugar (food) and oxygen as a byproduct. The sunlight powers the process, and the oxygen is released into the air [1].","promptFormat":"chat-template","retrievedTitles":["Photosynthesis"],"expectedKbTitles":["Photosynthesis"],"expectedKbHit":true,"timedOut":false,"createdAt":1791241755439} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"compare","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":25550.55234003067,"generationLatencyMs":20107.95134395361,"totalLatencyMs":45905.62075001001,"tokensGenerated":105,"tokPerSec":5.221814903166311,"peakRssBytes":2832691200,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":246.8630369901657,"contextTokensBefore":286,"contextTokensAfter":286,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-1","category":"comparison","query":"Compare the French Revolution and the Industrial Revolution.","answer":"The French Revolution (1789–1799) was a political and social upheaval that transformed France’s government and inspired democratic ideals, while the Industrial Revolution (beginning c. 1760) was an economic transformation marked by mechanization, urbanization, and industrial growth [1][2]. The former changed political structures and values, the latter shifted production methods and societal organization. Both were pivotal in shaping modern history but operated in different domains: political/social vs. economic/technological.","promptFormat":"chat-template","retrievedTitles":["French Revolution","Industrial Revolution"],"expectedKbTitles":["French Revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791241801434} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":26947.372367978096,"generationLatencyMs":19229.96188300848,"totalLatencyMs":46758.0605700016,"tokensGenerated":97,"tokPerSec":5.044211766519871,"peakRssBytes":2823933952,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":578.5366209745407,"contextTokensBefore":188,"contextTokensAfter":188,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-2","category":"comparison","query":"Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down?","answer":"The supply and demand model assumes rational, informed actors who act to maximize utility, leading to equilibrium prices [1]. Behavioral economics complicates this by showing that individuals often make irrational, emotionally driven, or biased decisions—such as herd behavior, loss aversion, or cognitive biases—that cause prices to deviate from equilibrium [2]. The simple model breaks down when real-world factors like emotions, social influences, or limited information override rational choice, leading to market anomalies like bubbles or crashes.","promptFormat":"chat-template","retrievedTitles":["Supply and demand","Behavioral economics"],"expectedKbTitles":["Supply and demand","Behavioral economics"],"expectedKbHit":true,"timedOut":false,"createdAt":1791241848431} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":36801.144549012184,"generationLatencyMs":15708.986132025719,"totalLatencyMs":52957.97544902563,"tokensGenerated":81,"tokPerSec":5.1562843915729415,"peakRssBytes":2845757440,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":447.05761700868607,"contextTokensBefore":308,"contextTokensAfter":307,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-1","category":"synthesis","query":"What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life?","answer":"Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society by transforming how people produced food and goods, leading to permanent settlements and population growth [1]. The Agricultural Revolution gradually changed daily life over centuries, transitioning societies from nomadic hunting to settled farming; in contrast, the Industrial Revolution brought rapid, dramatic changes in daily life within a few decades, especially through mechanization and urbanization [1].","promptFormat":"chat-template","retrievedTitles":["Industrial Revolution","Agricultural revolution"],"expectedKbTitles":["Agricultural revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791241901519} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":36718.25388401747,"generationLatencyMs":18597.464568018913,"totalLatencyMs":55708.17047804594,"tokensGenerated":93,"tokPerSec":5.000681660656433,"peakRssBytes":2847948800,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":391.97428399324417,"contextTokensBefore":330,"contextTokensAfter":298,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-2","category":"synthesis","query":"How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism.","answer":"The immune system detects pathogens and mounts a response by identifying foreign molecules, activating innate and adaptive defenses, and creating memory cells for future responses. A vaccine mimics this by introducing a harmless version of a pathogen (like a weakened or inactivated form or a surface protein), which triggers the adaptive immune system to recognize and respond to that specific pathogen. This creates memory cells that enable a faster, stronger response if the real pathogen is encountered later [2].","promptFormat":"chat-template","retrievedTitles":["Immune system","Vaccine"],"expectedKbTitles":["Immune system","Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791241957368} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":36237.69687604904,"generationLatencyMs":15343.156664967537,"totalLatencyMs":52953.36232995987,"tokensGenerated":73,"tokPerSec":4.757821457085047,"peakRssBytes":2929418240,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":1371.7060559988022,"contextTokensBefore":310,"contextTokensAfter":216,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-3","category":"synthesis","query":"Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species?","answer":"The Amazon rainforest is crucial for global climate regulation because it absorbs significant amounts of CO₂ and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens ecosystem resilience, disrupts climate feedback loops (e.g., reduced carbon sequestration), and compromises vital services like water cycling and soil fertility, which affect both regional and global environmental stability.","promptFormat":"chat-template","retrievedTitles":["Biodiversity loss","Amazon rainforest"],"expectedKbTitles":["Amazon rainforest"],"expectedKbHit":true,"timedOut":false,"createdAt":1791242010480} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"chat","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":23016.804820001125,"generationLatencyMs":10712.50659698248,"totalLatencyMs":34265.84889400005,"tokensGenerated":58,"tokPerSec":5.41423237175299,"peakRssBytes":2828165120,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":535.2910559773445,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-1","category":"reasoning","query":"A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive?","answer":"Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. \nArrival time: 6:15 pm.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791242045046} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"calculate","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":11504.366338014603,"generationLatencyMs":20001.079966008663,"totalLatencyMs":31506.014827013016,"tokensGenerated":94,"tokPerSec":4.699746221691561,"peakRssBytes":2824630272,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-2","category":"reasoning","query":"A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM?","answer":"Total used: 2GB (OS/app) + 200MB (embedding) + 1.5GB (KV cache) = 3.7GB. \nRemaining: 12GB - 3.7GB = 8.3GB. \nA 9GB model would exceed the available RAM and not fit if loaded fully. \nAnswer: 8.3GB left; no, 9GB model does not fit.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791242076730} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9741.85823994875,"generationLatencyMs":16057.01986503601,"totalLatencyMs":26267.168021976948,"tokensGenerated":76,"tokPerSec":4.73313233955008,"peakRssBytes":2834415616,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":467.4961349964142,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-3","category":"reasoning","query":"If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure?","answer":"Minimum disk footprint: 8B × 1 byte = 8 bytes per token. \nRAM usage doesn’t scale with 100B because only 8B per token are activated at inference—only the selected expert parameters are computed, not the full model. The total parameter count (100B) is a design choice for capacity, not runtime usage.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791242103195} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":72154.55468201637,"generationLatencyMs":14104.012579977512,"totalLatencyMs":86639.98425400257,"tokensGenerated":68,"tokPerSec":4.8213229826903925,"peakRssBytes":2887237632,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":380.9620360136032,"contextTokensBefore":396,"contextTokensAfter":359,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-1","category":"retrieval-grounded","query":"Why did the Western Roman Empire fall?","answer":"The Western Roman Empire fell due to a combination of factors including military weakening, economic decline, population decline, political instability, and external invasions by barbarian tribes. Climatic changes and widespread diseases also contributed to the collapse [1]. The Eastern Roman (Byzantine) Empire continued to thrive until 1453 [2].","promptFormat":"chat-template","retrievedTitles":["Fall of the Western Roman Empire","Byzantine Empire"],"expectedKbTitles":["Fall of the Western Roman Empire"],"expectedKbHit":true,"timedOut":false,"createdAt":1791242190015} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":18070.497444987297,"generationLatencyMs":13235.349452972412,"totalLatencyMs":31798.577244997025,"tokensGenerated":61,"tokPerSec":4.608869619706228,"peakRssBytes":2895564800,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":492.27783197164536,"contextTokensBefore":155,"contextTokensAfter":121,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-2","category":"retrieval-grounded","query":"What is a black hole and how does one form?","answer":"A black hole is an astronomical object so dense that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, compressing its mass into an extremely small volume, creating a region where spacetime curvature becomes extreme.","promptFormat":"chat-template","retrievedTitles":["Black hole"],"expectedKbTitles":["Black hole"],"expectedKbHit":true,"timedOut":false,"createdAt":1791242222047} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":20929.750986993313,"generationLatencyMs":16397.151537954807,"totalLatencyMs":37720.30775797367,"tokensGenerated":77,"tokPerSec":4.695937573167302,"peakRssBytes":2896564224,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":393.05497205257416,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-1","category":"no-kb-content","query":"Who was Napoleon Bonaparte?","answer":"Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He led France through a series of military campaigns across Europe, significantly shaping the political landscape of the continent, and later ruled as a dominant figure in European history until his exile in 1815.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791242259955} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":47560.44570201635,"generationLatencyMs":13518.014329016209,"totalLatencyMs":61535.68268799782,"tokensGenerated":62,"tokPerSec":4.586472427900743,"peakRssBytes":2886057984,"outcome":"success","runId":"eval-2026-10-05T23-07-42-918Z","evalSetVersion":"1","retrievalMs":456.7542319893837,"contextTokensBefore":175,"contextTokensAfter":147,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-2","category":"no-kb-content","query":"How do antibiotics work, and why does antibiotic resistance develop?","answer":"Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis or protein production, to kill or inhibit their growth [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to misuse or overuse in medicine and agriculture.","promptFormat":"chat-template","retrievedTitles":["Antimicrobial resistance"],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791242321742} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/final-4b-std/eval-2026-10-05T23-07-42-918Z.report.md b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-std/eval-2026-10-05T23-07-42-918Z.report.md new file mode 100644 index 000000000..44e4395fb --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-std/eval-2026-10-05T23-07-42-918Z.report.md @@ -0,0 +1,17 @@ +BOAR Device Evaluation — eval-2026-10-05T23-07-42-918Z (set v1, 17 queries) +──────────────────────────────────────────────────────────── + +Qwen3-4B-Instruct-2507 (Q4_K_M) + Config: model:qwen3-4b-instruct-2507-q4km · prompt chat-template + Queries: 17 ok · 0 failed · 0 cancelled (of 17) + TTFT: avg 25.4s · p50 23.0s + Model load: avg 4.3s · p50 4.3s over 1 load(s) + Generation: avg 12.6s · p50 14.1s + Tokens/sec: avg 5.5 · p50 5.0 + Search time: avg 0.5s · p50 0.4s (before the model starts) + Context tokens: 253 → 211 avg (trimmed to the answering sentences) + Total per query: avg 38.6s · p50 34.3s + Peak RSS: 2.86 GB + Residency: cold 0 · switched 1 · resident 16 + Model switches: 0 within plans · 1 across messages + Retrieval: used 11/17 · expected KB article found 10/10 diff --git a/docs/evidence/2026-10-05-android-conversations/harness/final-4b-std/eval-2026-10-05T23-07-42-918Z.status.json b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-std/eval-2026-10-05T23-07-42-918Z.status.json new file mode 100644 index 000000000..9087e7d23 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-std/eval-2026-10-05T23-07-42-918Z.status.json @@ -0,0 +1,17 @@ +{ + "requestId": "req-20261005t230739-i8tl", + "state": "done", + "configs": [ + "model:qwen3-4b-instruct-2507-q4km" + ], + "total": 17, + "completed": 17, + "runId": "eval-2026-10-05T23-07-42-918Z", + "resultPath": "files/eval/eval-2026-10-05T23-07-42-918Z.jsonl", + "stopped": false, + "installedModels": [ + "qwen3-4b-instruct-2507-q4km", + "qwen2.5-1.5b-instruct-q4km" + ], + "updatedAt": 1791242329186 +} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/final-4b-vitalik/eval-2026-10-05T23-39-41-470Z.answers.md b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-vitalik/eval-2026-10-05T23-39-41-470Z.answers.md new file mode 100644 index 000000000..7afe3efa7 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-vitalik/eval-2026-10-05T23-39-41-470Z.answers.md @@ -0,0 +1,50 @@ +## vitalik-about-x (vitalik) + +> Tell me about X.com. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +X.com was founded in 1999 by Elon Musk as a financial services company, primarily focused on online banking and payments. It was later rebranded as "X" in 2023, becoming the social media platform known today as X, formerly known as Twitter. The platform is now a major social media network where users share text, images, videos, and links. + +## vitalik-zk-snark (vitalik) + +> How does a zk-SNARK differ from a zk-STARK? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +zk-SNARKs and zk-STARKs are both zero-knowledge proof systems that allow one party to prove knowledge of a secret without revealing it. zk-SNARKs (Zero-Knowledge Succinct Non-Interactive Arguments of Knowledge) are more compact and efficient but require a trusted setup, while zk-STARKs (Zero-Knowledge Scalable Transparent Arguments of Knowledge) are transparent, do not require a trusted setup, and offer stronger security guarantees with better scalability. + +## vitalik-quadratic-funding (vitalik) + +> What is quadratic funding and what are its weaknesses? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Quadratic funding is a mechanism in decentralized funding where contributions to projects are scaled based on the number of contributors, encouraging diversity and reducing coordination costs. It allocates more funds to projects with broader participation, as the funding grows quadratically with the number of backers. + +Weaknesses include potential inefficiency in funding highly popular projects that may not be well-designed, and the risk of "winner-takes-most" dynamics where a few projects dominate due to network effects. It can also underfund smaller, niche projects if they lack sufficient participation. + +## vitalik-far-uvc (vitalik) + +> Does far-UVC light at 222 nm harm human skin? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Far-UVC light at 222 nm does not significantly harm human skin because it is largely absorbed by the outer dead layers of the skin and does not penetrate deeply into living tissue. Studies suggest it is safe for human exposure at low to moderate doses, as it primarily affects microorganisms without damaging human cells. + +## vitalik-zuzalu (vitalik) + +> What was Zuzalu? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Zuzalu was a short-lived, experimental internet-based social networking platform launched in the early 2000s, primarily known for its focus on user-generated content and community-driven interactions. It quickly faded from relevance due to technical limitations, lack of user engagement, and competition from more established platforms. No major public records or long-term impact remain linked to it. + +## vitalik-vegan-buenos-aires (vitalik) + +> Where can I find vegan food in Buenos Aires? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +In Buenos Aires, you can find vegan food at places like El Pescador (vegan options), La Vida de la Sopa, and Vegan Bistro. Many cafés in neighborhoods like Palermo and Recoleta also offer plant-based meals. Check local vegan food blogs or apps like HappyCow for updated listings. + diff --git a/docs/evidence/2026-10-05-android-conversations/harness/final-4b-vitalik/eval-2026-10-05T23-39-41-470Z.jsonl b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-vitalik/eval-2026-10-05T23-39-41-470Z.jsonl new file mode 100644 index 000000000..20a7274bd --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-vitalik/eval-2026-10-05T23-39-41-470Z.jsonl @@ -0,0 +1,6 @@ +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"chat","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":true,"modelResidency":"switched","modelLoadMs":4222.8620619773865,"ttftMs":15118.734179019928,"generationLatencyMs":10561.639286994934,"totalLatencyMs":30254.096979022026,"tokensGenerated":80,"tokPerSec":7.574581731692724,"peakRssBytes":3033673728,"outcome":"success","runId":"eval-2026-10-05T23-39-41-470Z","evalSetVersion":"vitalik-1","retrievalMs":350.1076250076294,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-about-x","category":"vitalik","query":"Tell me about X.com.","answer":"X.com was founded in 1999 by Elon Musk as a financial services company, primarily focused on online banking and payments. It was later rebranded as \"X\" in 2023, becoming the social media platform known today as X, formerly known as Twitter. The platform is now a major social media network where users share text, images, videos, and links.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["X (social network)"],"expectedKbHit":false,"timedOut":false,"createdAt":1791243611786} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":1667.131469964981,"generationLatencyMs":14828.398689031601,"totalLatencyMs":16790.934538006783,"tokensGenerated":95,"tokPerSec":6.406625691165859,"peakRssBytes":3047571456,"outcome":"success","runId":"eval-2026-10-05T23-39-41-470Z","evalSetVersion":"vitalik-1","retrievalMs":295.02445501089096,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-zk-snark","category":"vitalik","query":"How does a zk-SNARK differ from a zk-STARK?","answer":"zk-SNARKs and zk-STARKs are both zero-knowledge proof systems that allow one party to prove knowledge of a secret without revealing it. zk-SNARKs (Zero-Knowledge Succinct Non-Interactive Arguments of Knowledge) are more compact and efficient but require a trusted setup, while zk-STARKs (Zero-Knowledge Scalable Transparent Arguments of Knowledge) are transparent, do not require a trusted setup, and offer stronger security guarantees with better scalability.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Zero-knowledge proof","Non-interactive zero-knowledge proof"],"expectedKbHit":false,"timedOut":false,"createdAt":1791243628697} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":1546.7753499746323,"generationLatencyMs":17319.57658803463,"totalLatencyMs":19129.050790011883,"tokensGenerated":107,"tokPerSec":6.177980128793785,"peakRssBytes":3038425088,"outcome":"success","runId":"eval-2026-10-05T23-39-41-470Z","evalSetVersion":"vitalik-1","retrievalMs":262.3242189884186,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-quadratic-funding","category":"vitalik","query":"What is quadratic funding and what are its weaknesses?","answer":"Quadratic funding is a mechanism in decentralized funding where contributions to projects are scaled based on the number of contributors, encouraging diversity and reducing coordination costs. It allocates more funds to projects with broader participation, as the funding grows quadratically with the number of backers.\n\nWeaknesses include potential inefficiency in funding highly popular projects that may not be well-designed, and the risk of \"winner-takes-most\" dynamics where a few projects dominate due to network effects. It can also underfund smaller, niche projects if they lack sufficient participation.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Quadratic funding","Quadratic voting"],"expectedKbHit":false,"timedOut":false,"createdAt":1791243647893} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":2123.164958000183,"generationLatencyMs":10122.200038015842,"totalLatencyMs":12498.058395981789,"tokensGenerated":63,"tokPerSec":6.2239433881361315,"peakRssBytes":3040813056,"outcome":"success","runId":"eval-2026-10-05T23-39-41-470Z","evalSetVersion":"vitalik-1","retrievalMs":251.7274169921875,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-far-uvc","category":"vitalik","query":"Does far-UVC light at 222 nm harm human skin?","answer":"Far-UVC light at 222 nm does not significantly harm human skin because it is largely absorbed by the outer dead layers of the skin and does not penetrate deeply into living tissue. Studies suggest it is safe for human exposure at low to moderate doses, as it primarily affects microorganisms without damaging human cells.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Far-UVC","Ultraviolet germicidal irradiation"],"expectedKbHit":false,"timedOut":false,"createdAt":1791243660488} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":1156.8937579989433,"generationLatencyMs":11924.159186005592,"totalLatencyMs":13385.353806972504,"tokensGenerated":74,"tokPerSec":6.205888301696587,"peakRssBytes":3096952832,"outcome":"success","runId":"eval-2026-10-05T23-39-41-470Z","evalSetVersion":"vitalik-1","retrievalMs":303.97753900289536,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-zuzalu","category":"vitalik","query":"What was Zuzalu?","answer":"Zuzalu was a short-lived, experimental internet-based social networking platform launched in the early 2000s, primarily known for its focus on user-generated content and community-driven interactions. It quickly faded from relevance due to technical limitations, lack of user engagement, and competition from more established platforms. No major public records or long-term impact remain linked to it.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Zuzalu","Network state"],"expectedKbHit":false,"timedOut":false,"createdAt":1791243674308} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":1701.517782986164,"generationLatencyMs":10759.430099010468,"totalLatencyMs":12702.056443035603,"tokensGenerated":65,"tokPerSec":6.041212164757497,"peakRssBytes":3086950400,"outcome":"success","runId":"eval-2026-10-05T23-39-41-470Z","evalSetVersion":"vitalik-1","retrievalMs":240.78613299131393,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-vegan-buenos-aires","category":"vitalik","query":"Where can I find vegan food in Buenos Aires?","answer":"In Buenos Aires, you can find vegan food at places like El Pescador (vegan options), La Vida de la Sopa, and Vegan Bistro. Many cafés in neighborhoods like Palermo and Recoleta also offer plant-based meals. Check local vegan food blogs or apps like HappyCow for updated listings.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Buenos Aires","Veganism"],"expectedKbHit":false,"timedOut":false,"createdAt":1791243687086} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/final-4b-vitalik/eval-2026-10-05T23-39-41-470Z.report.md b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-vitalik/eval-2026-10-05T23-39-41-470Z.report.md new file mode 100644 index 000000000..a59064b74 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-vitalik/eval-2026-10-05T23-39-41-470Z.report.md @@ -0,0 +1,15 @@ +BOAR Device Evaluation — eval-2026-10-05T23-39-41-470Z (set vvitalik-1, 6 queries) +──────────────────────────────────────────────────────────── + +Qwen3-4B-Instruct-2507 (Q4_K_M) + Config: model:qwen3-4b-instruct-2507-q4km · prompt chat-template + Queries: 6 ok · 0 failed · 0 cancelled (of 6) + TTFT: avg 3.9s · p50 1.7s + Model load: avg 4.2s · p50 4.2s over 1 load(s) + Generation: avg 12.6s · p50 11.3s + Tokens/sec: avg 6.4 · p50 6.2 + Total per query: avg 17.5s · p50 15.1s + Peak RSS: 2.88 GB + Residency: cold 0 · switched 1 · resident 5 + Model switches: 0 within plans · 1 across messages + Retrieval: used 0/6 · expected KB article found 0/6 diff --git a/docs/evidence/2026-10-05-android-conversations/harness/final-4b-vitalik/eval-2026-10-05T23-39-41-470Z.status.json b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-vitalik/eval-2026-10-05T23-39-41-470Z.status.json new file mode 100644 index 000000000..53d23e965 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/final-4b-vitalik/eval-2026-10-05T23-39-41-470Z.status.json @@ -0,0 +1,17 @@ +{ + "requestId": "req-20261005t233937-3fjh", + "state": "done", + "configs": [ + "model:qwen3-4b-instruct-2507-q4km" + ], + "total": 6, + "completed": 6, + "runId": "eval-2026-10-05T23-39-41-470Z", + "resultPath": "files/eval/eval-2026-10-05T23-39-41-470Z.jsonl", + "stopped": false, + "installedModels": [ + "qwen3-4b-instruct-2507-q4km", + "qwen2.5-1.5b-instruct-q4km" + ], + "updatedAt": 1791243691232 +} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std-rerun/eval-2026-10-05T04-25-19-646Z.answers.md b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std-rerun/eval-2026-10-05T04-25-19-646Z.answers.md new file mode 100644 index 000000000..388b2fa83 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std-rerun/eval-2026-10-05T04-25-19-646Z.answers.md @@ -0,0 +1,140 @@ +## greeting-1 (greeting) + +> hey, what's up? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Not much, just running smoothly on the back end. How can I help you today? + +## factual-1 (factual) + +> What is the capital of Australia? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +The capital of Australia is Canberra. + +## factual-2 (factual) + +> Who proposed the theory of evolution by natural selection? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Evolution + +Charles Darwin proposed the theory of evolution by natural selection. [1] + +## explanation-1 (explanation) + +> How do vaccines work? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Vaccine + +Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed microorganism, or a surface protein—into the body, which triggers the immune system to produce antibodies and memory cells. This prepares the body to recognize and fight the actual disease if exposed later [1]. + +## explanation-2 (explanation) + +> Explain photosynthesis in simple terms. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Photosynthesis + +Photosynthesis is how plants use sunlight to turn carbon dioxide and water into sugar (food) and oxygen as a byproduct. The sunlight powers the process, which happens in the leaves, and the oxygen is released into the air [1]. + +## comparison-1 (comparison) + +> Compare the French Revolution and the Industrial Revolution. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: French Revolution, Industrial Revolution + +The French Revolution (1789–1799) was a political and social upheaval that transformed France’s government and established principles of liberty, equality, and fraternity, influencing modern democracies [1]. The Industrial Revolution (beginning c. 1760) was a period of technological and economic transformation that shifted societies from agrarian economies to industrialized manufacturing, fundamentally altering global economic structures [2]. While the French Revolution changed political ideologies, the Industrial Revolution transformed material and economic life worldwide. + +## comparison-2 (comparison) + +> Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Supply and demand, Behavioral economics + +The supply and demand model assumes rational, informed buyers and sellers who act to maximize utility, leading to equilibrium prices where quantity supplied equals quantity demanded [1]. Behavioral economics complicates this by showing that individuals often make irrational, emotionally driven, or bounded-rational decisions—such as anchoring, loss aversion, or herd behavior—that distort market outcomes and cause prices to deviate from equilibrium [2]. The simple model breaks down when explaining persistent price bubbles, sudden market crashes, or irrational consumer choices that defy traditional demand curves. + +## synthesis-1 (synthesis) + +> What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Industrial Revolution, Agricultural revolution + +Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society, transitioning from subsistence-based lifestyles to more productive, stable systems of food and manufacturing [1]. The Agricultural Revolution (starting ~10,000 BCE) transformed human life by enabling settled farming, population growth, and the rise of civilizations, while the Industrial Revolution (starting ~1760) drastically altered daily life through mechanization, urbanization, and mass production. The Industrial Revolution changed daily life more rapidly and dramatically than the Agricultural Revolution, shifting societal structures, work patterns, and living conditions within a few generations. + +## synthesis-2 (synthesis) + +> How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Immune system, Vaccine + +The immune system detects pathogens and mounts a response by identifying foreign molecules, triggering defense mechanisms like inflammation and cell destruction [1]. A vaccine mimics this process by introducing a harmless version of a pathogen—such as a weakened or inactivated microbe or a protein fragment—so the immune system learns to recognize and respond to it without causing disease. This creates memory cells that enable a faster, stronger response if the real pathogen is later encountered [2]. + +## synthesis-3 (synthesis) + +> Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Biodiversity loss, Amazon rainforest + +The Amazon rainforest is crucial for global climate regulation because it absorbs vast amounts of carbon dioxide and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens ecosystem stability, disrupts climate regulation, reduces resilience to environmental changes, and undermines vital services like water cycling and soil fertility—many of which are essential for both local and global environmental health. + +## reasoning-1 (reasoning) + +> A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. +The train arrives at 6:15 pm. + +## reasoning-2 (reasoning) + +> A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Total used: OS/app (2GB) + embedding model (0.2GB) + KV cache (1.5GB) = 3.7GB. +Remaining for LLM weights: 12GB - 3.7GB = 8.3GB. +A 9GB model would exceed the available RAM and not fit when loaded fully. + +## reasoning-3 (reasoning) + +> If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Minimum disk footprint: 8B × 1 byte = 8 bytes per token. +RAM usage doesn't scale with the 100B total because only 8B per token are activated at runtime (sparse activation), and parameters are not stored in memory simultaneously—only the selected expert weights are loaded and processed. + +## grounded-1 (retrieval-grounded) + +> Why did the Western Roman Empire fall? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Fall of the Western Roman Empire, Byzantine Empire + +The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, political instability, and invasions by external groups like the Visigoths and Vandals. Internal issues such as poor leadership, population decline, and disease, along with climatic changes, exacerbated the situation [1]. The empire lost its ability to maintain control over its provinces, leading to fragmentation and eventual collapse in the 5th century. + +## grounded-2 (retrieval-grounded) + +> What is a black hole and how does one form? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Black hole + +A black hole is an astronomical object so compact that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, compressing its mass into an extremely small volume, creating a region where spacetime is severely curved. + +## no-kb-1 (no-kb-content) + +> Who was Napoleon Bonaparte? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He led France in a series of successful military campaigns across Europe, significantly reshaping the political landscape of the continent, and is known for his legal reforms, including the Napoleonic Code. + +## no-kb-2 (no-kb-content) + +> How do antibiotics work, and why does antibiotic resistance develop? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Antimicrobial resistance + +Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis, protein production, or DNA replication, disrupting their ability to survive and multiply [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to misuse or overuse of these drugs, which selects for resistant strains in populations. + diff --git a/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std-rerun/eval-2026-10-05T04-25-19-646Z.jsonl b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std-rerun/eval-2026-10-05T04-25-19-646Z.jsonl new file mode 100644 index 000000000..67e808784 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std-rerun/eval-2026-10-05T04-25-19-646Z.jsonl @@ -0,0 +1,17 @@ +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"greeting","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9369.112471997738,"generationLatencyMs":3254.546794950962,"totalLatencyMs":12634.93522799015,"tokensGenerated":18,"tokPerSec":5.530723979118947,"peakRssBytes":2677043200,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"greeting-1","category":"greeting","query":"hey, what's up?","answer":"Not much, just running smoothly on the back end. How can I help you today?","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791174332378} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":515.1320390105247,"generationLatencyMs":887.4721279740334,"totalLatencyMs":1811.9172770380974,"tokensGenerated":7,"tokPerSec":7.887571653635993,"peakRssBytes":2795524096,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":408.7694099545479,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-1","category":"factual","query":"What is the capital of Australia?","answer":"The capital of Australia is Canberra.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791174334393} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9082.490685999393,"generationLatencyMs":2084.4692800045013,"totalLatencyMs":11406.805832982063,"tokensGenerated":14,"tokPerSec":6.716337887200606,"peakRssBytes":2854424576,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":239.5409340262413,"contextTokensBefore":111,"contextTokensAfter":111,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-2","category":"factual","query":"Who proposed the theory of evolution by natural selection?","answer":"Charles Darwin proposed the theory of evolution by natural selection. [1]","promptFormat":"chat-template","retrievedTitles":["Evolution"],"expectedKbTitles":["Evolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791174345853} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":3305.2833679914474,"generationLatencyMs":10380.72941595316,"totalLatencyMs":13891.33142799139,"tokensGenerated":62,"tokPerSec":5.972605345508581,"peakRssBytes":2867654656,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":205.0589599609375,"contextTokensBefore":154,"contextTokensAfter":102,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-1","category":"explanation","query":"How do vaccines work?","answer":"Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed microorganism, or a surface protein—into the body, which triggers the immune system to produce antibodies and memory cells. This prepares the body to recognize and fight the actual disease if exposed later [1].","promptFormat":"chat-template","retrievedTitles":["Vaccine"],"expectedKbTitles":["Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791174359805} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":6710.622642993927,"generationLatencyMs":8100.927941977978,"totalLatencyMs":15066.868414998055,"tokensGenerated":48,"tokPerSec":5.925247125242296,"peakRssBytes":2888142848,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":254.9776200056076,"contextTokensBefore":372,"contextTokensAfter":181,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-2","category":"explanation","query":"Explain photosynthesis in simple terms.","answer":"Photosynthesis is how plants use sunlight to turn carbon dioxide and water into sugar (food) and oxygen as a byproduct. The sunlight powers the process, which happens in the leaves, and the oxygen is released into the air [1].","promptFormat":"chat-template","retrievedTitles":["Photosynthesis"],"expectedKbTitles":["Photosynthesis"],"expectedKbHit":true,"timedOut":false,"createdAt":1791174374955} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"compare","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9678.09310400486,"generationLatencyMs":19423.23951101303,"totalLatencyMs":29464.727024018764,"tokensGenerated":104,"tokPerSec":5.3544106245012175,"peakRssBytes":2918711296,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":363.1490889787674,"contextTokensBefore":286,"contextTokensAfter":286,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-1","category":"comparison","query":"Compare the French Revolution and the Industrial Revolution.","answer":"The French Revolution (1789–1799) was a political and social upheaval that transformed France’s government and established principles of liberty, equality, and fraternity, influencing modern democracies [1]. The Industrial Revolution (beginning c. 1760) was a period of technological and economic transformation that shifted societies from agrarian economies to industrialized manufacturing, fundamentally altering global economic structures [2]. While the French Revolution changed political ideologies, the Industrial Revolution transformed material and economic life worldwide.","promptFormat":"chat-template","retrievedTitles":["French Revolution","Industrial Revolution"],"expectedKbTitles":["French Revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791174404502} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":7894.117556989193,"generationLatencyMs":18999.12220197916,"totalLatencyMs":27186.590223014355,"tokensGenerated":105,"tokPerSec":5.526571116483583,"peakRssBytes":2932064256,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":292.90860998630524,"contextTokensBefore":188,"contextTokensAfter":188,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-2","category":"comparison","query":"Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down?","answer":"The supply and demand model assumes rational, informed buyers and sellers who act to maximize utility, leading to equilibrium prices where quantity supplied equals quantity demanded [1]. Behavioral economics complicates this by showing that individuals often make irrational, emotionally driven, or bounded-rational decisions—such as anchoring, loss aversion, or herd behavior—that distort market outcomes and cause prices to deviate from equilibrium [2]. The simple model breaks down when explaining persistent price bubbles, sudden market crashes, or irrational consumer choices that defy traditional demand curves.","promptFormat":"chat-template","retrievedTitles":["Supply and demand","Behavioral economics"],"expectedKbTitles":["Supply and demand","Behavioral economics"],"expectedKbHit":true,"timedOut":false,"createdAt":1791174431763} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":12320.78487700224,"generationLatencyMs":27447.59282797575,"totalLatencyMs":40409.1975299716,"tokensGenerated":121,"tokPerSec":4.40840115773911,"peakRssBytes":3008323584,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":640.3153480291367,"contextTokensBefore":308,"contextTokensAfter":307,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-1","category":"synthesis","query":"What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life?","answer":"Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society, transitioning from subsistence-based lifestyles to more productive, stable systems of food and manufacturing [1]. The Agricultural Revolution (starting ~10,000 BCE) transformed human life by enabling settled farming, population growth, and the rise of civilizations, while the Industrial Revolution (starting ~1760) drastically altered daily life through mechanization, urbanization, and mass production. The Industrial Revolution changed daily life more rapidly and dramatically than the Agricultural Revolution, shifting societal structures, work patterns, and living conditions within a few generations.","promptFormat":"chat-template","retrievedTitles":["Industrial Revolution","Agricultural revolution"],"expectedKbTitles":["Agricultural revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791174472259} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":14622.823289990425,"generationLatencyMs":19939.24488198757,"totalLatencyMs":34957.03102296591,"tokensGenerated":92,"tokPerSec":4.614016255104507,"peakRssBytes":3005517824,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":394.3684089779854,"contextTokensBefore":330,"contextTokensAfter":298,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-2","category":"synthesis","query":"How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism.","answer":"The immune system detects pathogens and mounts a response by identifying foreign molecules, triggering defense mechanisms like inflammation and cell destruction [1]. A vaccine mimics this process by introducing a harmless version of a pathogen—such as a weakened or inactivated microbe or a protein fragment—so the immune system learns to recognize and respond to it without causing disease. This creates memory cells that enable a faster, stronger response if the real pathogen is later encountered [2].","promptFormat":"chat-template","retrievedTitles":["Immune system","Vaccine"],"expectedKbTitles":["Immune system","Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791174507361} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":13571.622159004211,"generationLatencyMs":15351.709275960922,"totalLatencyMs":29347.82171100378,"tokensGenerated":73,"tokPerSec":4.755170820900701,"peakRssBytes":3012403200,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":423.99800699949265,"contextTokensBefore":310,"contextTokensAfter":216,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-3","category":"synthesis","query":"Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species?","answer":"The Amazon rainforest is crucial for global climate regulation because it absorbs vast amounts of carbon dioxide and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens ecosystem stability, disrupts climate regulation, reduces resilience to environmental changes, and undermines vital services like water cycling and soil fertility—many of which are essential for both local and global environmental health.","promptFormat":"chat-template","retrievedTitles":["Biodiversity loss","Amazon rainforest"],"expectedKbTitles":["Amazon rainforest"],"expectedKbHit":true,"timedOut":false,"createdAt":1791174536856} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"chat","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9091.400273025036,"generationLatencyMs":10710.642786026001,"totalLatencyMs":20207.663380026817,"tokensGenerated":58,"tokPerSec":5.415174528616681,"peakRssBytes":3017535488,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":404.97530096769333,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-1","category":"reasoning","query":"A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive?","answer":"Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. \nThe train arrives at 6:15 pm.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791174557220} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"calculate","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":4661.497846007347,"generationLatencyMs":16711.135464966297,"totalLatencyMs":21373.176890015602,"tokensGenerated":76,"tokPerSec":4.547865712615913,"peakRssBytes":3009425408,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-2","category":"reasoning","query":"A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM?","answer":"Total used: OS/app (2GB) + embedding model (0.2GB) + KV cache (1.5GB) = 3.7GB. \nRemaining for LLM weights: 12GB - 3.7GB = 8.3GB. \nA 9GB model would exceed the available RAM and not fit when loaded fully.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791174578754} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":4054.8355330228806,"generationLatencyMs":14755.175544023514,"totalLatencyMs":19264.28081202507,"tokensGenerated":66,"tokPerSec":4.473006763157953,"peakRssBytes":3019108352,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":453.4828289747238,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-3","category":"reasoning","query":"If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure?","answer":"Minimum disk footprint: 8B × 1 byte = 8 bytes per token. \nRAM usage doesn't scale with the 100B total because only 8B per token are activated at runtime (sparse activation), and parameters are not stored in memory simultaneously—only the selected expert weights are loaded and processed.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791174598187} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":29101.656670987606,"generationLatencyMs":20813.523407042027,"totalLatencyMs":50300.68308997154,"tokensGenerated":86,"tokPerSec":4.1319289539849295,"peakRssBytes":3057799168,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":383.0218099951744,"contextTokensBefore":396,"contextTokensAfter":359,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-1","category":"retrieval-grounded","query":"Why did the Western Roman Empire fall?","answer":"The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, political instability, and invasions by external groups like the Visigoths and Vandals. Internal issues such as poor leadership, population decline, and disease, along with climatic changes, exacerbated the situation [1]. The empire lost its ability to maintain control over its provinces, leading to fragmentation and eventual collapse in the 5th century.","promptFormat":"chat-template","retrievedTitles":["Fall of the Western Roman Empire","Byzantine Empire"],"expectedKbTitles":["Fall of the Western Roman Empire"],"expectedKbHit":true,"timedOut":false,"createdAt":1791174648680} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":6685.446413993835,"generationLatencyMs":13465.068325042725,"totalLatencyMs":20568.519867002964,"tokensGenerated":61,"tokPerSec":4.530240658827585,"peakRssBytes":3062116352,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":417.5419110059738,"contextTokensBefore":155,"contextTokensAfter":121,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-2","category":"retrieval-grounded","query":"What is a black hole and how does one form?","answer":"A black hole is an astronomical object so compact that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, compressing its mass into an extremely small volume, creating a region where spacetime is severely curved.","promptFormat":"chat-template","retrievedTitles":["Black hole"],"expectedKbTitles":["Black hole"],"expectedKbHit":true,"timedOut":false,"createdAt":1791174669463} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":23111.104746997356,"generationLatencyMs":14702.77210199833,"totalLatencyMs":38989.993671000004,"tokensGenerated":74,"tokPerSec":5.033064478360668,"peakRssBytes":3065679872,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":1175.6134039759636,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-1","category":"no-kb-content","query":"Who was Napoleon Bonaparte?","answer":"Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He led France in a series of successful military campaigns across Europe, significantly reshaping the political landscape of the continent, and is known for his legal reforms, including the Napoleonic Code.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791174709091} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":18831.76128000021,"generationLatencyMs":16674.05278402567,"totalLatencyMs":35905.95676404238,"tokensGenerated":74,"tokPerSec":4.4380332099520885,"peakRssBytes":3025326080,"outcome":"success","runId":"eval-2026-10-05T04-25-19-646Z","evalSetVersion":"1","retrievalMs":399.72680699825287,"contextTokensBefore":175,"contextTokensAfter":147,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-2","category":"no-kb-content","query":"How do antibiotics work, and why does antibiotic resistance develop?","answer":"Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis, protein production, or DNA replication, disrupting their ability to survive and multiply [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to misuse or overuse of these drugs, which selects for resistant strains in populations.","promptFormat":"chat-template","retrievedTitles":["Antimicrobial resistance"],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791174745183} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std-rerun/eval-2026-10-05T04-25-19-646Z.report.md b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std-rerun/eval-2026-10-05T04-25-19-646Z.report.md new file mode 100644 index 000000000..7c6cf614f --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std-rerun/eval-2026-10-05T04-25-19-646Z.report.md @@ -0,0 +1,17 @@ +BOAR Device Evaluation — eval-2026-10-05T04-25-19-646Z (set v1, 17 queries) +──────────────────────────────────────────────────────────── + +Qwen3-4B-Instruct-2507 (Q4_K_M) + Config: model:qwen3-4b-instruct-2507-q4km · prompt chat-template + Queries: 17 ok · 0 failed · 0 cancelled (of 17) + TTFT: avg 10.7s · p50 9.1s + Model load: none (already resident) + Generation: avg 13.7s · p50 14.8s + Tokens/sec: avg 5.3 · p50 5.0 + Search time: avg 0.4s · p50 0.4s (before the model starts) + Context tokens: 253 → 211 avg (trimmed to the answering sentences) + Total per query: avg 24.9s · p50 21.4s + Peak RSS: 2.86 GB + Residency: cold 0 · switched 0 · resident 17 + Model switches: 0 within plans · 0 across messages + Retrieval: used 11/17 · expected KB article found 10/10 diff --git a/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std-rerun/eval-2026-10-05T04-25-19-646Z.status.json b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std-rerun/eval-2026-10-05T04-25-19-646Z.status.json new file mode 100644 index 000000000..c3d4b9f76 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std-rerun/eval-2026-10-05T04-25-19-646Z.status.json @@ -0,0 +1,17 @@ +{ + "requestId": "req-20261005t042516-9bgr", + "state": "done", + "configs": [ + "model:qwen3-4b-instruct-2507-q4km" + ], + "total": 17, + "completed": 17, + "runId": "eval-2026-10-05T04-25-19-646Z", + "resultPath": "files/eval/eval-2026-10-05T04-25-19-646Z.jsonl", + "stopped": false, + "installedModels": [ + "qwen3-4b-instruct-2507-q4km", + "qwen2.5-1.5b-instruct-q4km" + ], + "updatedAt": 1791174745411 +} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std/eval-2026-10-05T03-06-56-189Z.answers.md b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std/eval-2026-10-05T03-06-56-189Z.answers.md new file mode 100644 index 000000000..927448368 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std/eval-2026-10-05T03-06-56-189Z.answers.md @@ -0,0 +1,142 @@ +## greeting-1 (greeting) + +> hey, what's up? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Hey! Not much, just here and ready to help. What's on your mind? + +## factual-1 (factual) + +> What is the capital of Australia? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +The capital of Australia is Canberra. + +## factual-2 (factual) + +> Who proposed the theory of evolution by natural selection? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Evolution + +Charles Darwin proposed the theory of evolution by natural selection. [1] + +## explanation-1 (explanation) + +> How do vaccines work? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Vaccine + +Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed microorganism, or a protein—into the body, which triggers the immune system to produce antibodies. This prepares the body to recognize and fight the actual disease if exposed in the future [1]. + +## explanation-2 (explanation) + +> Explain photosynthesis in simple terms. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Photosynthesis + +Photosynthesis is how plants use sunlight to turn carbon dioxide and water into sugar (food) and oxygen as a byproduct. The sunlight powers the process, which happens in the leaves, and the oxygen is released into the air [1]. + +## comparison-1 (comparison) + +> Compare the French Revolution and the Industrial Revolution. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: French Revolution, Industrial Revolution + +The French Revolution (1789–1799) was a political and social upheaval that reshaped France’s government and introduced ideals of liberty, equality, and fraternity, influencing modern democratic values [1]. The Industrial Revolution (starting c. 1760) was a transformative period in economic and technological development, shifting societies from agrarian to industrial economies and fundamentally altering production and urban life [2]. While the French Revolution changed political structures, the Industrial Revolution transformed how societies produced and lived. + +## comparison-2 (comparison) + +> Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Supply and demand, Behavioral economics + +The supply and demand model assumes rational, informed buyers and sellers who act to maximize utility, leading to equilibrium prices [1]. Behavioral economics complicates this by showing that individuals often make irrational, emotionally driven, or bounded-rational decisions—such as anchoring, loss aversion, or herd behavior—leading to price deviations from equilibrium [2]. The simple model breaks down when real-world factors like cognitive biases, information asymmetry, or social influences cause prices to persistently deviate from what supply and demand alone would predict. + +## synthesis-1 (synthesis) + +> What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Industrial Revolution, Agricultural revolution + +Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society, transitioning from nomadic lifestyles to settled, productive systems that enabled population growth and technological advancement [1]. The Agricultural Revolution transformed daily life by enabling food surplus and permanent settlements, while the Industrial Revolution rapidly changed daily life through mechanization, urbanization, and mass production—occurring over a few decades rather than centuries. The Industrial Revolution thus accelerated societal change much more quickly than the earlier Agricultural Revolution. + +## synthesis-2 (synthesis) + +> How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Immune system, Vaccine + +The immune system detects pathogens and mounts a response by identifying foreign molecules, triggering both innate and adaptive defenses. Vaccines mimic this process by introducing a harmless version of a pathogen—like a weakened or killed microbe or a surface protein—so the immune system learns to recognize and remember it. This prepares the body to respond rapidly and effectively if the real pathogen is encountered later [2]. The adaptive immune system produces memory B and T cells that respond faster and stronger upon future exposure, providing long-term protection. + +## synthesis-3 (synthesis) + +> Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Biodiversity loss, Amazon rainforest + +The Amazon rainforest is crucial for global climate because it absorbs significant amounts of carbon dioxide and regulates regional and global weather patterns through evaporation and cloud formation [1]. Biodiversity loss in the Amazon threatens not only species survival but also ecosystem resilience, such as the forest's ability to sequester carbon, maintain soil fertility, and support water cycles—disruptions of which could accelerate climate change and reduce the region’s capacity to recover from environmental stress. + +## reasoning-1 (reasoning) + +> A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. +Arrival time: 6:15 pm. + +## reasoning-2 (reasoning) + +> A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Total used: 2GB (OS/app) + 200MB (embedding) + 1.5GB (KV cache) = 3.2GB. +Remaining for LLM weights: 12GB – 3.2GB = 8.8GB. +Yes, a 9GB model would fit if loaded fully into RAM, as 9GB ≤ 8.8GB? No — 9GB > 8.8GB. +So, **no**, a 9GB model would not fit. + +## reasoning-3 (reasoning) + +> If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Minimum disk footprint: 8B per token × 1 byte/Q8 = 8 bytes per token. For example, at 1 token/sec, it's 8 bytes/sec, or ~8 MB/s sustained. + +RAM usage doesn’t scale with 100B because only 8B per token are activated (via gating), not stored or processed simultaneously. The model uses sparse, token-wise computation—only relevant experts are engaged per input, so active parameters are minimal. + +## grounded-1 (retrieval-grounded) + +> Why did the Western Roman Empire fall? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Fall of the Western Roman Empire, Byzantine Empire + +The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, political instability, internal conflicts, religious changes, and external pressures from invading tribes. Climatic changes and widespread diseases also contributed to the collapse [1]. The Eastern Roman (Byzantine) Empire survived until 1453, continuing the Roman tradition [2]. + +## grounded-2 (retrieval-grounded) + +> What is a black hole and how does one form? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Black hole + +A black hole is an astronomical object so dense that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, squeezing its mass into an extremely small volume, creating a region where spacetime is severely curved. + +## no-kb-1 (no-kb-content) + +> Who was Napoleon Bonaparte? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He expanded the French Empire across Europe, implementing legal and political reforms, and was a major figure in shaping modern European history. + +## no-kb-2 (no-kb-content) + +> How do antibiotics work, and why does antibiotic resistance develop? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Antimicrobial resistance + +Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis or protein production, to kill or inhibit their growth [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to misuse or overuse of these drugs, which selects for resistant strains. + diff --git a/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std/eval-2026-10-05T03-06-56-189Z.jsonl b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std/eval-2026-10-05T03-06-56-189Z.jsonl new file mode 100644 index 000000000..fe0e5f32c --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std/eval-2026-10-05T03-06-56-189Z.jsonl @@ -0,0 +1,17 @@ +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"greeting","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9181.10209596157,"generationLatencyMs":3920.376711010933,"totalLatencyMs":13102.665045022964,"tokensGenerated":18,"tokPerSec":4.591395502744533,"peakRssBytes":2698526720,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"greeting-1","category":"greeting","query":"hey, what's up?","answer":"Hey! Not much, just here and ready to help. What's on your mind?","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791169629360} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":573.0953779816628,"generationLatencyMs":891.5152180194855,"totalLatencyMs":1806.3688969612122,"tokensGenerated":7,"tokPerSec":7.8518009098606365,"peakRssBytes":2815262720,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":341.08174699544907,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-1","category":"factual","query":"What is the capital of Australia?","answer":"The capital of Australia is Canberra.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791169631352} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9137.593469977379,"generationLatencyMs":2099.8929449915886,"totalLatencyMs":11438.390507996082,"tokensGenerated":14,"tokPerSec":6.667006541162544,"peakRssBytes":2867326976,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":200.6013190150261,"contextTokensBefore":111,"contextTokensAfter":111,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-2","category":"factual","query":"Who proposed the theory of evolution by natural selection?","answer":"Charles Darwin proposed the theory of evolution by natural selection. [1]","promptFormat":"chat-template","retrievedTitles":["Evolution"],"expectedKbTitles":["Evolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791169642832} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":3298.144573032856,"generationLatencyMs":11024.50212097168,"totalLatencyMs":14614.946621954441,"tokensGenerated":60,"tokPerSec":5.442422645632519,"peakRssBytes":2880995328,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":291.88183599710464,"contextTokensBefore":154,"contextTokensAfter":102,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-1","category":"explanation","query":"How do vaccines work?","answer":"Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed microorganism, or a protein—into the body, which triggers the immune system to produce antibodies. This prepares the body to recognize and fight the actual disease if exposed in the future [1].","promptFormat":"chat-template","retrievedTitles":["Vaccine"],"expectedKbTitles":["Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791169657474} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":7009.08728402853,"generationLatencyMs":8262.491458952427,"totalLatencyMs":15617.150315999985,"tokensGenerated":48,"tokPerSec":5.809385732918598,"peakRssBytes":2922352640,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":345.3010659813881,"contextTokensBefore":372,"contextTokensAfter":181,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-2","category":"explanation","query":"Explain photosynthesis in simple terms.","answer":"Photosynthesis is how plants use sunlight to turn carbon dioxide and water into sugar (food) and oxygen as a byproduct. The sunlight powers the process, which happens in the leaves, and the oxygen is released into the air [1].","promptFormat":"chat-template","retrievedTitles":["Photosynthesis"],"expectedKbTitles":["Photosynthesis"],"expectedKbHit":true,"timedOut":false,"createdAt":1791169673473} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"compare","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":10200.784550964832,"generationLatencyMs":20283.187184035778,"totalLatencyMs":30764.976902008057,"tokensGenerated":104,"tokPerSec":5.127399311379177,"peakRssBytes":2939781120,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":280.7298169732094,"contextTokensBefore":286,"contextTokensAfter":286,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-1","category":"comparison","query":"Compare the French Revolution and the Industrial Revolution.","answer":"The French Revolution (1789–1799) was a political and social upheaval that reshaped France’s government and introduced ideals of liberty, equality, and fraternity, influencing modern democratic values [1]. The Industrial Revolution (starting c. 1760) was a transformative period in economic and technological development, shifting societies from agrarian to industrial economies and fundamentally altering production and urban life [2]. While the French Revolution changed political structures, the Industrial Revolution transformed how societies produced and lived.","promptFormat":"chat-template","retrievedTitles":["French Revolution","Industrial Revolution"],"expectedKbTitles":["French Revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791169701482} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":10765.652918994427,"generationLatencyMs":42454.12278300524,"totalLatencyMs":53539.2359880209,"tokensGenerated":105,"tokPerSec":2.4732580281233,"peakRssBytes":2938007552,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":318.6109219789505,"contextTokensBefore":188,"contextTokensAfter":188,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-2","category":"comparison","query":"Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down?","answer":"The supply and demand model assumes rational, informed buyers and sellers who act to maximize utility, leading to equilibrium prices [1]. Behavioral economics complicates this by showing that individuals often make irrational, emotionally driven, or bounded-rational decisions—such as anchoring, loss aversion, or herd behavior—leading to price deviations from equilibrium [2]. The simple model breaks down when real-world factors like cognitive biases, information asymmetry, or social influences cause prices to persistently deviate from what supply and demand alone would predict.","promptFormat":"chat-template","retrievedTitles":["Supply and demand","Behavioral economics"],"expectedKbTitles":["Supply and demand","Behavioral economics"],"expectedKbHit":true,"timedOut":false,"createdAt":1791169755092} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":41832.80019199848,"generationLatencyMs":20443.053924024105,"totalLatencyMs":63011.610056996346,"tokensGenerated":93,"tokPerSec":4.5492224569592805,"peakRssBytes":2902634496,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":734.2030850052834,"contextTokensBefore":308,"contextTokensAfter":307,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-1","category":"synthesis","query":"What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life?","answer":"Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society, transitioning from nomadic lifestyles to settled, productive systems that enabled population growth and technological advancement [1]. The Agricultural Revolution transformed daily life by enabling food surplus and permanent settlements, while the Industrial Revolution rapidly changed daily life through mechanization, urbanization, and mass production—occurring over a few decades rather than centuries. The Industrial Revolution thus accelerated societal change much more quickly than the earlier Agricultural Revolution.","promptFormat":"chat-template","retrievedTitles":["Industrial Revolution","Agricultural revolution"],"expectedKbTitles":["Agricultural revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791169818499} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":37697.947445988655,"generationLatencyMs":22587.688854038715,"totalLatencyMs":61479.75698804855,"tokensGenerated":103,"tokPerSec":4.560006146072949,"peakRssBytes":2986250240,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":1193.469685971737,"contextTokensBefore":330,"contextTokensAfter":298,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-2","category":"synthesis","query":"How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism.","answer":"The immune system detects pathogens and mounts a response by identifying foreign molecules, triggering both innate and adaptive defenses. Vaccines mimic this process by introducing a harmless version of a pathogen—like a weakened or killed microbe or a surface protein—so the immune system learns to recognize and remember it. This prepares the body to respond rapidly and effectively if the real pathogen is encountered later [2]. The adaptive immune system produces memory B and T cells that respond faster and stronger upon future exposure, providing long-term protection.","promptFormat":"chat-template","retrievedTitles":["Immune system","Vaccine"],"expectedKbTitles":["Immune system","Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791169880578} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":13795.649136006832,"generationLatencyMs":20972.973358988762,"totalLatencyMs":35199.36927998066,"tokensGenerated":91,"tokPerSec":4.33891744591372,"peakRssBytes":2987171840,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":430.2558600306511,"contextTokensBefore":310,"contextTokensAfter":216,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-3","category":"synthesis","query":"Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species?","answer":"The Amazon rainforest is crucial for global climate because it absorbs significant amounts of carbon dioxide and regulates regional and global weather patterns through evaporation and cloud formation [1]. Biodiversity loss in the Amazon threatens not only species survival but also ecosystem resilience, such as the forest's ability to sequester carbon, maintain soil fertility, and support water cycles—disruptions of which could accelerate climate change and reduce the region’s capacity to recover from environmental stress.","promptFormat":"chat-template","retrievedTitles":["Biodiversity loss","Amazon rainforest"],"expectedKbTitles":["Amazon rainforest"],"expectedKbHit":true,"timedOut":false,"createdAt":1791169915934} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"chat","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":10160.975875973701,"generationLatencyMs":12641.030971050262,"totalLatencyMs":23283.069347023964,"tokensGenerated":58,"tokPerSec":4.588233359512222,"peakRssBytes":2983161856,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":480.40767496824265,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-1","category":"reasoning","query":"A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive?","answer":"Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. \nArrival time: 6:15 pm.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791169939371} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"calculate","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":5082.867515027523,"generationLatencyMs":27806.872491002083,"totalLatencyMs":32890.31882297993,"tokensGenerated":114,"tokPerSec":4.099705928341593,"peakRssBytes":2919481344,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-2","category":"reasoning","query":"A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM?","answer":"Total used: 2GB (OS/app) + 200MB (embedding) + 1.5GB (KV cache) = 3.2GB. \nRemaining for LLM weights: 12GB – 3.2GB = 8.8GB. \nYes, a 9GB model would fit if loaded fully into RAM, as 9GB ≤ 8.8GB? No — 9GB > 8.8GB. \nSo, **no**, a 9GB model would not fit.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791169972441} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":4068.552655041218,"generationLatencyMs":23966.005137979984,"totalLatencyMs":28600.689141988754,"tokensGenerated":99,"tokPerSec":4.130851154793017,"peakRssBytes":2929647616,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":565.3735349774361,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-3","category":"reasoning","query":"If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure?","answer":"Minimum disk footprint: 8B per token × 1 byte/Q8 = 8 bytes per token. For example, at 1 token/sec, it's 8 bytes/sec, or ~8 MB/s sustained. \n\nRAM usage doesn’t scale with 100B because only 8B per token are activated (via gating), not stored or processed simultaneously. The model uses sparse, token-wise computation—only relevant experts are engaged per input, so active parameters are minimal.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791170001232} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":32054.34763199091,"generationLatencyMs":18541.973113000393,"totalLatencyMs":50996.73794001341,"tokensGenerated":72,"tokPerSec":3.883081890002225,"peakRssBytes":2952048640,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":400.01131200790405,"contextTokensBefore":396,"contextTokensAfter":359,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-1","category":"retrieval-grounded","query":"Why did the Western Roman Empire fall?","answer":"The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, political instability, internal conflicts, religious changes, and external pressures from invading tribes. Climatic changes and widespread diseases also contributed to the collapse [1]. The Eastern Roman (Byzantine) Empire survived until 1453, continuing the Roman tradition [2].","promptFormat":"chat-template","retrievedTitles":["Fall of the Western Roman Empire","Byzantine Empire"],"expectedKbTitles":["Fall of the Western Roman Empire"],"expectedKbHit":true,"timedOut":false,"createdAt":1791170052407} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":7066.144371986389,"generationLatencyMs":13591.3068100214,"totalLatencyMs":21320.144379019737,"tokensGenerated":60,"tokPerSec":4.414586532308995,"peakRssBytes":3007877120,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":662.1398519873619,"contextTokensBefore":155,"contextTokensAfter":121,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-2","category":"retrieval-grounded","query":"What is a black hole and how does one form?","answer":"A black hole is an astronomical object so dense that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, squeezing its mass into an extremely small volume, creating a region where spacetime is severely curved.","promptFormat":"chat-template","retrievedTitles":["Black hole"],"expectedKbTitles":["Black hole"],"expectedKbHit":true,"timedOut":false,"createdAt":1791170073932} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":8777.678552985191,"generationLatencyMs":13544.695399999619,"totalLatencyMs":22707.59791100025,"tokensGenerated":61,"tokPerSec":4.503608106240744,"peakRssBytes":2992648192,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":384.7342129945755,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-1","category":"no-kb-content","query":"Who was Napoleon Bonaparte?","answer":"Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He expanded the French Empire across Europe, implementing legal and political reforms, and was a major figure in shaping modern European history.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791170096828} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":18739.548389971256,"generationLatencyMs":15283.745734989643,"totalLatencyMs":34429.25148200989,"tokensGenerated":67,"tokPerSec":4.3837421245902055,"peakRssBytes":2994024448,"outcome":"success","runId":"eval-2026-10-05T03-06-56-189Z","evalSetVersion":"1","retrievalMs":405.53462797403336,"contextTokensBefore":175,"contextTokensAfter":147,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-2","category":"no-kb-content","query":"How do antibiotics work, and why does antibiotic resistance develop?","answer":"Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis or protein production, to kill or inhibit their growth [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to misuse or overuse of these drugs, which selects for resistant strains.","promptFormat":"chat-template","retrievedTitles":["Antimicrobial resistance"],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791170131452} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std/eval-2026-10-05T03-06-56-189Z.report.md b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std/eval-2026-10-05T03-06-56-189Z.report.md new file mode 100644 index 000000000..193af5f86 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std/eval-2026-10-05T03-06-56-189Z.report.md @@ -0,0 +1,17 @@ +BOAR Device Evaluation — eval-2026-10-05T03-06-56-189Z (set v1, 17 queries) +──────────────────────────────────────────────────────────── + +Qwen3-4B-Instruct-2507 (Q4_K_M) + Config: model:qwen3-4b-instruct-2507-q4km · prompt chat-template + Queries: 17 ok · 0 failed · 0 cancelled (of 17) + TTFT: avg 13.5s · p50 9.2s + Model load: none (already resident) + Generation: avg 16.4s · p50 15.3s + Tokens/sec: avg 4.8 · p50 4.5 + Search time: avg 0.5s · p50 0.4s (before the model starts) + Context tokens: 253 → 211 avg (trimmed to the answering sentences) + Total per query: avg 30.3s · p50 28.6s + Peak RSS: 2.80 GB + Residency: cold 0 · switched 0 · resident 17 + Model switches: 0 within plans · 0 across messages + Retrieval: used 11/17 · expected KB article found 10/10 diff --git a/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std/eval-2026-10-05T03-06-56-189Z.status.json b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std/eval-2026-10-05T03-06-56-189Z.status.json new file mode 100644 index 000000000..bec6c45c2 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-std/eval-2026-10-05T03-06-56-189Z.status.json @@ -0,0 +1,17 @@ +{ + "requestId": "req-20261005t030650-ky0k", + "state": "done", + "configs": [ + "model:qwen3-4b-instruct-2507-q4km" + ], + "total": 17, + "completed": 17, + "runId": "eval-2026-10-05T03-06-56-189Z", + "resultPath": "files/eval/eval-2026-10-05T03-06-56-189Z.jsonl", + "stopped": false, + "installedModels": [ + "qwen3-4b-instruct-2507-q4km", + "qwen2.5-1.5b-instruct-q4km" + ], + "updatedAt": 1791170131648 +} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-vitalik/eval-2026-10-05T03-34-26-654Z.answers.md b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-vitalik/eval-2026-10-05T03-34-26-654Z.answers.md new file mode 100644 index 000000000..444bbc0fd --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-vitalik/eval-2026-10-05T03-34-26-654Z.answers.md @@ -0,0 +1,50 @@ +## vitalik-about-x (vitalik) + +> Tell me about X.com. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +X.com was founded in 1999 by Elon Musk as a peer-to-peer payment platform, later evolving into the social media platform known as X (formerly Twitter). It was acquired by Elon Musk in 2018, who rebranded it from Twitter to X in 2023 to reflect its broader focus on communication, creativity, and innovation. The platform is now a major social media network with real-time updates, tweets, and multimedia sharing. + +## vitalik-zk-snark (vitalik) + +> How does a zk-SNARK differ from a zk-STARK? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +zk-SNARKs and zk-STARKs are both zero-knowledge proof systems that allow one party to prove knowledge of a secret without revealing it. zk-SNARKs (Zero-Knowledge Succinct Non-Interactive Arguments of Knowledge) are more compact and efficient but require a trusted setup phase and are less transparent in their assumptions. zk-STARKs (Zero-Knowledge Scalable Transparent Arguments of Knowledge) are transparent, do not need a trusted setup, and offer stronger security guarantees with better scalability, though they are typically larger in size. + +## vitalik-quadratic-funding (vitalik) + +> What is quadratic funding and what are its weaknesses? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Quadratic funding is a mechanism used in decentralized governance and grant allocation where contributions to a project are weighted by the square of the number of contributors, aiming to encourage diversity and reduce the risk of over-reliance on a few large donors. + +Weaknesses include inefficiency in allocating funds to small or niche projects, potential for high costs due to the quadratic formula, and difficulty in accurately measuring project value or impact, which can lead to suboptimal outcomes. + +## vitalik-far-uvc (vitalik) + +> Does far-UVC light at 222 nm harm human skin? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Far-UVC light at 222 nm does not penetrate human skin deeply and is generally considered safe for human exposure at typical doses. It primarily affects microorganisms on the surface of the skin and in the air, with minimal absorption by human tissue. However, long-term or high-dose exposure effects are still under study. + +## vitalik-zuzalu (vitalik) + +> What was Zuzalu? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Zuzalu was a short-lived, experimental online community and social networking platform that emerged in the early 2000s, primarily focused on user-generated content and real-time interaction. It was not widely adopted or commercially successful and eventually faded from public view. + +## vitalik-vegan-buenos-aires (vitalik) + +> Where can I find vegan food in Buenos Aires? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +In Buenos Aires, you can find vegan food at places like La Sirena, a popular vegan restaurant in the Palermo area, and El Bodeguero, which offers plant-based options. Other great spots include Botánica and Veggie Bistro, both with a mix of creative dishes and local flavors. + diff --git a/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-vitalik/eval-2026-10-05T03-34-26-654Z.jsonl b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-vitalik/eval-2026-10-05T03-34-26-654Z.jsonl new file mode 100644 index 000000000..9c84468ff --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-vitalik/eval-2026-10-05T03-34-26-654Z.jsonl @@ -0,0 +1,6 @@ +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"chat","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9164.876225948334,"generationLatencyMs":14159.873704016209,"totalLatencyMs":23663.272229015827,"tokensGenerated":96,"tokPerSec":6.779721486694562,"peakRssBytes":2681417728,"outcome":"success","runId":"eval-2026-10-05T03-34-26-654Z","evalSetVersion":"vitalik-1","retrievalMs":337.7263999581337,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-about-x","category":"vitalik","query":"Tell me about X.com.","answer":"X.com was founded in 1999 by Elon Musk as a peer-to-peer payment platform, later evolving into the social media platform known as X (formerly Twitter). It was acquired by Elon Musk in 2018, who rebranded it from Twitter to X in 2023 to reflect its broader focus on communication, creativity, and innovation. The platform is now a major social media network with real-time updates, tweets, and multimedia sharing.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["X (social network)"],"expectedKbHit":false,"timedOut":false,"createdAt":1791171290379} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":911.9529629945755,"generationLatencyMs":17345.58619004488,"totalLatencyMs":18579.480762004852,"tokensGenerated":110,"tokPerSec":6.341670946994694,"peakRssBytes":2823032832,"outcome":"success","runId":"eval-2026-10-05T03-34-26-654Z","evalSetVersion":"vitalik-1","retrievalMs":321.17952501773834,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-zk-snark","category":"vitalik","query":"How does a zk-SNARK differ from a zk-STARK?","answer":"zk-SNARKs and zk-STARKs are both zero-knowledge proof systems that allow one party to prove knowledge of a secret without revealing it. zk-SNARKs (Zero-Knowledge Succinct Non-Interactive Arguments of Knowledge) are more compact and efficient but require a trusted setup phase and are less transparent in their assumptions. zk-STARKs (Zero-Knowledge Scalable Transparent Arguments of Knowledge) are transparent, do not need a trusted setup, and offer stronger security guarantees with better scalability, though they are typically larger in size.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Zero-knowledge proof","Non-interactive zero-knowledge proof"],"expectedKbHit":false,"timedOut":false,"createdAt":1791171309135} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":877.0980640053749,"generationLatencyMs":14695.8605209589,"totalLatencyMs":15826.932096004486,"tokensGenerated":92,"tokPerSec":6.260266274900452,"peakRssBytes":2839707648,"outcome":"success","runId":"eval-2026-10-05T03-34-26-654Z","evalSetVersion":"vitalik-1","retrievalMs":253.55509495735168,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-quadratic-funding","category":"vitalik","query":"What is quadratic funding and what are its weaknesses?","answer":"Quadratic funding is a mechanism used in decentralized governance and grant allocation where contributions to a project are weighted by the square of the number of contributors, aiming to encourage diversity and reduce the risk of over-reliance on a few large donors.\n\nWeaknesses include inefficiency in allocating funds to small or niche projects, potential for high costs due to the quadratic formula, and difficulty in accurately measuring project value or impact, which can lead to suboptimal outcomes.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Quadratic funding","Quadratic voting"],"expectedKbHit":false,"timedOut":false,"createdAt":1791171325053} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":1029.4694830179214,"generationLatencyMs":10470.768845021725,"totalLatencyMs":11910.594691991806,"tokensGenerated":66,"tokPerSec":6.303262060013804,"peakRssBytes":2901143552,"outcome":"success","runId":"eval-2026-10-05T03-34-26-654Z","evalSetVersion":"vitalik-1","retrievalMs":407.425822019577,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-far-uvc","category":"vitalik","query":"Does far-UVC light at 222 nm harm human skin?","answer":"Far-UVC light at 222 nm does not penetrate human skin deeply and is generally considered safe for human exposure at typical doses. It primarily affects microorganisms on the surface of the skin and in the air, with minimal absorption by human tissue. However, long-term or high-dose exposure effects are still under study.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Far-UVC","Ultraviolet germicidal irradiation"],"expectedKbHit":false,"timedOut":false,"createdAt":1791171337051} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":693.5279949903488,"generationLatencyMs":7836.203048050404,"totalLatencyMs":8773.389815986156,"tokensGenerated":53,"tokPerSec":6.763479669300562,"peakRssBytes":2890821632,"outcome":"success","runId":"eval-2026-10-05T03-34-26-654Z","evalSetVersion":"vitalik-1","retrievalMs":243.33748298883438,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-zuzalu","category":"vitalik","query":"What was Zuzalu?","answer":"Zuzalu was a short-lived, experimental online community and social networking platform that emerged in the early 2000s, primarily focused on user-generated content and real-time interaction. It was not widely adopted or commercially successful and eventually faded from public view.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Zuzalu","Network state"],"expectedKbHit":false,"timedOut":false,"createdAt":1791171345909} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":896.4000660181046,"generationLatencyMs":10363.973800003529,"totalLatencyMs":11527.896978974342,"tokensGenerated":64,"tokPerSec":6.175237532921804,"peakRssBytes":2898931712,"outcome":"success","runId":"eval-2026-10-05T03-34-26-654Z","evalSetVersion":"vitalik-1","retrievalMs":266.9514570236206,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"vitalik-vegan-buenos-aires","category":"vitalik","query":"Where can I find vegan food in Buenos Aires?","answer":"In Buenos Aires, you can find vegan food at places like La Sirena, a popular vegan restaurant in the Palermo area, and El Bodeguero, which offers plant-based options. Other great spots include Botánica and Veggie Bistro, both with a mix of creative dishes and local flavors.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":["Buenos Aires","Veganism"],"expectedKbHit":false,"timedOut":false,"createdAt":1791171357521} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-vitalik/eval-2026-10-05T03-34-26-654Z.report.md b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-vitalik/eval-2026-10-05T03-34-26-654Z.report.md new file mode 100644 index 000000000..b63ca9b96 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-vitalik/eval-2026-10-05T03-34-26-654Z.report.md @@ -0,0 +1,15 @@ +BOAR Device Evaluation — eval-2026-10-05T03-34-26-654Z (set vvitalik-1, 6 queries) +──────────────────────────────────────────────────────────── + +Qwen3-4B-Instruct-2507 (Q4_K_M) + Config: model:qwen3-4b-instruct-2507-q4km · prompt chat-template + Queries: 6 ok · 0 failed · 0 cancelled (of 6) + TTFT: avg 2.3s · p50 0.9s + Model load: none (already resident) + Generation: avg 12.5s · p50 12.3s + Tokens/sec: avg 6.4 · p50 6.3 + Total per query: avg 15.0s · p50 13.9s + Peak RSS: 2.70 GB + Residency: cold 0 · switched 0 · resident 6 + Model switches: 0 within plans · 0 across messages + Retrieval: used 0/6 · expected KB article found 0/6 diff --git a/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-vitalik/eval-2026-10-05T03-34-26-654Z.status.json b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-vitalik/eval-2026-10-05T03-34-26-654Z.status.json new file mode 100644 index 000000000..9ec277247 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/fix-4b-vitalik/eval-2026-10-05T03-34-26-654Z.status.json @@ -0,0 +1,17 @@ +{ + "requestId": "req-20261005t033424-r0ow", + "state": "done", + "configs": [ + "model:qwen3-4b-instruct-2507-q4km" + ], + "total": 6, + "completed": 6, + "runId": "eval-2026-10-05T03-34-26-654Z", + "resultPath": "files/eval/eval-2026-10-05T03-34-26-654Z.jsonl", + "stopped": false, + "installedModels": [ + "qwen3-4b-instruct-2507-q4km", + "qwen2.5-1.5b-instruct-q4km" + ], + "updatedAt": 1791171357584 +} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/narrow-4b-std/eval-2026-10-06T00-28-48-018Z.answers.md b/docs/evidence/2026-10-05-android-conversations/harness/narrow-4b-std/eval-2026-10-06T00-28-48-018Z.answers.md new file mode 100644 index 000000000..2993bf878 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/narrow-4b-std/eval-2026-10-06T00-28-48-018Z.answers.md @@ -0,0 +1,141 @@ +## greeting-1 (greeting) + +> hey, what's up? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Hey! Just checking in. How can I help you today? + +## factual-1 (factual) + +> What is the capital of Australia? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +The capital of Australia is Canberra. + +## factual-2 (factual) + +> Who proposed the theory of evolution by natural selection? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Evolution + +Charles Darwin proposed the theory of evolution by natural selection. [1] + +## explanation-1 (explanation) + +> How do vaccines work? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Vaccine + +Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed microorganism, or a protein fragment—into the body, which triggers the immune system to produce antibodies and memory cells. This prepares the body to recognize and fight the actual disease more effectively if exposed later [1]. + +## explanation-2 (explanation) + +> Explain photosynthesis in simple terms. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Photosynthesis + +Photosynthesis is how plants use sunlight to turn carbon dioxide and water into food (like sugar) and release oxygen as a byproduct. It's like a natural power plant using the sun's energy to grow and stay alive [1]. + +## comparison-1 (comparison) + +> Compare the French Revolution and the Industrial Revolution. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: French Revolution, Industrial Revolution + +The French Revolution (1789–1799) was a political and social upheaval that dismantled monarchy and aristocracy, promoting ideals of liberty, equality, and democracy [1]. The Industrial Revolution (beginning c. 1760) was an economic transformation marked by mechanized manufacturing, urbanization, and technological innovation, fundamentally altering economies and societies [2]. While the French Revolution changed political structures, the Industrial Revolution transformed production and daily life globally. + +## comparison-2 (comparison) + +> Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Supply and demand, Behavioral economics + +The supply and demand model assumes rational, informed buyers and sellers who act to maximize utility, leading to equilibrium prices where quantity demanded equals quantity supplied [1]. Behavioral economics complicates this by showing that individuals often make irrational, emotionally driven, or bounded-rational decisions—such as herd behavior, loss aversion, or cognitive biases—that cause prices to deviate from equilibrium [2]. The simple model breaks down when prices are driven by sentiment, speculation, or social influences rather than objective supply and demand, especially in markets like real estate, financial assets, or during crises. + +## synthesis-1 (synthesis) + +> What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Industrial Revolution, Agricultural revolution + +Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society: the Agricultural Revolution enabled settled farming and population growth, while the Industrial Revolution transformed economies to rely on machine-based manufacturing [1]. The Agricultural Revolution changed daily life gradually over centuries, while the Industrial Revolution brought rapid, dramatic changes in work, living conditions, and urbanization within a few decades. + +## synthesis-2 (synthesis) + +> How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Immune system, Vaccine + +The immune system detects pathogens and mounts a response by identifying foreign molecules and activating defenses—innate responses provide immediate, broad protection, while adaptive responses develop memory for specific threats. Vaccines mimic this process by introducing a harmless version of a pathogen (like a weakened or inactivated form or surface protein), triggering the adaptive immune system to recognize and remember the pathogen. This prepares the body to respond quickly and effectively if the actual pathogen is later encountered [2]. + +## synthesis-3 (synthesis) + +> Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Biodiversity loss, Amazon rainforest + +The Amazon rainforest is crucial for global climate regulation because it absorbs large amounts of carbon dioxide and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens not only species survival but also ecosystem resilience, such as the rainforest's ability to regulate water cycles, maintain soil fertility, and buffer against climate extremes—functions essential for global environmental stability. + +## reasoning-1 (reasoning) + +> A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. +The train arrives at 6:15 pm. + +## reasoning-2 (reasoning) + +> A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Total used: OS/app (2GB) + embedding model (0.2GB) + KV cache (1.5GB) = 3.7GB. +Remaining for LLM weights: 12GB - 3.7GB = 8.3GB. +A 9GB model would not fit fully in RAM — it exceeds the available 8.3GB. + +## reasoning-3 (reasoning) + +> If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Minimum disk footprint: 8B × 1 byte = 8 bytes per token (since only 8B are activated per token at Q8). Total disk usage per token is 8 bytes. + +RAM usage doesn’t scale with the 100B total parameters because only a subset (8B) of the model parameters is activated per token via the experts' routing mechanism—this is known as parameter efficiency. The rest remain inactive and unused in memory at any given time. + +## grounded-1 (retrieval-grounded) + +> Why did the Western Roman Empire fall? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Fall of the Western Roman Empire, Byzantine Empire + +The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, political instability, and invasions by external groups like the Visigoths, Vandals, and Huns. Internal struggles, population decline, and climate change also exacerbated the situation [1]. The empire lost effective control over its provinces, leading to fragmentation among successor states. + +## grounded-2 (retrieval-grounded) + +> What is a black hole and how does one form? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Black hole + +A black hole is an astronomical object so dense that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, compressing its mass into an extremely small volume. + +## no-kb-1 (no-kb-content) + +> Who was Napoleon Bonaparte? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He led France through a period of significant expansion and reform, creating a powerful empire that influenced European politics and law for decades. + +## no-kb-2 (no-kb-content) + +> How do antibiotics work, and why does antibiotic resistance develop? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Antimicrobial resistance + +Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis, protein production, or DNA replication, thereby killing or inhibiting their growth [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to overuse or misuse in humans, animals, or agriculture [1]. + diff --git a/docs/evidence/2026-10-05-android-conversations/harness/narrow-4b-std/eval-2026-10-06T00-28-48-018Z.jsonl b/docs/evidence/2026-10-05-android-conversations/harness/narrow-4b-std/eval-2026-10-06T00-28-48-018Z.jsonl new file mode 100644 index 000000000..3c5cc7a9a --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/narrow-4b-std/eval-2026-10-06T00-28-48-018Z.jsonl @@ -0,0 +1,17 @@ +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"greeting","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":true,"modelResidency":"switched","modelLoadMs":15191.457690000534,"ttftMs":12133.379563987255,"generationLatencyMs":3689.63362801075,"totalLatencyMs":31026.180719971657,"tokensGenerated":13,"tokPerSec":3.523385059510338,"peakRssBytes":3319259136,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"greeting-1","category":"greeting","query":"hey, what's up?","answer":"Hey! Just checking in. How can I help you today?","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791246559152} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":593.1583260297775,"generationLatencyMs":893.3210859894753,"totalLatencyMs":1913.5764990448952,"tokensGenerated":7,"tokPerSec":7.835928323852944,"peakRssBytes":2769592320,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":426.4809980392456,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-1","category":"factual","query":"What is the capital of Australia?","answer":"The capital of Australia is Canberra.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791246561276} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":11645.847010970116,"generationLatencyMs":2441.673789024353,"totalLatencyMs":14331.726853966713,"tokensGenerated":14,"tokPerSec":5.733771670454855,"peakRssBytes":2802892800,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":243.77547299861908,"contextTokensBefore":111,"contextTokensAfter":111,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-2","category":"factual","query":"Who proposed the theory of evolution by natural selection?","answer":"Charles Darwin proposed the theory of evolution by natural selection. [1]","promptFormat":"chat-template","retrievedTitles":["Evolution"],"expectedKbTitles":["Evolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791246575658} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":3949.7735210061073,"generationLatencyMs":9918.266768991947,"totalLatencyMs":14253.30807095766,"tokensGenerated":64,"tokPerSec":6.452740331615894,"peakRssBytes":2809626624,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":384.7128090262413,"contextTokensBefore":154,"contextTokensAfter":102,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-1","category":"explanation","query":"How do vaccines work?","answer":"Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed microorganism, or a protein fragment—into the body, which triggers the immune system to produce antibodies and memory cells. This prepares the body to recognize and fight the actual disease more effectively if exposed later [1].","promptFormat":"chat-template","retrievedTitles":["Vaccine"],"expectedKbTitles":["Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791246590094} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":7931.291589021683,"generationLatencyMs":7080.155032992363,"totalLatencyMs":15262.001838982105,"tokensGenerated":47,"tokPerSec":6.638272718745239,"peakRssBytes":2828668928,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":250.27754801511765,"contextTokensBefore":372,"contextTokensAfter":181,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-2","category":"explanation","query":"Explain photosynthesis in simple terms.","answer":"Photosynthesis is how plants use sunlight to turn carbon dioxide and water into food (like sugar) and release oxygen as a byproduct. It's like a natural power plant using the sun's energy to grow and stay alive [1].","promptFormat":"chat-template","retrievedTitles":["Photosynthesis"],"expectedKbTitles":["Photosynthesis"],"expectedKbHit":true,"timedOut":false,"createdAt":1791246605453} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"compare","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":11914.672653973103,"generationLatencyMs":15389.840990006924,"totalLatencyMs":27627.017795026302,"tokensGenerated":98,"tokPerSec":6.3678370727569105,"peakRssBytes":2853670912,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":322.26049798727036,"contextTokensBefore":286,"contextTokensAfter":286,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-1","category":"comparison","query":"Compare the French Revolution and the Industrial Revolution.","answer":"The French Revolution (1789–1799) was a political and social upheaval that dismantled monarchy and aristocracy, promoting ideals of liberty, equality, and democracy [1]. The Industrial Revolution (beginning c. 1760) was an economic transformation marked by mechanized manufacturing, urbanization, and technological innovation, fundamentally altering economies and societies [2]. While the French Revolution changed political structures, the Industrial Revolution transformed production and daily life globally.","promptFormat":"chat-template","retrievedTitles":["French Revolution","Industrial Revolution"],"expectedKbTitles":["French Revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791246633183} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":10023.646815001965,"generationLatencyMs":18689.357837975025,"totalLatencyMs":28975.350193977356,"tokensGenerated":114,"tokPerSec":6.099728037116539,"peakRssBytes":2863337472,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":262.01456797122955,"contextTokensBefore":188,"contextTokensAfter":188,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-2","category":"comparison","query":"Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down?","answer":"The supply and demand model assumes rational, informed buyers and sellers who act to maximize utility, leading to equilibrium prices where quantity demanded equals quantity supplied [1]. Behavioral economics complicates this by showing that individuals often make irrational, emotionally driven, or bounded-rational decisions—such as herd behavior, loss aversion, or cognitive biases—that cause prices to deviate from equilibrium [2]. The simple model breaks down when prices are driven by sentiment, speculation, or social influences rather than objective supply and demand, especially in markets like real estate, financial assets, or during crises.","promptFormat":"chat-template","retrievedTitles":["Supply and demand","Behavioral economics"],"expectedKbTitles":["Supply and demand","Behavioral economics"],"expectedKbHit":true,"timedOut":false,"createdAt":1791246662258} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":16281.115159988403,"generationLatencyMs":12947.103888988495,"totalLatencyMs":29710.77886301279,"tokensGenerated":72,"tokPerSec":5.561089230251405,"peakRssBytes":2847952896,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":481.96590197086334,"contextTokensBefore":308,"contextTokensAfter":307,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-1","category":"synthesis","query":"What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life?","answer":"Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society: the Agricultural Revolution enabled settled farming and population growth, while the Industrial Revolution transformed economies to rely on machine-based manufacturing [1]. The Agricultural Revolution changed daily life gradually over centuries, while the Industrial Revolution brought rapid, dramatic changes in work, living conditions, and urbanization within a few decades.","promptFormat":"chat-template","retrievedTitles":["Industrial Revolution","Agricultural revolution"],"expectedKbTitles":["Agricultural revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791246692092} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":18716.070850014687,"generationLatencyMs":16446.310717999935,"totalLatencyMs":35541.810198009014,"tokensGenerated":94,"tokPerSec":5.715567558693887,"peakRssBytes":2862694400,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":378.78128999471664,"contextTokensBefore":330,"contextTokensAfter":298,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-2","category":"synthesis","query":"How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism.","answer":"The immune system detects pathogens and mounts a response by identifying foreign molecules and activating defenses—innate responses provide immediate, broad protection, while adaptive responses develop memory for specific threats. Vaccines mimic this process by introducing a harmless version of a pathogen (like a weakened or inactivated form or surface protein), triggering the adaptive immune system to recognize and remember the pathogen. This prepares the body to respond quickly and effectively if the actual pathogen is later encountered [2].","promptFormat":"chat-template","retrievedTitles":["Immune system","Vaccine"],"expectedKbTitles":["Immune system","Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791246727761} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":39904.605040967464,"generationLatencyMs":12288.84831303358,"totalLatencyMs":53931.65875899792,"tokensGenerated":73,"tokPerSec":5.940345111313322,"peakRssBytes":2947092480,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":1737.4586600065231,"contextTokensBefore":310,"contextTokensAfter":216,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-3","category":"synthesis","query":"Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species?","answer":"The Amazon rainforest is crucial for global climate regulation because it absorbs large amounts of carbon dioxide and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens not only species survival but also ecosystem resilience, such as the rainforest's ability to regulate water cycles, maintain soil fertility, and buffer against climate extremes—functions essential for global environmental stability.","promptFormat":"chat-template","retrievedTitles":["Biodiversity loss","Amazon rainforest"],"expectedKbTitles":["Amazon rainforest"],"expectedKbHit":true,"timedOut":false,"createdAt":1791246782770} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"chat","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9751.4599249959,"generationLatencyMs":9109.234257996082,"totalLatencyMs":19363.03597998619,"tokensGenerated":58,"tokPerSec":6.367165269582086,"peakRssBytes":2949427200,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":501.64188599586487,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-1","category":"reasoning","query":"A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive?","answer":"Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. \nThe train arrives at 6:15 pm.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791246802293} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"calculate","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":5008.0810160040855,"generationLatencyMs":13416.496954977512,"totalLatencyMs":18425.178272008896,"tokensGenerated":81,"tokPerSec":6.037343449025197,"peakRssBytes":2952474624,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-2","category":"reasoning","query":"A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM?","answer":"Total used: OS/app (2GB) + embedding model (0.2GB) + KV cache (1.5GB) = 3.7GB. \nRemaining for LLM weights: 12GB - 3.7GB = 8.3GB. \nA 9GB model would not fit fully in RAM — it exceeds the available 8.3GB.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791246820877} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":4225.274659991264,"generationLatencyMs":16608.479377985,"totalLatencyMs":21351.314220011234,"tokensGenerated":98,"tokPerSec":5.900600396319348,"peakRssBytes":2962313216,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":516.7987880110741,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-3","category":"reasoning","query":"If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure?","answer":"Minimum disk footprint: 8B × 1 byte = 8 bytes per token (since only 8B are activated per token at Q8). Total disk usage per token is 8 bytes.\n\nRAM usage doesn’t scale with the 100B total parameters because only a subset (8B) of the model parameters is activated per token via the experts' routing mechanism—this is known as parameter efficiency. The rest remain inactive and unused in memory at any given time.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791246842408} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":34393.27137899399,"generationLatencyMs":12835.775478959084,"totalLatencyMs":47627.10589802265,"tokensGenerated":72,"tokPerSec":5.609322172861724,"peakRssBytes":2972241920,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":397.6827810406685,"contextTokensBefore":396,"contextTokensAfter":359,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-1","category":"retrieval-grounded","query":"Why did the Western Roman Empire fall?","answer":"The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, political instability, and invasions by external groups like the Visigoths, Vandals, and Huns. Internal struggles, population decline, and climate change also exacerbated the situation [1]. The empire lost effective control over its provinces, leading to fragmentation among successor states.","promptFormat":"chat-template","retrievedTitles":["Fall of the Western Roman Empire","Byzantine Empire"],"expectedKbTitles":["Fall of the Western Roman Empire"],"expectedKbHit":true,"timedOut":false,"createdAt":1791246890213} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":7765.439456999302,"generationLatencyMs":8674.210208952427,"totalLatencyMs":16842.01721996069,"tokensGenerated":51,"tokPerSec":5.879497818413972,"peakRssBytes":2978095104,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":400.68282103538513,"contextTokensBefore":155,"contextTokensAfter":121,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-2","category":"retrieval-grounded","query":"What is a black hole and how does one form?","answer":"A black hole is an astronomical object so dense that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, compressing its mass into an extremely small volume.","promptFormat":"chat-template","retrievedTitles":["Black hole"],"expectedKbTitles":["Black hole"],"expectedKbHit":true,"timedOut":false,"createdAt":1791246907236} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":22602.36947697401,"generationLatencyMs":9869.275802016258,"totalLatencyMs":32814.957860946655,"tokensGenerated":61,"tokPerSec":6.1807979859614335,"peakRssBytes":3000492032,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":342.88700395822525,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-1","category":"no-kb-content","query":"Who was Napoleon Bonaparte?","answer":"Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He led France through a period of significant expansion and reform, creating a powerful empire that influenced European politics and law for decades.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791246940681} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":19814.460417985916,"generationLatencyMs":12446.915615022182,"totalLatencyMs":32640.150976002216,"tokensGenerated":72,"tokPerSec":5.784565608615776,"peakRssBytes":2983440384,"outcome":"success","runId":"eval-2026-10-06T00-28-48-018Z","evalSetVersion":"1","retrievalMs":378.22469103336334,"contextTokensBefore":175,"contextTokensAfter":147,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-2","category":"no-kb-content","query":"How do antibiotics work, and why does antibiotic resistance develop?","answer":"Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis, protein production, or DNA replication, thereby killing or inhibiting their growth [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to overuse or misuse in humans, animals, or agriculture [1].","promptFormat":"chat-template","retrievedTitles":["Antimicrobial resistance"],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791246973519} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/narrow-4b-std/eval-2026-10-06T00-28-48-018Z.report.md b/docs/evidence/2026-10-05-android-conversations/harness/narrow-4b-std/eval-2026-10-06T00-28-48-018Z.report.md new file mode 100644 index 000000000..dc73385be --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/narrow-4b-std/eval-2026-10-06T00-28-48-018Z.report.md @@ -0,0 +1,17 @@ +BOAR Device Evaluation — eval-2026-10-06T00-28-48-018Z (set v1, 17 queries) +──────────────────────────────────────────────────────────── + +Qwen3-4B-Instruct-2507 (Q4_K_M) + Config: model:qwen3-4b-instruct-2507-q4km · prompt chat-template + Queries: 17 ok · 0 failed · 0 cancelled (of 17) + TTFT: avg 13.9s · p50 11.6s + Model load: avg 15.2s · p50 15.2s over 1 load(s) + Generation: avg 10.7s · p50 12.3s + Tokens/sec: avg 6.0 · p50 5.9 + Search time: avg 0.5s · p50 0.4s (before the model starts) + Context tokens: 253 → 211 avg (trimmed to the answering sentences) + Total per query: avg 26.0s · p50 27.6s + Peak RSS: 3.09 GB + Residency: cold 0 · switched 1 · resident 16 + Model switches: 0 within plans · 1 across messages + Retrieval: used 11/17 · expected KB article found 10/10 diff --git a/docs/evidence/2026-10-05-android-conversations/harness/narrow-4b-std/eval-2026-10-06T00-28-48-018Z.status.json b/docs/evidence/2026-10-05-android-conversations/harness/narrow-4b-std/eval-2026-10-06T00-28-48-018Z.status.json new file mode 100644 index 000000000..bd6932099 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/narrow-4b-std/eval-2026-10-06T00-28-48-018Z.status.json @@ -0,0 +1,17 @@ +{ + "requestId": "req-20261006t002845-n566", + "state": "done", + "configs": [ + "model:qwen3-4b-instruct-2507-q4km" + ], + "total": 17, + "completed": 17, + "runId": "eval-2026-10-06T00-28-48-018Z", + "resultPath": "files/eval/eval-2026-10-06T00-28-48-018Z.jsonl", + "stopped": false, + "installedModels": [ + "qwen3-4b-instruct-2507-q4km", + "qwen2.5-1.5b-instruct-q4km" + ], + "updatedAt": 1791246980370 +} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/narrow-f16kv-4b-std/eval-2026-10-06T01-40-22-114Z.answers.md b/docs/evidence/2026-10-05-android-conversations/harness/narrow-f16kv-4b-std/eval-2026-10-06T01-40-22-114Z.answers.md new file mode 100644 index 000000000..85497e975 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/narrow-f16kv-4b-std/eval-2026-10-06T01-40-22-114Z.answers.md @@ -0,0 +1,142 @@ +## greeting-1 (greeting) + +> hey, what's up? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Hey! Just checking in. How can I help you today? + +## factual-1 (factual) + +> What is the capital of Australia? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +The capital of Australia is Canberra. + +## factual-2 (factual) + +> Who proposed the theory of evolution by natural selection? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Evolution + +Charles Darwin proposed the theory of evolution by natural selection. He developed this idea through extensive observations and research, particularly during his voyage on the HMS Beagle, and published it in his 1859 book *On the Origin of Species* [1]. + +## explanation-1 (explanation) + +> How do vaccines work? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Vaccine + +Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed microorganism, or a piece of its surface protein—into the body. This triggers the immune system to produce antibodies and memory cells, so if the person is later exposed to the real disease, their immune system can respond quickly and effectively [1]. + +## explanation-2 (explanation) + +> Explain photosynthesis in simple terms. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Photosynthesis + +Photosynthesis is how plants use sunlight to turn carbon dioxide and water into food (like sugar) and release oxygen as a byproduct. It’s like a natural energy conversion process where light power fuels growth [1]. + +## comparison-1 (comparison) + +> Compare the French Revolution and the Industrial Revolution. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: French Revolution, Industrial Revolution + +The French Revolution (1789–1799) was a political and social upheaval that transformed France’s governance, promoting ideals like liberty, equality, and fraternity [1], while the Industrial Revolution (beginning c. 1760) was a period of economic transformation marked by mechanized manufacturing, urbanization, and technological innovation [2]. The French Revolution focused on political restructuring and democratic ideals, whereas the Industrial Revolution shifted societal structures through industrial growth and economic change. + +## comparison-2 (comparison) + +> Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Supply and demand, Behavioral economics + +The simple supply-and-demand model assumes rational, informed actors who react predictably to price changes, leading to equilibrium [1]. Behavioral economics complicates this by showing that individuals often act out of cognitive biases, emotions, or limited information—such as herd behavior, loss aversion, or overconfidence—leading to prices that deviate from equilibrium [2]. The model breaks down when prices persist beyond supply-demand balance due to irrational behavior, market sentiment, or information asymmetry. + +## synthesis-1 (synthesis) + +> What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Industrial Revolution, Agricultural revolution + +Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society: the Agricultural Revolution transitioned humans from nomadic hunting and gathering to settled farming, enabling population growth and surplus [2]; the Industrial Revolution shifted production from manual labor to mechanized, scalable manufacturing, driving urbanization and economic growth [1]. + +The Agricultural Revolution changed daily life gradually over thousands of years, as farming practices evolved slowly across regions. In contrast, the Industrial Revolution brought rapid changes—within decades—transforming work, living conditions, and social structures, especially in Europe and North America [1]. + +## synthesis-2 (synthesis) + +> How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Immune system, Vaccine + +The immune system detects pathogens and mounts a response by recognizing foreign molecules, using innate and adaptive mechanisms to eliminate them [1]. A vaccine mimics this process by introducing a harmless version of a pathogen—such as a weakened or inactivated microbe or its proteins—so the immune system learns to recognize and respond to it without causing disease [2]. This priming creates memory cells that enable a faster, stronger response if the real pathogen is later encountered, providing immunity. + +## synthesis-3 (synthesis) + +> Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Biodiversity loss, Amazon rainforest + +The Amazon rainforest is crucial for global climate regulation because it absorbs vast amounts of carbon dioxide and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens ecosystem resilience, disrupts climate stability through reduced carbon sequestration, and undermines essential services like water cycling, soil fertility, and pollination—many of which are vital for agriculture and human survival beyond just species extinction. + +## reasoning-1 (reasoning) + +> A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. +The train arrives at 6:15 pm. + +## reasoning-2 (reasoning) + +> A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Total used: OS/app (2GB) + embedding model (0.2GB) + KV cache (1.5GB) = 3.7GB. +Remaining for LLM weights: 12GB - 3.7GB = 8.3GB. +A 9GB model would exceed the available memory and cannot fit fully in RAM. + +## reasoning-3 (reasoning) + +> If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Minimum disk footprint: 8B × 1 byte = 8 bytes per token. +RAM usage doesn’t scale with the 100B total because only 8B per token are activated at inference (sparse activation), and parameters are not stored in memory simultaneously — only the active expert weights are loaded, not the full 100B model. + +## grounded-1 (retrieval-grounded) + +> Why did the Western Roman Empire fall? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Fall of the Western Roman Empire, Byzantine Empire + +The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, population decline, political instability, and external pressures from invading tribes. Climatic changes and widespread diseases also contributed to the erosion of its strength [1]. The Eastern Roman (Byzantine) Empire continued to thrive, preserving Roman institutions and culture [2]. + +## grounded-2 (retrieval-grounded) + +> What is a black hole and how does one form? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Black hole + +A black hole is an astronomical object so dense that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, squeezing its mass into an extremely small volume, creating a region where spacetime is severely curved. + +## no-kb-1 (no-kb-content) + +> Who was Napoleon Bonaparte? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He led successful military campaigns across Europe, reshaped European politics, and introduced significant reforms in law, education, and administration. + +## no-kb-2 (no-kb-content) + +> How do antibiotics work, and why does antibiotic resistance develop? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Antimicrobial resistance + +Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis, protein production, or DNA replication, to kill or inhibit their growth [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to overuse or improper prescribing, leading to reduced treatment effectiveness. + diff --git a/docs/evidence/2026-10-05-android-conversations/harness/narrow-f16kv-4b-std/eval-2026-10-06T01-40-22-114Z.jsonl b/docs/evidence/2026-10-05-android-conversations/harness/narrow-f16kv-4b-std/eval-2026-10-06T01-40-22-114Z.jsonl new file mode 100644 index 000000000..dc1c4f078 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/narrow-f16kv-4b-std/eval-2026-10-06T01-40-22-114Z.jsonl @@ -0,0 +1,17 @@ +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"greeting","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":true,"modelResidency":"switched","modelLoadMs":14965.563686966896,"ttftMs":9521.426110982895,"generationLatencyMs":2657.7795019745827,"totalLatencyMs":27160.769870996475,"tokensGenerated":13,"tokPerSec":4.891301174661677,"peakRssBytes":3058024448,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"greeting-1","category":"greeting","query":"hey, what's up?","answer":"Hey! Just checking in. How can I help you today?","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791250849440} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":15389.631599009037,"generationLatencyMs":1026.5719410181046,"totalLatencyMs":16977.17888200283,"tokensGenerated":7,"tokPerSec":6.81881095742568,"peakRssBytes":2792259584,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":559.9174000024796,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-1","category":"factual","query":"What is the capital of Australia?","answer":"The capital of Australia is Canberra.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791250866912} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":10595.122849047184,"generationLatencyMs":8920.52799898386,"totalLatencyMs":19896.571828007698,"tokensGenerated":52,"tokPerSec":5.8292513633636185,"peakRssBytes":2818990080,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":379.9300130009651,"contextTokensBefore":111,"contextTokensAfter":111,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-2","category":"factual","query":"Who proposed the theory of evolution by natural selection?","answer":"Charles Darwin proposed the theory of evolution by natural selection. He developed this idea through extensive observations and research, particularly during his voyage on the HMS Beagle, and published it in his 1859 book *On the Origin of Species* [1].","promptFormat":"chat-template","retrievedTitles":["Evolution"],"expectedKbTitles":["Evolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791250887024} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":3898.6448179483414,"generationLatencyMs":12067.934373021126,"totalLatencyMs":16250.570849001408,"tokensGenerated":71,"tokPerSec":5.883359803375002,"peakRssBytes":2841292800,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":283.66658502817154,"contextTokensBefore":154,"contextTokensAfter":102,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-1","category":"explanation","query":"How do vaccines work?","answer":"Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed microorganism, or a piece of its surface protein—into the body. This triggers the immune system to produce antibodies and memory cells, so if the person is later exposed to the real disease, their immune system can respond quickly and effectively [1].","promptFormat":"chat-template","retrievedTitles":["Vaccine"],"expectedKbTitles":["Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791250903377} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":6739.10290902853,"generationLatencyMs":7250.154788970947,"totalLatencyMs":14270.924404978752,"tokensGenerated":43,"tokPerSec":5.930907856672563,"peakRssBytes":2854617088,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":281.39868199825287,"contextTokensBefore":372,"contextTokensAfter":181,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-2","category":"explanation","query":"Explain photosynthesis in simple terms.","answer":"Photosynthesis is how plants use sunlight to turn carbon dioxide and water into food (like sugar) and release oxygen as a byproduct. It’s like a natural energy conversion process where light power fuels growth [1].","promptFormat":"chat-template","retrievedTitles":["Photosynthesis"],"expectedKbTitles":["Photosynthesis"],"expectedKbHit":true,"timedOut":false,"createdAt":1791250917723} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"compare","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":10174.929123044014,"generationLatencyMs":18634.900561988354,"totalLatencyMs":29168.019463956356,"tokensGenerated":100,"tokPerSec":5.366274945624392,"peakRssBytes":2912296960,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":357.94409197568893,"contextTokensBefore":286,"contextTokensAfter":286,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-1","category":"comparison","query":"Compare the French Revolution and the Industrial Revolution.","answer":"The French Revolution (1789–1799) was a political and social upheaval that transformed France’s governance, promoting ideals like liberty, equality, and fraternity [1], while the Industrial Revolution (beginning c. 1760) was a period of economic transformation marked by mechanized manufacturing, urbanization, and technological innovation [2]. The French Revolution focused on political restructuring and democratic ideals, whereas the Industrial Revolution shifted societal structures through industrial growth and economic change.","promptFormat":"chat-template","retrievedTitles":["French Revolution","Industrial Revolution"],"expectedKbTitles":["French Revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791250946976} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":7631.700688004494,"generationLatencyMs":17294.947070002556,"totalLatencyMs":25275.230562984943,"tokensGenerated":95,"tokPerSec":5.492933838738019,"peakRssBytes":2928164864,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":348.21891301870346,"contextTokensBefore":188,"contextTokensAfter":188,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-2","category":"comparison","query":"Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down?","answer":"The simple supply-and-demand model assumes rational, informed actors who react predictably to price changes, leading to equilibrium [1]. Behavioral economics complicates this by showing that individuals often act out of cognitive biases, emotions, or limited information—such as herd behavior, loss aversion, or overconfidence—leading to prices that deviate from equilibrium [2]. The model breaks down when prices persist beyond supply-demand balance due to irrational behavior, market sentiment, or information asymmetry.","promptFormat":"chat-template","retrievedTitles":["Supply and demand","Behavioral economics"],"expectedKbTitles":["Supply and demand","Behavioral economics"],"expectedKbHit":true,"timedOut":false,"createdAt":1791250972346} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":13707.48983502388,"generationLatencyMs":24301.484060943127,"totalLatencyMs":38401.93426299095,"tokensGenerated":116,"tokPerSec":4.7733710298966034,"peakRssBytes":2936389632,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":392.3512780070305,"contextTokensBefore":308,"contextTokensAfter":307,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-1","category":"synthesis","query":"What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life?","answer":"Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society: the Agricultural Revolution transitioned humans from nomadic hunting and gathering to settled farming, enabling population growth and surplus [2]; the Industrial Revolution shifted production from manual labor to mechanized, scalable manufacturing, driving urbanization and economic growth [1]. \n\nThe Agricultural Revolution changed daily life gradually over thousands of years, as farming practices evolved slowly across regions. In contrast, the Industrial Revolution brought rapid changes—within decades—transforming work, living conditions, and social structures, especially in Europe and North America [1].","promptFormat":"chat-template","retrievedTitles":["Industrial Revolution","Agricultural revolution"],"expectedKbTitles":["Agricultural revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791251010876} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":36270.305070996284,"generationLatencyMs":18485.796844005585,"totalLatencyMs":56793.943102002144,"tokensGenerated":96,"tokPerSec":5.193176188730542,"peakRssBytes":3021701120,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":2037.255127966404,"contextTokensBefore":330,"contextTokensAfter":298,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-2","category":"synthesis","query":"How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism.","answer":"The immune system detects pathogens and mounts a response by recognizing foreign molecules, using innate and adaptive mechanisms to eliminate them [1]. A vaccine mimics this process by introducing a harmless version of a pathogen—such as a weakened or inactivated microbe or its proteins—so the immune system learns to recognize and respond to it without causing disease [2]. This priming creates memory cells that enable a faster, stronger response if the real pathogen is later encountered, providing immunity.","promptFormat":"chat-template","retrievedTitles":["Immune system","Vaccine"],"expectedKbTitles":["Immune system","Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791251067826} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":13291.22522598505,"generationLatencyMs":15723.801399052143,"totalLatencyMs":29369.649509966373,"tokensGenerated":79,"tokPerSec":5.024230336867665,"peakRssBytes":3028439040,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":354.19181299209595,"contextTokensBefore":310,"contextTokensAfter":216,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-3","category":"synthesis","query":"Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species?","answer":"The Amazon rainforest is crucial for global climate regulation because it absorbs vast amounts of carbon dioxide and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens ecosystem resilience, disrupts climate stability through reduced carbon sequestration, and undermines essential services like water cycling, soil fertility, and pollination—many of which are vital for agriculture and human survival beyond just species extinction.","promptFormat":"chat-template","retrievedTitles":["Biodiversity loss","Amazon rainforest"],"expectedKbTitles":["Amazon rainforest"],"expectedKbHit":true,"timedOut":false,"createdAt":1791251097321} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"chat","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9039.62390500307,"generationLatencyMs":10067.461389958858,"totalLatencyMs":19543.386117994785,"tokensGenerated":58,"tokPerSec":5.761134585313471,"peakRssBytes":3028320256,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":435.5917969942093,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-1","category":"reasoning","query":"A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive?","answer":"Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. \nThe train arrives at 6:15 pm.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791251117027} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"calculate","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":4523.011192023754,"generationLatencyMs":14371.49105501175,"totalLatencyMs":18895.015471994877,"tokensGenerated":76,"tokPerSec":5.288247385680738,"peakRssBytes":3024855040,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-2","category":"reasoning","query":"A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM?","answer":"Total used: OS/app (2GB) + embedding model (0.2GB) + KV cache (1.5GB) = 3.7GB. \nRemaining for LLM weights: 12GB - 3.7GB = 8.3GB. \nA 9GB model would exceed the available memory and cannot fit fully in RAM.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791251136076} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":3643.658937036991,"generationLatencyMs":14031.870733976364,"totalLatencyMs":18271.239022016525,"tokensGenerated":74,"tokPerSec":5.273708787868075,"peakRssBytes":3015266304,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":594.967611014843,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-3","category":"reasoning","query":"If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure?","answer":"Minimum disk footprint: 8B × 1 byte = 8 bytes per token. \nRAM usage doesn’t scale with the 100B total because only 8B per token are activated at inference (sparse activation), and parameters are not stored in memory simultaneously — only the active expert weights are loaded, not the full 100B model.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791251154543} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":27321.99053299427,"generationLatencyMs":14660.357876002789,"totalLatencyMs":42370.35695397854,"tokensGenerated":69,"tokPerSec":4.706569961224791,"peakRssBytes":3022278656,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":387.61901795864105,"contextTokensBefore":396,"contextTokensAfter":359,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-1","category":"retrieval-grounded","query":"Why did the Western Roman Empire fall?","answer":"The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, population decline, political instability, and external pressures from invading tribes. Climatic changes and widespread diseases also contributed to the erosion of its strength [1]. The Eastern Roman (Byzantine) Empire continued to thrive, preserving Roman institutions and culture [2].","promptFormat":"chat-template","retrievedTitles":["Fall of the Western Roman Empire","Byzantine Empire"],"expectedKbTitles":["Fall of the Western Roman Empire"],"expectedKbHit":true,"timedOut":false,"createdAt":1791251197092} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":6441.923790991306,"generationLatencyMs":11761.223272025585,"totalLatencyMs":18628.79920297861,"tokensGenerated":60,"tokPerSec":5.1015101586168985,"peakRssBytes":3028123648,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":425.19360399246216,"contextTokensBefore":155,"contextTokensAfter":121,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-2","category":"retrieval-grounded","query":"What is a black hole and how does one form?","answer":"A black hole is an astronomical object so dense that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, squeezing its mass into an extremely small volume, creating a region where spacetime is severely curved.","promptFormat":"chat-template","retrievedTitles":["Black hole"],"expectedKbTitles":["Black hole"],"expectedKbHit":true,"timedOut":false,"createdAt":1791251215910} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":7845.328047990799,"generationLatencyMs":10276.478114008904,"totalLatencyMs":18472.967335045338,"tokensGenerated":61,"tokPerSec":5.935885750279053,"peakRssBytes":3026788352,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":350.74767994880676,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-1","category":"no-kb-content","query":"Who was Napoleon Bonaparte?","answer":"Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He led successful military campaigns across Europe, reshaped European politics, and introduced significant reforms in law, education, and administration.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791251234574} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":16912.242439985275,"generationLatencyMs":13506.301968038082,"totalLatencyMs":31153.227351009846,"tokensGenerated":69,"tokPerSec":5.10872629408736,"peakRssBytes":3028000768,"outcome":"success","runId":"eval-2026-10-06T01-40-22-114Z","evalSetVersion":"1","retrievalMs":733.8732500076294,"contextTokensBefore":175,"contextTokensAfter":147,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-2","category":"no-kb-content","query":"How do antibiotics work, and why does antibiotic resistance develop?","answer":"Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis, protein production, or DNA replication, to kill or inhibit their growth [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to overuse or improper prescribing, leading to reduced treatment effectiveness.","promptFormat":"chat-template","retrievedTitles":["Antimicrobial resistance"],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791251265924} diff --git a/docs/evidence/2026-10-05-android-conversations/harness/narrow-f16kv-4b-std/eval-2026-10-06T01-40-22-114Z.report.md b/docs/evidence/2026-10-05-android-conversations/harness/narrow-f16kv-4b-std/eval-2026-10-06T01-40-22-114Z.report.md new file mode 100644 index 000000000..b10bb80a1 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/narrow-f16kv-4b-std/eval-2026-10-06T01-40-22-114Z.report.md @@ -0,0 +1,17 @@ +BOAR Device Evaluation — eval-2026-10-06T01-40-22-114Z (set v1, 17 queries) +──────────────────────────────────────────────────────────── + +Qwen3-4B-Instruct-2507 (Q4_K_M) + Config: model:qwen3-4b-instruct-2507-q4km · prompt chat-template + Queries: 17 ok · 0 failed · 0 cancelled (of 17) + TTFT: avg 11.9s · p50 9.5s + Model load: avg 15.0s · p50 15.0s over 1 load(s) + Generation: avg 12.6s · p50 13.5s + Tokens/sec: avg 5.4 · p50 5.3 + Search time: avg 0.5s · p50 0.4s (before the model starts) + Context tokens: 253 → 211 avg (trimmed to the answering sentences) + Total per query: avg 25.9s · p50 19.9s + Peak RSS: 2.85 GB + Residency: cold 0 · switched 1 · resident 16 + Model switches: 0 within plans · 1 across messages + Retrieval: used 11/17 · expected KB article found 10/10 diff --git a/docs/evidence/2026-10-05-android-conversations/harness/narrow-f16kv-4b-std/eval-2026-10-06T01-40-22-114Z.status.json b/docs/evidence/2026-10-05-android-conversations/harness/narrow-f16kv-4b-std/eval-2026-10-06T01-40-22-114Z.status.json new file mode 100644 index 000000000..06f5be1e7 --- /dev/null +++ b/docs/evidence/2026-10-05-android-conversations/harness/narrow-f16kv-4b-std/eval-2026-10-06T01-40-22-114Z.status.json @@ -0,0 +1,17 @@ +{ + "requestId": "req-20261006t014017-qi7q", + "state": "done", + "configs": [ + "model:qwen3-4b-instruct-2507-q4km" + ], + "total": 17, + "completed": 17, + "runId": "eval-2026-10-06T01-40-22-114Z", + "resultPath": "files/eval/eval-2026-10-06T01-40-22-114Z.jsonl", + "stopped": false, + "installedModels": [ + "qwen3-4b-instruct-2507-q4km", + "qwen2.5-1.5b-instruct-q4km" + ], + "updatedAt": 1791251272859 +} diff --git a/docs/evidence/2026-10-06-repack-under-pressure/README.md b/docs/evidence/2026-10-06-repack-under-pressure/README.md new file mode 100644 index 000000000..940e252be --- /dev/null +++ b/docs/evidence/2026-10-06-repack-under-pressure/README.md @@ -0,0 +1,40 @@ +# Repacked weights vs the mapped file under memory pressure, Pixel 6a (6 Oct 2026) + +Why the adaptive-repack PR doesn't pick the mapped file from the memory free at load time. + +## Setup + +Pixel 6a (6 GB RAM, 5.46 GiB usable), Android 16, dev build served by Metro, Qwen3-4B-Instruct-2507 Q4_K_M +(2.33 GiB file), Standard library. Each run: battery cooled to 33 °C or below and thermal status 0; then five +Chrome tabs and twelve apps opened (YouTube, Maps, Photos, Camera, Gmail, Play Store, Docs, Calendar, +Messages, Contacts, Clock, Settings); BOAR force-stopped and restarted; `npm run eval:device -- --models +qwen3-4b --no-reload` (standard set, 17 questions, Succinct). + +| Run | Code | Free before BOAR started | Weights | After the load | +|---|---|---|---|---| +| `mapped-std` | the PR's first version (mapped file when the file is larger than what's free) | 2.1 GB (1.8 GiB at the load) | mapped file | 2.8 GB free, 0.26 GB of the app anonymous, 0.37 GB in swap | +| `repacked-std` | #72 at 9670543f (repacks every model whose file fits the budget) | 2.1 GB | repacked | 0.7 GB free, swap nearly full (71 MB left), 1.9 GB of the app in swap | + +## Results + +| | mapped file | repacked | +|---|---|---| +| Answered | 17/17 | 17/17 | +| First word, median | 25.4 s | **10.2 s** | +| First word, slowest | 74.7 s | **29.8 s** | +| Writing speed, median (slowest) | 5.3 (4.9) tok/s | 4.5 (3.9) tok/s | +| Total, median | 43.6 s | **25.3 s** | + +With far less memory to spare, the repacked model was still ~2.5× faster to the first word and didn't stall. +On the same day, three repacked loads with 1.6–2.4 GB free (more apps open each time) all succeeded: +Android stopped background apps instead of BOAR. The load-time kills seen on 5 Oct (~0.8 GB free) weren't +reproduced. So the PR keeps repacking by default, and only switches to the mapped file after a repacked load +was actually killed or an answer actually stalled. + +## Caveats + +- One phone, one run per arm; `mapped-std` ran first, `repacked-std` ~30 minutes later after cooling back to + 33 °C. Memory pressure from the same apps, but not identical states. +- A killed repacked load was verified by writing the marker it leaves (`files/model-load.marker.json` with + `"repacked": true`) and restarting: the 4B loaded as the mapped file and stayed the active model. The stall + fallback is covered by unit tests only; no stall happened in these runs. diff --git a/docs/evidence/2026-10-06-repack-under-pressure/mapped-std/eval-2026-10-06T02-57-23-093Z.answers.md b/docs/evidence/2026-10-06-repack-under-pressure/mapped-std/eval-2026-10-06T02-57-23-093Z.answers.md new file mode 100644 index 000000000..9308e8051 --- /dev/null +++ b/docs/evidence/2026-10-06-repack-under-pressure/mapped-std/eval-2026-10-06T02-57-23-093Z.answers.md @@ -0,0 +1,140 @@ +## greeting-1 (greeting) + +> hey, what's up? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Hey! Not much, just here ready to help with whatever you need. What's on your mind? + +## factual-1 (factual) + +> What is the capital of Australia? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +The capital of Australia is Canberra. + +## factual-2 (factual) + +> Who proposed the theory of evolution by natural selection? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Evolution + +Charles Darwin proposed the theory of evolution by natural selection. [1] + +## explanation-1 (explanation) + +> How do vaccines work? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Vaccine + +Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed virus or a piece of its surface protein—into the body. This triggers the immune system to produce antibodies and memory cells, so if the person is later exposed to the real disease, their immune system can respond quickly and effectively [1]. + +## explanation-2 (explanation) + +> Explain photosynthesis in simple terms. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Photosynthesis + +Photosynthesis is how plants use sunlight to turn carbon dioxide and water into sugar (food) and oxygen as a byproduct. The sunlight powers the process, which plants use to grow and live [1]. + +## comparison-1 (comparison) + +> Compare the French Revolution and the Industrial Revolution. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: French Revolution, Industrial Revolution + +The French Revolution (1789–1799) was a political and social upheaval that transformed France’s government and introduced democratic ideals like equality and liberty [1], while the Industrial Revolution (beginning c. 1760) was a period of economic transformation marked by mechanized manufacturing, urbanization, and technological innovation [2]. The former changed societal structures and political systems, whereas the latter reshaped economies and daily life through industrialization. + +## comparison-2 (comparison) + +> Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Supply and demand, Behavioral economics + +The supply and demand model assumes rational, informed buyers and sellers who act to maximize utility, leading to equilibrium prices when supply equals demand [1]. Behavioral economics complicates this by showing that individuals often make irrational, emotionally driven, or biased decisions—such as anchoring, loss aversion, or herd behavior—leading to market anomalies like price bubbles or crashes that the simple model cannot explain. The model breaks down when real-world factors like psychology, social influence, or cognitive biases significantly alter purchasing or production decisions. + +## synthesis-1 (synthesis) + +> What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Industrial Revolution, Agricultural revolution + +Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society by transforming how people produced food and goods, leading to permanent settlements and sustained population growth [1]. The Agricultural Revolution gradually changed daily life over centuries, as farming replaced hunting and gathering, while the Industrial Revolution dramatically accelerated changes in a few generations, introducing factories, mass production, and urbanization. The Industrial Revolution thus transformed daily life more rapidly and intensely than the earlier Agricultural Revolution. + +## synthesis-2 (synthesis) + +> How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Immune system, Vaccine + +The immune system detects pathogens and mounts a response by identifying foreign molecules, triggering defenses that eliminate the threat and remember the pathogen for future encounters. Vaccines mimic this process by introducing a harmless version of a pathogen—such as a weakened or inactivated form or a surface protein—that stimulates the immune system to recognize and respond to the specific pathogen without causing disease. This creates memory cells so that upon actual exposure, the immune system responds faster and more effectively [2]. + +## synthesis-3 (synthesis) + +> Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Biodiversity loss, Amazon rainforest + +The Amazon rainforest is vital for global climate regulation because it absorbs large amounts of carbon dioxide and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens ecosystem resilience, such as reduced ability to recover from disturbances, disruption of climate feedback loops, and loss of natural pest and disease controls, which can destabilize regional and global environmental systems. + +## reasoning-1 (reasoning) + +> A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. +The train arrives at 6:15 pm. + +## reasoning-2 (reasoning) + +> A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Total used: 2GB (OS/app) + 200MB (embedding) + 1.5GB (KV cache) = 3.7GB. +Remaining: 12GB - 3.7GB = 8.3GB. +A 9GB LLM model would exceed the available RAM, so it cannot be loaded fully into RAM. + +## reasoning-3 (reasoning) + +> If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Minimum disk footprint: 8B × 1 byte = 8 bytes per token. +RAM usage doesn’t scale with 100B because only 8B per token are actively activated (via expert selection), not all 100B parameters stored simultaneously — only the selected experts’ parameters are loaded into memory, not the full model. + +## grounded-1 (retrieval-grounded) + +> Why did the Western Roman Empire fall? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Fall of the Western Roman Empire, Byzantine Empire + +The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, political instability, and external pressures from invading barbarian groups. Climatic changes and widespread diseases also contributed to the collapse [1]. The Eastern Roman (Byzantine) Empire survived until 1453, continuing the Roman tradition in the East [2]. + +## grounded-2 (retrieval-grounded) + +> What is a black hole and how does one form? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Black hole + +A black hole is an astronomical object so dense that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, squeezing its mass into an extremely small volume, creating a region where spacetime is severely warped. + +## no-kb-1 (no-kb-content) + +> Who was Napoleon Bonaparte? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution, became the first Consul of France, and later Emperor of the French. He expanded the French Empire across Europe, implemented significant legal and administrative reforms, and was ultimately defeated in 1815 after the Battle of Waterloo. + +## no-kb-2 (no-kb-content) + +> How do antibiotics work, and why does antibiotic resistance develop? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Antimicrobial resistance + +Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis, protein production, or DNA replication, which disrupts their ability to survive and multiply [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to misuse, overprescription, or improper disposal of antibiotics. These resistant bacteria can then spread, reducing the effectiveness of treatments. + diff --git a/docs/evidence/2026-10-06-repack-under-pressure/mapped-std/eval-2026-10-06T02-57-23-093Z.jsonl b/docs/evidence/2026-10-06-repack-under-pressure/mapped-std/eval-2026-10-06T02-57-23-093Z.jsonl new file mode 100644 index 000000000..995b450c2 --- /dev/null +++ b/docs/evidence/2026-10-06-repack-under-pressure/mapped-std/eval-2026-10-06T02-57-23-093Z.jsonl @@ -0,0 +1,17 @@ +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"greeting","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":17730.283822000027,"generationLatencyMs":3276.5763769745827,"totalLatencyMs":21019.14718604088,"tokensGenerated":21,"tokPerSec":6.4091287929598915,"peakRssBytes":2687193088,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"greeting-1","category":"greeting","query":"hey, what's up?","answer":"Hey! Not much, just here ready to help with whatever you need. What's on your mind?","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791255464224} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":1181.8941649794579,"generationLatencyMs":1007.5526130199432,"totalLatencyMs":2583.261475980282,"tokensGenerated":7,"tokPerSec":6.947528009499037,"peakRssBytes":2779545600,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":393.1999109983444,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-1","category":"factual","query":"What is the capital of Australia?","answer":"The capital of Australia is Canberra.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791255466993} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":25414.29416203499,"generationLatencyMs":2088.9762780070305,"totalLatencyMs":27798.75595396757,"tokensGenerated":14,"tokPerSec":6.701847286344763,"peakRssBytes":2819846144,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":295.03556299209595,"contextTokensBefore":111,"contextTokensAfter":111,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-2","category":"factual","query":"Who proposed the theory of evolution by natural selection?","answer":"Charles Darwin proposed the theory of evolution by natural selection. [1]","promptFormat":"chat-template","retrievedTitles":["Evolution"],"expectedKbTitles":["Evolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791255494855} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9102.31372499466,"generationLatencyMs":10738.166550040245,"totalLatencyMs":20089.664112985134,"tokensGenerated":68,"tokPerSec":6.332552180404125,"peakRssBytes":2829864960,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":248.91516202688217,"contextTokensBefore":154,"contextTokensAfter":102,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-1","category":"explanation","query":"How do vaccines work?","answer":"Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed virus or a piece of its surface protein—into the body. This triggers the immune system to produce antibodies and memory cells, so if the person is later exposed to the real disease, their immune system can respond quickly and effectively [1].","promptFormat":"chat-template","retrievedTitles":["Vaccine"],"expectedKbTitles":["Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791255515024} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":16852.39034897089,"generationLatencyMs":6672.74243503809,"totalLatencyMs":23844.261324048042,"tokensGenerated":41,"tokPerSec":6.144400207134019,"peakRssBytes":2842890240,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":318.52209401130676,"contextTokensBefore":372,"contextTokensAfter":181,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-2","category":"explanation","query":"Explain photosynthesis in simple terms.","answer":"Photosynthesis is how plants use sunlight to turn carbon dioxide and water into sugar (food) and oxygen as a byproduct. The sunlight powers the process, which plants use to grow and live [1].","promptFormat":"chat-template","retrievedTitles":["Photosynthesis"],"expectedKbTitles":["Photosynthesis"],"expectedKbHit":true,"timedOut":false,"createdAt":1791255538907} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"compare","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":31330.164036989212,"generationLatencyMs":18137.342375040054,"totalLatencyMs":49805.26373702288,"tokensGenerated":95,"tokPerSec":5.237812576705589,"peakRssBytes":2855342080,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":337.50166803598404,"contextTokensBefore":286,"contextTokensAfter":286,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-1","category":"comparison","query":"Compare the French Revolution and the Industrial Revolution.","answer":"The French Revolution (1789–1799) was a political and social upheaval that transformed France’s government and introduced democratic ideals like equality and liberty [1], while the Industrial Revolution (beginning c. 1760) was a period of economic transformation marked by mechanized manufacturing, urbanization, and technological innovation [2]. The former changed societal structures and political systems, whereas the latter reshaped economies and daily life through industrialization.","promptFormat":"chat-template","retrievedTitles":["French Revolution","Industrial Revolution"],"expectedKbTitles":["French Revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791255588799} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":23738.30478000641,"generationLatencyMs":19347.670622944832,"totalLatencyMs":43571.89186698198,"tokensGenerated":102,"tokPerSec":5.271952473650028,"peakRssBytes":2864877568,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":485.44071501493454,"contextTokensBefore":188,"contextTokensAfter":188,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-2","category":"comparison","query":"Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down?","answer":"The supply and demand model assumes rational, informed buyers and sellers who act to maximize utility, leading to equilibrium prices when supply equals demand [1]. Behavioral economics complicates this by showing that individuals often make irrational, emotionally driven, or biased decisions—such as anchoring, loss aversion, or herd behavior—leading to market anomalies like price bubbles or crashes that the simple model cannot explain. The model breaks down when real-world factors like psychology, social influence, or cognitive biases significantly alter purchasing or production decisions.","promptFormat":"chat-template","retrievedTitles":["Supply and demand","Behavioral economics"],"expectedKbTitles":["Supply and demand","Behavioral economics"],"expectedKbHit":true,"timedOut":false,"createdAt":1791255632492} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":35023.910824000835,"generationLatencyMs":17113.329639017582,"totalLatencyMs":52727.7291110158,"tokensGenerated":89,"tokPerSec":5.20062441835306,"peakRssBytes":2848919552,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":589.8246260285378,"contextTokensBefore":308,"contextTokensAfter":307,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-1","category":"synthesis","query":"What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life?","answer":"Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society by transforming how people produced food and goods, leading to permanent settlements and sustained population growth [1]. The Agricultural Revolution gradually changed daily life over centuries, as farming replaced hunting and gathering, while the Industrial Revolution dramatically accelerated changes in a few generations, introducing factories, mass production, and urbanization. The Industrial Revolution thus transformed daily life more rapidly and intensely than the earlier Agricultural Revolution.","promptFormat":"chat-template","retrievedTitles":["Industrial Revolution","Agricultural revolution"],"expectedKbTitles":["Agricultural revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791255685372} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":36227.66448700428,"generationLatencyMs":18516.089323997498,"totalLatencyMs":55161.381699979305,"tokensGenerated":95,"tokPerSec":5.130673023751116,"peakRssBytes":2852872192,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":417.14786797761917,"contextTokensBefore":330,"contextTokensAfter":298,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-2","category":"synthesis","query":"How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism.","answer":"The immune system detects pathogens and mounts a response by identifying foreign molecules, triggering defenses that eliminate the threat and remember the pathogen for future encounters. Vaccines mimic this process by introducing a harmless version of a pathogen—such as a weakened or inactivated form or a surface protein—that stimulates the immune system to recognize and respond to the specific pathogen without causing disease. This creates memory cells so that upon actual exposure, the immune system responds faster and more effectively [2].","promptFormat":"chat-template","retrievedTitles":["Immune system","Vaccine"],"expectedKbTitles":["Immune system","Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791255740674} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":34820.53574299812,"generationLatencyMs":14577.482836008072,"totalLatencyMs":49806.632185041904,"tokensGenerated":72,"tokPerSec":4.939124320019891,"peakRssBytes":2848866304,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":407.9243569970131,"contextTokensBefore":310,"contextTokensAfter":216,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-3","category":"synthesis","query":"Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species?","answer":"The Amazon rainforest is vital for global climate regulation because it absorbs large amounts of carbon dioxide and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens ecosystem resilience, such as reduced ability to recover from disturbances, disruption of climate feedback loops, and loss of natural pest and disease controls, which can destabilize regional and global environmental systems.","promptFormat":"chat-template","retrievedTitles":["Biodiversity loss","Amazon rainforest"],"expectedKbTitles":["Amazon rainforest"],"expectedKbHit":true,"timedOut":false,"createdAt":1791255790629} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"chat","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":53172.804143965244,"generationLatencyMs":10372.954595029354,"totalLatencyMs":64297.053294956684,"tokensGenerated":58,"tokPerSec":5.591463788706179,"peakRssBytes":2880475136,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":750.3787439465523,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-1","category":"reasoning","query":"A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive?","answer":"Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. \nThe train arrives at 6:15 pm.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791255856152} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"calculate","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":10492.09672498703,"generationLatencyMs":14585.923794984818,"totalLatencyMs":25078.68629205227,"tokensGenerated":79,"tokPerSec":5.416180771982583,"peakRssBytes":2883874816,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-2","category":"reasoning","query":"A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM?","answer":"Total used: 2GB (OS/app) + 200MB (embedding) + 1.5GB (KV cache) = 3.7GB. \nRemaining: 12GB - 3.7GB = 8.3GB. \nA 9GB LLM model would exceed the available RAM, so it cannot be loaded fully into RAM.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791255881400} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9607.715255022049,"generationLatencyMs":14213.37317597866,"totalLatencyMs":24337.49883198738,"tokensGenerated":72,"tokPerSec":5.065651841301384,"peakRssBytes":2897674240,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":515.5834559798241,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-3","category":"reasoning","query":"If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure?","answer":"Minimum disk footprint: 8B × 1 byte = 8 bytes per token. \nRAM usage doesn’t scale with 100B because only 8B per token are actively activated (via expert selection), not all 100B parameters stored simultaneously — only the selected experts’ parameters are loaded into memory, not the full model.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791255905920} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":70273.9792419672,"generationLatencyMs":14143.484544992447,"totalLatencyMs":84979.22416198254,"tokensGenerated":71,"tokPerSec":5.0199793250481415,"peakRssBytes":2939289600,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":561.0416260361671,"contextTokensBefore":396,"contextTokensAfter":359,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-1","category":"retrieval-grounded","query":"Why did the Western Roman Empire fall?","answer":"The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, political instability, and external pressures from invading barbarian groups. Climatic changes and widespread diseases also contributed to the collapse [1]. The Eastern Roman (Byzantine) Empire survived until 1453, continuing the Roman tradition in the East [2].","promptFormat":"chat-template","retrievedTitles":["Fall of the Western Roman Empire","Byzantine Empire"],"expectedKbTitles":["Fall of the Western Roman Empire"],"expectedKbHit":true,"timedOut":false,"createdAt":1791255991071} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":74655.19787198305,"generationLatencyMs":11552.151210010052,"totalLatencyMs":86795.74906599522,"tokensGenerated":60,"tokPerSec":5.193837832386526,"peakRssBytes":2865733632,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":587.2962650060654,"contextTokensBefore":155,"contextTokensAfter":121,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-2","category":"retrieval-grounded","query":"What is a black hole and how does one form?","answer":"A black hole is an astronomical object so dense that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, squeezing its mass into an extremely small volume, creating a region where spacetime is severely warped.","promptFormat":"chat-template","retrievedTitles":["Black hole"],"expectedKbTitles":["Black hole"],"expectedKbHit":true,"timedOut":false,"createdAt":1791256078258} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":19621.371224999428,"generationLatencyMs":12714.941534996033,"totalLatencyMs":32867.3134920001,"tokensGenerated":68,"tokPerSec":5.34803874739336,"peakRssBytes":2827153408,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":530.4024260044098,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-1","category":"no-kb-content","query":"Who was Napoleon Bonaparte?","answer":"Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution, became the first Consul of France, and later Emperor of the French. He expanded the French Empire across Europe, implemented significant legal and administrative reforms, and was ultimately defeated in 1815 after the Battle of Waterloo.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791256111511} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":44421.51959300041,"generationLatencyMs":16644.44914597273,"totalLatencyMs":61497.41730695963,"tokensGenerated":85,"tokPerSec":5.1068076362603145,"peakRssBytes":2831142912,"outcome":"success","runId":"eval-2026-10-06T02-57-23-093Z","evalSetVersion":"1","retrievalMs":430.9143880009651,"contextTokensBefore":175,"contextTokensAfter":147,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-2","category":"no-kb-content","query":"How do antibiotics work, and why does antibiotic resistance develop?","answer":"Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis, protein production, or DNA replication, which disrupts their ability to survive and multiply [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to misuse, overprescription, or improper disposal of antibiotics. These resistant bacteria can then spread, reducing the effectiveness of treatments.","promptFormat":"chat-template","retrievedTitles":["Antimicrobial resistance"],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791256173236} diff --git a/docs/evidence/2026-10-06-repack-under-pressure/mapped-std/eval-2026-10-06T02-57-23-093Z.report.md b/docs/evidence/2026-10-06-repack-under-pressure/mapped-std/eval-2026-10-06T02-57-23-093Z.report.md new file mode 100644 index 000000000..983d3525e --- /dev/null +++ b/docs/evidence/2026-10-06-repack-under-pressure/mapped-std/eval-2026-10-06T02-57-23-093Z.report.md @@ -0,0 +1,17 @@ +BOAR Device Evaluation — eval-2026-10-06T02-57-23-093Z (set v1, 17 queries) +──────────────────────────────────────────────────────────── + +Qwen3-4B-Instruct-2507 (Q4_K_M) + Config: model:qwen3-4b-instruct-2507-q4km · prompt chat-template + Queries: 17 ok · 0 failed · 0 cancelled (of 17) + TTFT: avg 30.2s · p50 25.4s + Model load: none (already resident) + Generation: avg 12.1s · p50 14.1s + Tokens/sec: avg 5.6 · p50 5.3 + Search time: avg 0.4s · p50 0.4s (before the model starts) + Context tokens: 253 → 211 avg (trimmed to the answering sentences) + Total per query: avg 42.7s · p50 43.6s + Peak RSS: 2.74 GB + Residency: cold 0 · switched 0 · resident 17 + Model switches: 0 within plans · 0 across messages + Retrieval: used 11/17 · expected KB article found 10/10 diff --git a/docs/evidence/2026-10-06-repack-under-pressure/mapped-std/eval-2026-10-06T02-57-23-093Z.status.json b/docs/evidence/2026-10-06-repack-under-pressure/mapped-std/eval-2026-10-06T02-57-23-093Z.status.json new file mode 100644 index 000000000..04c12eaeb --- /dev/null +++ b/docs/evidence/2026-10-06-repack-under-pressure/mapped-std/eval-2026-10-06T02-57-23-093Z.status.json @@ -0,0 +1,17 @@ +{ + "requestId": "req-20261006t025719-wiv4", + "state": "done", + "configs": [ + "model:qwen3-4b-instruct-2507-q4km" + ], + "total": 17, + "completed": 17, + "runId": "eval-2026-10-06T02-57-23-093Z", + "resultPath": "files/eval/eval-2026-10-06T02-57-23-093Z.jsonl", + "stopped": false, + "installedModels": [ + "qwen3-4b-instruct-2507-q4km", + "qwen2.5-1.5b-instruct-q4km" + ], + "updatedAt": 1791256173477 +} diff --git a/docs/evidence/2026-10-06-repack-under-pressure/repacked-std/eval-2026-10-06T03-29-11-385Z.answers.md b/docs/evidence/2026-10-06-repack-under-pressure/repacked-std/eval-2026-10-06T03-29-11-385Z.answers.md new file mode 100644 index 000000000..b9085d7c7 --- /dev/null +++ b/docs/evidence/2026-10-06-repack-under-pressure/repacked-std/eval-2026-10-06T03-29-11-385Z.answers.md @@ -0,0 +1,141 @@ +## greeting-1 (greeting) + +> hey, what's up? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Hey! Just checking in. How can I help you today? + +## factual-1 (factual) + +> What is the capital of Australia? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +The capital of Australia is Canberra. + +## factual-2 (factual) + +> Who proposed the theory of evolution by natural selection? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Evolution + +Charles Darwin proposed the theory of evolution by natural selection. [1] + +## explanation-1 (explanation) + +> How do vaccines work? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Vaccine + +Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed virus, or a piece of its surface protein—into the body. This triggers the immune system to produce antibodies and memory cells that recognize and fight the actual pathogen if encountered later [1]. This prepares the body to respond quickly and effectively, preventing or reducing the severity of disease. + +## explanation-2 (explanation) + +> Explain photosynthesis in simple terms. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Photosynthesis + +Photosynthesis is how plants use sunlight to turn carbon dioxide and water into food (like sugar) and oxygen. The oxygen is released into the air as a byproduct. [1] + +## comparison-1 (comparison) + +> Compare the French Revolution and the Industrial Revolution. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: French Revolution, Industrial Revolution + +The French Revolution (1789–1799) was a political and social upheaval that transformed France’s government and introduced democratic ideals, while the Industrial Revolution (beginning c. 1760) was an economic transformation marked by mechanization, urbanization, and industrial growth [1][2]. The French Revolution focused on political rights and equality, whereas the Industrial Revolution shifted production methods and global economies. Both were pivotal in shaping modern history, but they addressed different domains: governance versus economic systems. + +## comparison-2 (comparison) + +> Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Supply and demand, Behavioral economics + +The supply and demand model assumes rational, informed buyers and sellers who act to maximize utility, leading to equilibrium prices [1]. Behavioral economics complicates this by showing that individuals often make irrational, biased decisions—such as overpaying for perceived value or ignoring price signals—due to cognitive biases, emotions, or social influences [2]. This breaks down the model in real-world markets where prices don’t always reflect true value or equilibrium, and where demand can be inelastic or volatile due to psychological factors rather than pure utility maximization. + +## synthesis-1 (synthesis) + +> What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Industrial Revolution, Agricultural revolution + +Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society by transforming how people produced food and goods, leading to permanent settlements and population growth [1]. The Agricultural Revolution slowly changed daily life over centuries, with gradual adoption of farming practices, while the Industrial Revolution brought rapid, dramatic changes in daily life within a few generations due to mechanization, urbanization, and new technologies. + +## synthesis-2 (synthesis) + +> How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism. + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Immune system, Vaccine + +The immune system detects pathogens and mounts a response by identifying foreign molecules, activating innate and adaptive defenses, and creating memory cells for future encounters. A vaccine mimics this by introducing a harmless form of a pathogen (like a weakened virus or protein) to stimulate the adaptive immune system to recognize and remember it, enabling a faster, stronger response if the real pathogen ever appears [2]. This prepares the body without causing disease. + +## synthesis-3 (synthesis) + +> Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Biodiversity loss, Amazon rainforest + +The Amazon rainforest is crucial for global climate regulation because it absorbs large amounts of carbon dioxide and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens ecosystem resilience, disrupts climate stability through reduced carbon sequestration, and diminishes essential services like water cycling and soil fertility, which affect both local and global environmental systems. + +## reasoning-1 (reasoning) + +> A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. +The train arrives at 6:15 pm. + +## reasoning-2 (reasoning) + +> A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Total used: 2GB (OS/app) + 200MB (embedding) + 1.5GB (KV cache) = 3.2GB. +Left for LLM weights: 12GB – 3.2GB = 8.8GB. +A 9GB model would exceed the available memory and not fit fully in RAM. + +## reasoning-3 (reasoning) + +> If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Minimum disk footprint: 8B per token × 1 byte/Q8 = 8 bytes per token. For a single token, it's 8 bytes; for a sequence of N tokens, it's 8N bytes. Total disk usage scales linearly with token count, not with the 100B total parameters. + +RAM usage doesn’t scale with 100B because only 8B per token is actively accessed at inference — the model uses expert routing and sparse activation, so only a subset of parameters is loaded into memory at any time, not all 100B. + +## grounded-1 (retrieval-grounded) + +> Why did the Western Roman Empire fall? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Fall of the Western Roman Empire, Byzantine Empire + +The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, population decline, political instability, and invasions by Germanic tribes and other barbarian groups. Climatic changes and widespread diseases also contributed to the collapse [1]. The Eastern Roman (Byzantine) Empire survived until 1453, continuing the Roman tradition in the East [2]. + +## grounded-2 (retrieval-grounded) + +> What is a black hole and how does one form? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Black hole + +A black hole is an astronomical object so compact that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, squeezing its mass into an extremely small volume, creating a region where spacetime is severely curved. + +## no-kb-1 (no-kb-content) + +> Who was Napoleon Bonaparte? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: none + +Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He led France in a series of military campaigns across Europe, expanding its influence and reshaping the continent's political landscape, and later established a major impact on law, governance, and education through the Napoleonic Code. + +## no-kb-2 (no-kb-content) + +> How do antibiotics work, and why does antibiotic resistance develop? + +### Qwen3-4B-Instruct-2507 (Q4_K_M) — qwen3-4b-instruct-2507-q4km · chat-template · success · retrieval: Antimicrobial resistance + +Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis, protein production, or DNA replication, disrupting their ability to survive and multiply [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to misuse, overprescription, or improper disposal of antimicrobials, leading to the spread of resistant strains. + diff --git a/docs/evidence/2026-10-06-repack-under-pressure/repacked-std/eval-2026-10-06T03-29-11-385Z.jsonl b/docs/evidence/2026-10-06-repack-under-pressure/repacked-std/eval-2026-10-06T03-29-11-385Z.jsonl new file mode 100644 index 000000000..49c3c810d --- /dev/null +++ b/docs/evidence/2026-10-06-repack-under-pressure/repacked-std/eval-2026-10-06T03-29-11-385Z.jsonl @@ -0,0 +1,17 @@ +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"greeting","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":10203.150802969933,"generationLatencyMs":2112.393800020218,"totalLatencyMs":12330.455165982246,"tokensGenerated":13,"tokPerSec":6.154155536659677,"peakRssBytes":2691964928,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"greeting-1","category":"greeting","query":"hey, what's up?","answer":"Hey! Just checking in. How can I help you today?","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791257363827} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":619.2413740158081,"generationLatencyMs":977.4587410092354,"totalLatencyMs":2003.0545260310173,"tokensGenerated":7,"tokPerSec":7.16142759414319,"peakRssBytes":2803081216,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":405.62902903556824,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-1","category":"factual","query":"What is the capital of Australia?","answer":"The capital of Australia is Canberra.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791257366155} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":10700.251307964325,"generationLatencyMs":2303.1382660269737,"totalLatencyMs":13301.656459987164,"tokensGenerated":14,"tokPerSec":6.078662408814337,"peakRssBytes":2858733568,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":297.905110001564,"contextTokensBefore":111,"contextTokensAfter":111,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"factual-2","category":"factual","query":"Who proposed the theory of evolution by natural selection?","answer":"Charles Darwin proposed the theory of evolution by natural selection. [1]","promptFormat":"chat-template","retrievedTitles":["Evolution"],"expectedKbTitles":["Evolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791257379519} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":3829.7363709807396,"generationLatencyMs":13562.176602005959,"totalLatencyMs":17639.33956795931,"tokensGenerated":77,"tokPerSec":5.677554736207392,"peakRssBytes":2868690944,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":247.15446001291275,"contextTokensBefore":154,"contextTokensAfter":102,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-1","category":"explanation","query":"How do vaccines work?","answer":"Vaccines work by introducing a harmless component of a pathogen—such as a weakened or killed virus, or a piece of its surface protein—into the body. This triggers the immune system to produce antibodies and memory cells that recognize and fight the actual pathogen if encountered later [1]. This prepares the body to respond quickly and effectively, preventing or reducing the severity of disease.","promptFormat":"chat-template","retrievedTitles":["Vaccine"],"expectedKbTitles":["Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791257397228} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":7560.562462985516,"generationLatencyMs":7093.935061991215,"totalLatencyMs":14999.751756966114,"tokensGenerated":37,"tokPerSec":5.215722962878994,"peakRssBytes":2890080256,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":344.65437799692154,"contextTokensBefore":372,"contextTokensAfter":181,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"explanation-2","category":"explanation","query":"Explain photosynthesis in simple terms.","answer":"Photosynthesis is how plants use sunlight to turn carbon dioxide and water into food (like sugar) and oxygen. The oxygen is released into the air as a byproduct. [1]","promptFormat":"chat-template","retrievedTitles":["Photosynthesis"],"expectedKbTitles":["Photosynthesis"],"expectedKbHit":true,"timedOut":false,"createdAt":1791257412268} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"compare","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":10844.122766971588,"generationLatencyMs":25818.50758099556,"totalLatencyMs":37059.35320901871,"tokensGenerated":106,"tokPerSec":4.105582000333136,"peakRssBytes":2913394688,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":396.39005595445633,"contextTokensBefore":286,"contextTokensAfter":286,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-1","category":"comparison","query":"Compare the French Revolution and the Industrial Revolution.","answer":"The French Revolution (1789–1799) was a political and social upheaval that transformed France’s government and introduced democratic ideals, while the Industrial Revolution (beginning c. 1760) was an economic transformation marked by mechanization, urbanization, and industrial growth [1][2]. The French Revolution focused on political rights and equality, whereas the Industrial Revolution shifted production methods and global economies. Both were pivotal in shaping modern history, but they addressed different domains: governance versus economic systems.","promptFormat":"chat-template","retrievedTitles":["French Revolution","Industrial Revolution"],"expectedKbTitles":["French Revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791257449443} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":10469.430710017681,"generationLatencyMs":24351.979829013348,"totalLatencyMs":35348.85125398636,"tokensGenerated":107,"tokPerSec":4.393893258424863,"peakRssBytes":2926813184,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":526.9552810192108,"contextTokensBefore":188,"contextTokensAfter":188,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"comparison-2","category":"comparison","query":"Contrast how supply and demand explains price changes with how behavioral economics complicates that picture — where does the simple model break down?","answer":"The supply and demand model assumes rational, informed buyers and sellers who act to maximize utility, leading to equilibrium prices [1]. Behavioral economics complicates this by showing that individuals often make irrational, biased decisions—such as overpaying for perceived value or ignoring price signals—due to cognitive biases, emotions, or social influences [2]. This breaks down the model in real-world markets where prices don’t always reflect true value or equilibrium, and where demand can be inelastic or volatile due to psychological factors rather than pure utility maximization.","promptFormat":"chat-template","retrievedTitles":["Supply and demand","Behavioral economics"],"expectedKbTitles":["Supply and demand","Behavioral economics"],"expectedKbHit":true,"timedOut":false,"createdAt":1791257484914} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":14474.300014972687,"generationLatencyMs":17285.079231977463,"totalLatencyMs":32206.286310970783,"tokensGenerated":77,"tokPerSec":4.454709114526342,"peakRssBytes":2914033664,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":446.2185059785843,"contextTokensBefore":308,"contextTokensAfter":307,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-1","category":"synthesis","query":"What do the Agricultural Revolution and the Industrial Revolution have in common as turning points in human history, and how did they differ in how quickly they changed daily life?","answer":"Both the Agricultural and Industrial Revolutions marked fundamental shifts in human society by transforming how people produced food and goods, leading to permanent settlements and population growth [1]. The Agricultural Revolution slowly changed daily life over centuries, with gradual adoption of farming practices, while the Industrial Revolution brought rapid, dramatic changes in daily life within a few generations due to mechanization, urbanization, and new technologies.","promptFormat":"chat-template","retrievedTitles":["Industrial Revolution","Agricultural revolution"],"expectedKbTitles":["Agricultural revolution","Industrial Revolution"],"expectedKbHit":true,"timedOut":false,"createdAt":1791257517254} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":14666.426195979118,"generationLatencyMs":18875.68742799759,"totalLatencyMs":33951.69382297993,"tokensGenerated":86,"tokPerSec":4.55612545651924,"peakRssBytes":2928730112,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":409.1256110072136,"contextTokensBefore":330,"contextTokensAfter":298,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-2","category":"synthesis","query":"How does the immune system's response to a pathogen relate to how a vaccine works — walk through the mechanism.","answer":"The immune system detects pathogens and mounts a response by identifying foreign molecules, activating innate and adaptive defenses, and creating memory cells for future encounters. A vaccine mimics this by introducing a harmless form of a pathogen (like a weakened virus or protein) to stimulate the adaptive immune system to recognize and remember it, enabling a faster, stronger response if the real pathogen ever appears [2]. This prepares the body without causing disease.","promptFormat":"chat-template","retrievedTitles":["Immune system","Vaccine"],"expectedKbTitles":["Immune system","Vaccine"],"expectedKbHit":true,"timedOut":false,"createdAt":1791257551349} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":14127.547737002373,"generationLatencyMs":15604.757169008255,"totalLatencyMs":30204.461969971657,"tokensGenerated":70,"tokPerSec":4.485811553608993,"peakRssBytes":2996375552,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":471.6721190214157,"contextTokensBefore":310,"contextTokensAfter":216,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"synthesis-3","category":"synthesis","query":"Why is the Amazon rainforest considered important for global climate, and what does biodiversity loss there actually threaten beyond the obvious loss of species?","answer":"The Amazon rainforest is crucial for global climate regulation because it absorbs large amounts of carbon dioxide and releases oxygen, helping mitigate climate change [1]. Biodiversity loss there threatens ecosystem resilience, disrupts climate stability through reduced carbon sequestration, and diminishes essential services like water cycling and soil fertility, which affect both local and global environmental systems.","promptFormat":"chat-template","retrievedTitles":["Biodiversity loss","Amazon rainforest"],"expectedKbTitles":["Amazon rainforest"],"expectedKbHit":true,"timedOut":false,"createdAt":1791257581699} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"chat","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":9233.407882034779,"generationLatencyMs":11746.781825959682,"totalLatencyMs":21395.001150012016,"tokensGenerated":58,"tokPerSec":4.937522536753298,"peakRssBytes":2976555008,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":414.2077639698982,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-1","category":"reasoning","query":"A train leaves at 3:40 pm and the trip takes 2 hours and 35 minutes. What time does it arrive?","answer":"Add 2 hours to 3:40 pm to get 5:40 pm. Then add 35 minutes: 5:40 pm + 35 minutes = 6:15 pm. \nThe train arrives at 6:15 pm.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791257603250} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"calculate","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:skipped-task-not-knowledge-based","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":4962.220704972744,"generationLatencyMs":17679.42896401882,"totalLatencyMs":22642.187267005444,"tokensGenerated":78,"tokPerSec":4.411907203493145,"peakRssBytes":2973212672,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-2","category":"reasoning","query":"A device has a 12GB RAM budget. The OS and app overhead take 2GB, the embedding model needs 200MB, and the LLM's KV cache needs 1.5GB. How much is left for the LLM's weights, and would a 9GB model fit if loaded fully into RAM?","answer":"Total used: 2GB (OS/app) + 200MB (embedding) + 1.5GB (KV cache) = 3.2GB. \nLeft for LLM weights: 12GB – 3.2GB = 8.8GB. \nA 9GB model would exceed the available memory and not fit fully in RAM.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791257626066} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":4069.672038972378,"generationLatencyMs":28711.20525497198,"totalLatencyMs":33246.96384203434,"tokensGenerated":123,"tokPerSec":4.28404168016248,"peakRssBytes":2985725952,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":465.2252609729767,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"reasoning-3","category":"reasoning","query":"If a Mixture-of-Experts model has 100B total parameters but only activates 8B per token, and each parameter needs 1 byte at Q8, what's the minimum disk footprint, and why doesn't RAM usage scale with the 100B figure?","answer":"Minimum disk footprint: 8B per token × 1 byte/Q8 = 8 bytes per token. For a single token, it's 8 bytes; for a sequence of N tokens, it's 8N bytes. Total disk usage scales linearly with token count, not with the 100B total parameters.\n\nRAM usage doesn’t scale with 100B because only 8B per token is actively accessed at inference — the model uses expert routing and sparse activation, so only a subset of parameters is loaded into memory at any time, not all 100B.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791257659478} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":29776.950576007366,"generationLatencyMs":20217.021615982056,"totalLatencyMs":50410.14695703983,"tokensGenerated":78,"tokPerSec":3.8581350646793133,"peakRssBytes":3028447232,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":415.80061799287796,"contextTokensBefore":396,"contextTokensAfter":359,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-1","category":"retrieval-grounded","query":"Why did the Western Roman Empire fall?","answer":"The Western Roman Empire fell due to a combination of factors including military weakness, economic decline, population decline, political instability, and invasions by Germanic tribes and other barbarian groups. Climatic changes and widespread diseases also contributed to the collapse [1]. The Eastern Roman (Byzantine) Empire survived until 1453, continuing the Roman tradition in the East [2].","promptFormat":"chat-template","retrievedTitles":["Fall of the Western Roman Empire","Byzantine Empire"],"expectedKbTitles":["Fall of the Western Roman Empire"],"expectedKbHit":true,"timedOut":false,"createdAt":1791257710072} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":6568.592004001141,"generationLatencyMs":14603.040045976639,"totalLatencyMs":21578.524546980858,"tokensGenerated":60,"tokPerSec":4.108733511042511,"peakRssBytes":3032731648,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":406.3762210011482,"contextTokensBefore":155,"contextTokensAfter":121,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"grounded-2","category":"retrieval-grounded","query":"What is a black hole and how does one form?","answer":"A black hole is an astronomical object so compact that its gravity prevents anything, including light, from escaping [1]. It forms when a massive star collapses under its own gravity at the end of its life, squeezing its mass into an extremely small volume, creating a region where spacetime is severely curved.","promptFormat":"chat-template","retrievedTitles":["Black hole"],"expectedKbTitles":["Black hole"],"expectedKbHit":true,"timedOut":false,"createdAt":1791257731837} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"lookup","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":false,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":8461.133751988411,"generationLatencyMs":16428.38758099079,"totalLatencyMs":25332.41765499115,"tokensGenerated":80,"tokPerSec":4.869619711953206,"peakRssBytes":3035643904,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":442.33536797761917,"contextTokensBefore":0,"contextTokensAfter":0,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-1","category":"no-kb-content","query":"Who was Napoleon Bonaparte?","answer":"Napoleon Bonaparte was a French military leader and emperor who rose to power during the French Revolution and became the first Emperor of the French in 1804. He led France in a series of military campaigns across Europe, expanding its influence and reshaping the continent's political landscape, and later established a major impact on law, governance, and education through the Napoleonic Code.","promptFormat":"chat-template","retrievedTitles":[],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791257757399} +{"modelId":"qwen3-4b-instruct-2507-q4km","taskType":"research","adaptiveRoutingUsed":false,"reasonCodes":["eval:fixed-model","retrieve:relevant-to-task","generate:fixed-qwen3-4b-instruct-2507-q4km"],"retrievalUsed":true,"modelSwitches":0,"crossMessageModelSwitch":false,"modelResidency":"resident","modelLoadMs":0,"ttftMs":19313.344573020935,"generationLatencyMs":19378.240935981274,"totalLatencyMs":39309.57517403364,"tokensGenerated":81,"tokPerSec":4.179945964527679,"peakRssBytes":3036282880,"outcome":"success","runId":"eval-2026-10-06T03-29-11-385Z","evalSetVersion":"1","retrievalMs":616.9361169934273,"contextTokensBefore":175,"contextTokensAfter":147,"configId":"model:qwen3-4b-instruct-2507-q4km","configLabel":"Qwen3-4B-Instruct-2507 (Q4_K_M)","personalityId":"succinct","maxTokens":512,"queryId":"no-kb-2","category":"no-kb-content","query":"How do antibiotics work, and why does antibiotic resistance develop?","answer":"Antibiotics work by targeting specific structures or processes in bacteria, such as cell wall synthesis, protein production, or DNA replication, disrupting their ability to survive and multiply [1]. Antibiotic resistance develops when bacteria mutate or acquire genes that allow them to survive exposure to antibiotics, often due to misuse, overprescription, or improper disposal of antimicrobials, leading to the spread of resistant strains.","promptFormat":"chat-template","retrievedTitles":["Antimicrobial resistance"],"expectedKbTitles":[],"expectedKbHit":null,"timedOut":false,"createdAt":1791257796758} diff --git a/docs/evidence/2026-10-06-repack-under-pressure/repacked-std/eval-2026-10-06T03-29-11-385Z.report.md b/docs/evidence/2026-10-06-repack-under-pressure/repacked-std/eval-2026-10-06T03-29-11-385Z.report.md new file mode 100644 index 000000000..01abaf268 --- /dev/null +++ b/docs/evidence/2026-10-06-repack-under-pressure/repacked-std/eval-2026-10-06T03-29-11-385Z.report.md @@ -0,0 +1,17 @@ +BOAR Device Evaluation — eval-2026-10-06T03-29-11-385Z (set v1, 17 queries) +──────────────────────────────────────────────────────────── + +Qwen3-4B-Instruct-2507 (Q4_K_M) + Config: model:qwen3-4b-instruct-2507-q4km · prompt chat-template + Queries: 17 ok · 0 failed · 0 cancelled (of 17) + TTFT: avg 10.6s · p50 10.2s + Model load: none (already resident) + Generation: avg 15.1s · p50 16.4s + Tokens/sec: avg 4.9 · p50 4.5 + Search time: avg 0.4s · p50 0.4s (before the model starts) + Context tokens: 253 → 211 avg (trimmed to the answering sentences) + Total per query: avg 26.1s · p50 25.3s + Peak RSS: 2.83 GB + Residency: cold 0 · switched 0 · resident 17 + Model switches: 0 within plans · 0 across messages + Retrieval: used 11/17 · expected KB article found 10/10 diff --git a/docs/evidence/2026-10-06-repack-under-pressure/repacked-std/eval-2026-10-06T03-29-11-385Z.status.json b/docs/evidence/2026-10-06-repack-under-pressure/repacked-std/eval-2026-10-06T03-29-11-385Z.status.json new file mode 100644 index 000000000..8755777bb --- /dev/null +++ b/docs/evidence/2026-10-06-repack-under-pressure/repacked-std/eval-2026-10-06T03-29-11-385Z.status.json @@ -0,0 +1,17 @@ +{ + "requestId": "req-20261006t032908-k7rw", + "state": "done", + "configs": [ + "model:qwen3-4b-instruct-2507-q4km" + ], + "total": 17, + "completed": 17, + "runId": "eval-2026-10-06T03-29-11-385Z", + "resultPath": "files/eval/eval-2026-10-06T03-29-11-385Z.jsonl", + "stopped": false, + "installedModels": [ + "qwen3-4b-instruct-2507-q4km", + "qwen2.5-1.5b-instruct-q4km" + ], + "updatedAt": 1791257796984 +} diff --git a/src/inference/LlamaEngine.test.ts b/src/inference/LlamaEngine.test.ts index 6ca7d8535..bd1f55811 100644 --- a/src/inference/LlamaEngine.test.ts +++ b/src/inference/LlamaEngine.test.ts @@ -20,6 +20,8 @@ let maxConcurrentInits = 0; /** Native messages the next initLlama calls reject with, in order. */ const initFailures: string[] = []; const platform = { OS: "android" }; +/** When set, every context's completion resolves at once with this (one setting reaches contexts created later). */ +let reply: unknown = null; vi.mock("react-native", () => ({ Platform: { get OS() { return platform.OS; } } })); vi.mock("llama.rn", () => ({ @@ -55,7 +57,7 @@ vi.mock("llama.rn", () => ({ }, // Like llama.cpp: runs until stopped, and settles a moment after the stop (prompt still processing). completion: (_params, onToken) => - new Promise((resolve) => { + reply ? Promise.resolve(reply as { text: string }) : new Promise((resolve) => { generating = true; onToken({ token: "Hi" }); finish = () => @@ -83,7 +85,7 @@ vi.mock("ram-monitor", () => ({ getAvailableRamBytes: () => ram.avail, })); -import { LlamaEngine, PREFIX_WARM_IDLE_MS, defaultContextSize } from "./LlamaEngine"; +import { LlamaEngine, PREFIX_WARM_IDLE_MS, WARM_N_PREDICT, defaultContextSize } from "./LlamaEngine"; const live = () => created.filter((c) => !c.released); @@ -98,6 +100,7 @@ beforeEach(() => { initFailures.length = 0; platform.OS = "android"; stopReturnsNothing = false; + reply = null; }); describe("LlamaEngine Metal fallback (iPhone 13)", () => { @@ -298,6 +301,63 @@ describe("LlamaEngine pre-flight memory check (mmap-aware)", () => { }); }); +describe("repacked weights or the mapped file (memoryFit.ts shouldRepack)", () => { + const MiB = 1024 ** 2; + const stalled = { text: "x", timings: { prompt_n: 40, prompt_ms: 2000, predicted_n: 9, predicted_ms: 90_000 } }; + const normal = { text: "x", timings: { prompt_n: 40, prompt_ms: 2000, predicted_n: 400, predicted_ms: 90_000 } }; + const answerWith = (r: unknown) => void (reply = r); + + it("marks the load as repacked, and keeps the mapped file once when the guard says the last repacked load closed the app", async () => { + const begins: any[] = []; + let keep = true; + const guard = { + begin: async (_m: any, _p: any, o: any) => void begins.push(o), + end: async () => {}, + takeKeepMapped: async () => { + const k = keep; + keep = false; + return k; + }, + }; + const engine = new LlamaEngine(guard as any); + await engine.load("models/q4.gguf"); + expect(initParams[0]).toMatchObject({ no_extra_bufts: true }); + await engine.load("models/q15.gguf"); + expect(initParams[1]).toMatchObject({ no_extra_bufts: false }); + expect(begins).toEqual([{ repacked: false }, { repacked: true }]); + }); + + it("after a stalled answer, reloads the model as the mapped file before the next question (Android)", async () => { + const engine = new LlamaEngine(null); + await engine.load("models/q4.gguf", { nCtx: 3072 }); + answerWith(normal); + await engine.generate({ prompt: "a" }); + expect(created).toHaveLength(1); + answerWith(stalled); + await engine.generate({ prompt: "b" }); + answerWith(normal); + // Sent while the reload runs: it waits for the reloaded model instead of finding none. + await expect(engine.generate({ prompt: "c" })).resolves.toBe("x"); + expect(created).toHaveLength(2); + expect(created[0].released).toBe(true); + expect(initParams[1]).toMatchObject({ no_extra_bufts: true, n_ctx: 3072 }); + expect(engine.getModelInfo()?.repacked).toBe(false); + // Same model asked again (the chat re-mounting): stays mapped, no third load. + await engine.load("models/q4.gguf", { nCtx: 3072 }); + expect(created).toHaveLength(2); + }); + + it("does not reload on iOS (no swap: a slow answer is a slow model) or when already mapped", async () => { + platform.OS = "ios"; + const engine = new LlamaEngine(null); + await engine.load("models/q4.gguf"); + answerWith(stalled); + await engine.generate({ prompt: "b" }); + await engine.generate({ prompt: "c" }).catch(() => {}); + expect(created).toHaveLength(1); + }); +}); + describe("LlamaEngine generate queue (double-send race)", () => { it("never runs two completions on one context at once", async () => { const engine = new LlamaEngine(); @@ -410,15 +470,15 @@ describe("answer prefix kept in the KV cache (iPhone 13: 4-5 s to the first toke isJinjaSupported: () => true, completion: (p: any) => { seen.push(p); - return Promise.resolve({ text: p.n_predict === 0 ? "" : "ok", timings: { prompt_n: 60, prompt_ms: 600 } }); + return Promise.resolve({ text: p.n_predict === WARM_N_PREDICT ? "" : "ok", timings: { prompt_n: 60, prompt_ms: 600 } }); }, }); vi.useFakeTimers(); return { engine, seen }; } - const warms = (seen: any[]) => seen.filter((p) => p.n_predict === 0); + const warms = (seen: any[]) => seen.filter((p) => p.n_predict === WARM_N_PREDICT); - it("prefills the tone's prefix after the load, with no token generated, in the answer's chat format", async () => { + it("prefills the tone's prefix after the load, generating one discarded token (n_predict 0 skipped the decode), in the answer's chat format", async () => { try { const { engine, seen } = await loadedEngine(); engine.setAnswerPrefix(PREFIX); @@ -449,7 +509,7 @@ describe("answer prefix kept in the KV cache (iPhone 13: 4-5 s to the first toke (created[0] as any).isJinjaSupported = () => false; await engine.generate({ prompt: "Title: q" }); await vi.advanceTimersByTimeAsync(PREFIX_WARM_IDLE_MS + 20); - expect(warms(seen)[2]).toEqual({ prompt: PREFIX.prompt, n_predict: 0 }); + expect(warms(seen)[2]).toEqual({ prompt: PREFIX.prompt, n_predict: WARM_N_PREDICT }); } finally { vi.useRealTimers(); } diff --git a/src/inference/LlamaEngine.ts b/src/inference/LlamaEngine.ts index 4d4dca4ce..c1c7bbfed 100644 --- a/src/inference/LlamaEngine.ts +++ b/src/inference/LlamaEngine.ts @@ -11,8 +11,10 @@ import { parseGgufShape, toGb, contextSizeForRam, + isDecodeStall, + shouldRepack, } from "./memoryFit"; -import { BackendInfo, cpuDeviceNames, initWithCpuFallback } from "./initFallback"; +import { BackendInfo, cpuDeviceNames, initWithCpuFallback, kvCacheParams } from "./initFallback"; import { describeModelFile, type LoadGuard, type LoadMeta } from "./loadMarker"; import { MODEL_CATALOG } from "../models/manifest"; import { listDiscoveredModels } from "../models/discoveredModels"; @@ -80,6 +82,17 @@ export interface PromptPrefix { */ export const PREFIX_WARM_IDLE_MS = 750; +/** + * Tokens the prefix warm-up generates (and discards). Not 0: with n_predict 0, llama.rn 0.13 rc.6 returns + * before the prompt is decoded, so the KV cache never held the prefix and every answer re-read it (Pixel 6a, + * 2026-10-04: warm "1 token in 0 ms" even after a title prompt replaced the cache; answers evaluated their + * whole prompt although 229 of its first tokens matched the prefix). One token makes the decode loop run. + */ +export const WARM_N_PREDICT = 1; + +/** Fewer evaluated tokens than this say little about prompt reading speed. */ +const MIN_PREFILL_SAMPLE_TOKENS = 32; + export interface GenerationTimings { promptTokens: number; promptMs: number; @@ -108,6 +121,8 @@ export interface LoadedModelInfo { nThreads: number; /** "cpu-fallback" with the native reason when the GPU backend failed to start (initFallback.ts). */ backend?: BackendInfo; + /** Whether the weights were repacked for the ARM kernels (in app memory) or kept as the mapped file. */ + repacked?: boolean; } export function defaultContextSize(): number { @@ -189,6 +204,10 @@ export class LlamaEngine { private prefix: PromptPrefix | null = null; /** Whether the KV cache holds `prefix` (a load or another prompt drops it). */ private prefixCached = false; + /** Tokens the warm prefix fills (the warm-up's whole prompt), once it has run. */ + private prefixTokens = 0; + /** The last answer's prompt reading: tokens llama.cpp evaluated and the time to its first token. */ + private lastPrefill: { tokens: number; ms: number } | null = null; private warmTimer: ReturnType | null = null; /** generate() calls not settled yet: a prefix refill never waits in front of one of them. */ private pending = 0; @@ -200,6 +219,10 @@ export class LlamaEngine { } private lastLoad: LoadResult = { fit: null, warning: null }; + /** How the loaded model was loaded (not its progress callback), to reload it as the mapped file after a stall. */ + private lastLoadOpts: Omit | undefined; + /** Models that stalled repacked in this process: they keep the mapped file until the app restarts. */ + private keepMapped = new Set(); // general.architecture of the loaded model (from the GGUF header), for its thinking tags. private arch: string | null = null; private lastHeaderArch: string | null = null; @@ -223,7 +246,8 @@ export class LlamaEngine { this.context && this.modelInfo?.filename === modelFilename && this.modelInfo.nCtx === nCtx && - this.modelInfo.nThreads === nThreads + this.modelInfo.nThreads === nThreads && + !(this.modelInfo.repacked && this.keepMapped.has(modelFilename)) ) { return this.lastLoad; } @@ -260,8 +284,16 @@ export class LlamaEngine { : describeModelFile(modelFilename, [...MODEL_CATALOG, ...(await listDiscoveredModels().catch(() => []))]); const given = Object.fromEntries(Object.entries(opts?.meta ?? {}).filter(([, v]) => v !== undefined)); const meta: LoadMeta = { filename: modelFilename, ...known, ...given }; + // The mapped file after a repacked load killed the app (once) or stalled (this process); otherwise from + // memory (memoryFit.ts shouldRepack). No RAM readouts: repack, as before this check existed. + const afterCrash = (await this.guard?.takeKeepMapped?.(meta).catch(() => false)) === true; + const repack = !afterCrash && !this.keepMapped.has(modelFilename) && (fit ? shouldRepack(fit) : true); + if (fit) { + const why = afterCrash ? "the last repacked load closed the app" : this.keepMapped.has(modelFilename) ? "it stalled repacked" : `${toGb(fit.availableBytes)}GB budget`; + console.log(`[engine] ${repack ? "repacking" : "keeping the mapped file for"} ${modelFilename} (${toGb(fit.fileBytes)}GB; ${why})`); + } // Marker on disk while initLlama runs: if the OS kills the app here, the next start knows (CR-2). - await this.guard?.begin(meta, previous).catch((e: any) => console.warn("[engine] load marker:", e?.message ?? e)); + await this.guard?.begin(meta, previous, { repacked: repack }).catch((e: any) => console.warn("[engine] load marker:", e?.message ?? e)); let loadedOk = false; try { const onProgress = opts?.onProgress; @@ -274,12 +306,18 @@ export class LlamaEngine { n_ctx: nCtx, n_threads: nThreads, n_gpu_layers: 0, // CPU-only for broad device compatibility; adjust per-device + // A model too big for memory streams from its mapped file (memoryFit.ts shouldRepack), and there the + // 8-bit KV cache's ~half-size cache matters. Elsewhere it isn't worth it: on a Pixel 6a the 4B read its + // prompts ~19% slower with it (first word 11.6 s vs 9.5 s median, eval:device standard set, 2026-10-05). + ...(repack ? {} : kvCacheParams(Platform.OS)), + no_extra_bufts: !repack, }, { platform: Platform.OS, cpuDevices: () => cpuDeviceNames(getBackendDevicesInfo), log: (m) => console.warn(m) } ); this.context = loaded.context; backend = loaded.backend; - this.modelInfo = { filename: modelFilename, nCtx, nThreads, backend }; + this.modelInfo = { filename: modelFilename, nCtx, nThreads, backend, repacked: repack }; + this.lastLoadOpts = { nCtx, nThreads, meta: opts?.meta }; this.loadedMeta = meta; loadedOk = true; this.arch = fit ? this.lastHeaderArch : await this.readArch(modelPath); @@ -373,6 +411,9 @@ export class LlamaEngine { const context = this.context; this.context = null; this.prefixCached = false; + // Another model reads prompts at another speed (estimateFirstTokenMs). + this.prefixTokens = 0; + this.lastPrefill = null; this.modelInfo = null; this.loadedMeta = null; this.lastLoad = { fit: null, warning: null }; @@ -449,19 +490,43 @@ export class LlamaEngine { }, delayMs); } - /** Prefills the prefix with no token generated (n_predict 0), in the same format an answer uses. */ + /** + * Expected time to the first token for this prompt, from the last answer's prompt reading speed: the + * prompt's tokens (in the format generate() will send) minus the warm prefix when it is cached. Null when + * there is no measurement yet or the prompt can't be tokenized. + */ + async estimateFirstTokenMs(opts: Pick): Promise { + const ctx = this.context; + const rate = this.lastPrefill; + if (!ctx || !rate || rate.ms <= 0) return null; + try { + let text = opts.prompt ?? ""; + if (opts.messages) { + const r: any = await ctx.getFormattedChat(opts.messages as any, null, { jinja: true }); + text = typeof r === "string" ? r : (r?.prompt ?? ""); + } + const tokens = ((await ctx.tokenize(text)) as any).tokens?.length ?? 0; + const cached = this.prefixCached && this.sharesPrefix(opts) ? Math.min(this.prefixTokens, tokens) : 0; + return (Math.max(tokens - cached, 1) / rate.tokens) * rate.ms; + } catch { + return null; + } + } + + /** Prefills the prefix (one discarded token, WARM_N_PREDICT), in the same format an answer uses. */ private async warmNow(): Promise { const prefix = this.prefix; const context = this.context; if (!prefix || !context || this.prefixCached || this.pending > 0) return; const params = context.isJinjaSupported() ? // The empty user turn keeps templates that require one (Qwen3) rendering; the prefix ends before it. - { messages: [{ role: "system", content: prefix.system }, { role: "user", content: "" }], jinja: true, n_predict: 0 } - : { prompt: prefix.prompt, n_predict: 0 }; + { messages: [{ role: "system", content: prefix.system }, { role: "user", content: "" }], jinja: true, n_predict: WARM_N_PREDICT } + : { prompt: prefix.prompt, n_predict: WARM_N_PREDICT }; const completion = context.completion(params); this.inFlight = completion; try { - const result = (await completion) as { timings?: { prompt_n?: number; prompt_ms?: number } }; + const result = (await completion) as { timings?: { prompt_n?: number; prompt_ms?: number }; tokens_evaluated?: number }; + if (result?.tokens_evaluated) this.prefixTokens = result.tokens_evaluated; if (this.prefix === prefix && this.context === context) this.prefixCached = true; const t = result?.timings; if (t) console.log(`[engine] answer prefix prefilled: ${t.prompt_n} tokens in ${Math.round(t.prompt_ms ?? 0)} ms`); @@ -528,7 +593,10 @@ export class LlamaEngine { : { prompt: prompt!, n_predict: nPredict, temperature, stop: stop ?? DEFAULT_STOP_SEQUENCES }; let full = ""; + const startedAt = Date.now(); + let firstTokenAt: number | null = null; const completion = this.context.completion(completionParams, (data) => { + if (firstTokenAt === null) firstTokenAt = Date.now(); full += data.token; onToken?.(data.token); }); @@ -536,6 +604,11 @@ export class LlamaEngine { try { const result = await completion; const t = (result as { timings?: any; tokens_cached?: number }).timings; + // Prompt reading speed for the next answer's length (prompt_n: tokens evaluated, the cached prefix excluded). + if (t?.prompt_n >= MIN_PREFILL_SAMPLE_TOKENS && firstTokenAt !== null) this.lastPrefill = { tokens: t.prompt_n, ms: firstTokenAt - startedAt }; + if (t && Platform.OS === "android" && this.modelInfo?.repacked && isDecodeStall({ predictedTokens: t.predicted_n ?? 0, predictedMs: t.predicted_ms ?? 0 })) { + this.reloadMapped(this.modelInfo.filename, t); + } if (t && onTimings) { onTimings({ promptTokens: t.prompt_n ?? 0, @@ -552,6 +625,22 @@ export class LlamaEngine { } } + /** + * The answer stalled with repacked weights (swapped out: Android can only compress them, not drop and re-read + * them). Reload the model as the mapped file before the next generation: prompt reading ~2x slower, but its + * weights are page cache, which the kernel re-reads from storage instead of stalling. Android only: iOS has no + * swap (jetsam ends the app instead), so a slow answer there is a slow model, not a stall. + */ + private reloadMapped(filename: string, t: { predicted_n?: number; predicted_ms?: number }): void { + if (this.keepMapped.has(filename)) return; + this.keepMapped.add(filename); + console.warn(`[engine] ${filename} stalled repacked (${t.predicted_n ?? 0} tokens in ${Math.round(t.predicted_ms ?? 0)} ms); reloading it as the mapped file`); + const opts = this.lastLoadOpts; + // On the generation queue: questions sent meanwhile wait for the reload instead of finding no model. + const run = this.genQueue.then(() => this.load(filename, opts)); + this.genQueue = run.catch((e: any) => console.warn("[engine] reload as mapped file failed:", e?.message ?? e)); + } + /** * Signals the native completion loop to stop. The in-flight generate() * call's completion() promise resolves normally with whatever text was diff --git a/src/inference/initFallback.test.ts b/src/inference/initFallback.test.ts index 76a3261eb..39da8c6d5 100644 --- a/src/inference/initFallback.test.ts +++ b/src/inference/initFallback.test.ts @@ -1,5 +1,5 @@ import { describe, it, expect } from "vitest"; -import { cpuDeviceNames, initWithCpuFallback } from "./initFallback"; +import { cpuDeviceNames, initWithCpuFallback, kvBytesPerElem, kvCacheParams } from "./initFallback"; // Plain functions, not vi.fn(): a rejecting vi.fn() fails the test in vitest 5. function fakeInit(failures: string[]) { @@ -65,3 +65,28 @@ describe("cpuDeviceNames", () => { expect(await cpuDeviceNames(() => Promise.reject(new Error("no")))).toEqual(["CPU"]); }); }); + +describe("8-bit KV cache", () => { + it("Android: K and V at 8 bits, with the flash attention a quantized V needs", () => { + expect(kvCacheParams("android")).toEqual({ cache_type_k: "q8_0", cache_type_v: "q8_0", flash_attn_type: "on" }); + expect(kvBytesPerElem(kvCacheParams("android"))).toBeCloseTo(1.0625); + }); + + it("iOS: K only, since flash attention stays off there", () => { + expect(kvCacheParams("ios")).toEqual({ cache_type_k: "q8_0" }); + expect(kvBytesPerElem(kvCacheParams("ios"))).toBeCloseTo((1.0625 + 2) / 2); + }); + + it("the CPU retry turns flash attention off and so keeps V at 16 bits", async () => { + const f = fakeInit(["failed to initialize context"]); + await initWithCpuFallback(f.init, { model: "m.gguf", ...kvCacheParams("android") }, { platform: "android", cpuDevices: cpu }); + expect(f.calls[0]).toMatchObject({ cache_type_k: "q8_0", cache_type_v: "q8_0", flash_attn_type: "on" }); + expect(f.calls[1]).toEqual({ model: "m.gguf", cache_type_k: "q8_0", n_gpu_layers: 0, devices: ["CPU"], flash_attn_type: "off" }); + }); + + it("iOS never asks for a quantized V, even if given one", async () => { + const f = fakeInit([]); + await initWithCpuFallback(f.init, { model: "m.gguf", cache_type_k: "q8_0", cache_type_v: "q8_0" }, { platform: "ios", cpuDevices: cpu }); + expect(f.calls[0]).toEqual({ model: "m.gguf", cache_type_k: "q8_0", flash_attn_type: "off" }); + }); +}); diff --git a/src/inference/initFallback.ts b/src/inference/initFallback.ts index 91d5dd847..5261a152d 100644 --- a/src/inference/initFallback.ts +++ b/src/inference/initFallback.ts @@ -11,6 +11,38 @@ export interface BackendInitParams { n_gpu_layers?: number; devices?: string[]; flash_attn_type?: "auto" | "on" | "off"; + cache_type_k?: KvCacheType; + cache_type_v?: KvCacheType; +} + +/** The KV cache types this app uses (a subset of llama.rn's). */ +export type KvCacheType = "f16" | "q8_0"; + +/** + * KV cache at 8 bits instead of 16: the one allocation mmap can't page out, halved. On a Pixel 6a (6 GB) + * the 4B left ~290 MB free with swap full, and answers stalled at 0.1 tok/s while its weights were + * re-read from storage. llama.cpp needs flash attention for a quantized V cache, so V is 8-bit only + * where flash attention is on (Android); iOS keeps it off (Metal, see initWithCpuFallback) and gets K only. + */ +export const KV_CACHE_TYPE: KvCacheType = "q8_0"; + +export function kvCacheParams(platform: string): Pick { + return platform === "ios" + ? { cache_type_k: KV_CACHE_TYPE } + : { cache_type_k: KV_CACHE_TYPE, cache_type_v: KV_CACHE_TYPE, flash_attn_type: "on" }; +} + +/** Average bytes per KV element for these cache types (q8_0: 34 bytes per 32 values), for memoryFit. */ +export function kvBytesPerElem(p: Pick): number { + const bytes = (t?: KvCacheType) => (t === "q8_0" ? 34 / 32 : 2); + return (bytes(p.cache_type_k) + bytes(p.cache_type_v)) / 2; +} + +/** Without flash attention llama.cpp can't use a quantized V cache: keep V at 16 bits. */ +function withFlashAttnOff

(p: P): P { + const out = { ...p, flash_attn_type: "off" as const }; + if (out.cache_type_v && out.cache_type_v !== "f16") delete out.cache_type_v; + return out; } export interface BackendInfo { @@ -46,14 +78,14 @@ export async function initWithCpuFallback

( opts: { platform: string; cpuDevices: () => Promise; log?: (message: string) => void } ): Promise<{ context: C; backend: BackendInfo }> { // iOS: the library that fails to compile is the flash-attention one; don't ask for it. - const first: P & BackendInitParams = opts.platform === "ios" ? { ...params, flash_attn_type: "off" } : params; + const first: P & BackendInitParams = opts.platform === "ios" ? withFlashAttnOff(params) : params; try { return { context: await init(first), backend: { kind: "default" } }; } catch (e: any) { const reason = String(e?.message ?? e); if (!BACKEND_INIT_ERROR.test(reason)) throw e; opts.log?.(`[engine] backend init failed (${reason}); retrying on CPU only`); - const cpu = { ...params, n_gpu_layers: 0, devices: await opts.cpuDevices(), flash_attn_type: "off" as const }; + const cpu = withFlashAttnOff({ ...params, n_gpu_layers: 0, devices: await opts.cpuDevices() }); return { context: await init(cpu), backend: { kind: "cpu-fallback", reason } }; } } diff --git a/src/inference/loadGuard.ts b/src/inference/loadGuard.ts index 6723e9592..3cf84e5a6 100644 --- a/src/inference/loadGuard.ts +++ b/src/inference/loadGuard.ts @@ -4,7 +4,7 @@ * calls once after start to tell the user a model closed the app. */ import * as FileSystem from "expo-file-system/legacy"; -import { recordLoadCrash, recordLoadSuccess, takePendingLoadCrash } from "../models/settings"; +import { recordLoadCrash, recordLoadSuccess, recordRepackCrash, takeKeepMapped, takePendingLoadCrash } from "../models/settings"; import { createLoadGuard, LoadCrash, MarkerStore } from "./loadMarker"; const MARKER = `${FileSystem.documentDirectory}model-load.marker.json`; @@ -22,6 +22,8 @@ export const loadGuard = createLoadGuard(fileStore, { recordCrash: recordLoadCrash, recordSuccess: recordLoadSuccess, takePending: takePendingLoadCrash, + recordRepackCrash, + takeKeepMapped, }); /** The last load that killed the app, once (null afterwards, or when there was none). */ diff --git a/src/inference/loadMarker.test.ts b/src/inference/loadMarker.test.ts index 6760fbaf4..2d7b3d557 100644 --- a/src/inference/loadMarker.test.ts +++ b/src/inference/loadMarker.test.ts @@ -7,7 +7,14 @@ function memory() { const crashes: LoadCrash[] = []; const successes: string[] = []; let pending: LoadCrash | null = null; + let keepMapped: string[] = []; const ledger: CrashLedger = { + recordRepackCrash: async (id) => void keepMapped.push(id), + takeKeepMapped: async (id) => { + const had = keepMapped.includes(id); + keepMapped = keepMapped.filter((k) => k !== id); + return had; + }, recordCrash: async (c) => void (crashes.push(c), (pending = c)), recordSuccess: async (id) => void successes.push(id), takePending: async () => { @@ -16,7 +23,7 @@ function memory() { return c; }, }; - return { store, ledger, crashes, successes, file: () => file }; + return { store, ledger, crashes, successes, file: () => file, keepMapped: () => keepMapped }; } describe("load marker (CR-2)", () => { @@ -55,6 +62,30 @@ describe("load marker (CR-2)", () => { }); }); +describe("a repacked load that killed the app", () => { + const q15 = { filename: "models/q15.gguf", modelId: "qwen2.5-1.5b", label: "Qwen2.5 1.5B" }; + const q4 = { filename: "models/q4.gguf", modelId: "qwen3-4b", label: "Qwen3 4B" }; + + it("is not a CR-2 crash: the model loads once more as the mapped file instead of falling back", async () => { + const m = memory(); + await createLoadGuard(m.store, m.ledger).begin(q4, q15, { repacked: true }); // ... killed while repacking + const next = createLoadGuard(m.store, m.ledger); + expect(await next.consume()).toBeNull(); + expect(m.crashes).toEqual([]); + expect(await next.takeKeepMapped(q4)).toBe(true); + expect(await next.takeKeepMapped(q4)).toBe(false); // once: the load after that decides from memory again + }); + + it("if the mapped load dies too, that one is a CR-2 crash", async () => { + const m = memory(); + await createLoadGuard(m.store, m.ledger).begin(q4, q15, { repacked: true }); + const second = createLoadGuard(m.store, m.ledger); + expect(await second.takeKeepMapped(q4)).toBe(true); + await second.begin(q4, q15, { repacked: false }); // ... killed again + expect((await createLoadGuard(m.store, m.ledger).consume())?.crashedModelId).toBe("qwen3-4b"); + }); +}); + describe("describeModelFile (Prism CR-3: a load without meta)", () => { const catalog = [{ id: "qwen3-4b-instruct-2507-q4km", label: "Qwen3-4B-Instruct-2507 (Q4_K_M)", filename: "models/qwen3-4b-instruct-2507-q4km.gguf" }]; it("the catalog's id and label, by path or by file name", () => { diff --git a/src/inference/loadMarker.ts b/src/inference/loadMarker.ts index b06be1447..56af348db 100644 --- a/src/inference/loadMarker.ts +++ b/src/inference/loadMarker.ts @@ -14,6 +14,8 @@ export interface LoadMeta { export interface LoadMarker extends LoadMeta { /** The model that was loaded before (the one the app falls back to). */ previous?: LoadMeta | null; + /** The load repacked the weights into app memory (LlamaEngine, memoryFit.ts shouldRepack). */ + repacked?: boolean; at: number; } @@ -37,6 +39,9 @@ export interface CrashLedger { recordCrash(crash: LoadCrash): Promise; recordSuccess(modelId: string): Promise; takePending(): Promise; + /** A repacked load killed the app: load that model once more without repacking, instead of blocking it. */ + recordRepackCrash(modelId: string): Promise; + takeKeepMapped(modelId: string): Promise; } /** @@ -70,7 +75,11 @@ export function createLoadGuard(store: MarkerStore, ledger: CrashLedger, now: () const raw = await store.read().catch(() => null); if (!raw) return; try { - await ledger.recordCrash(toLoadCrash(JSON.parse(raw) as LoadMarker)); + const marker = JSON.parse(raw) as LoadMarker; + // Repacking is what needs the memory (~the file size in app memory at the load's peak, on top of what + // the phone holds); the same model as a mapped file usually loads. Try that before giving up on it. + if (marker.repacked) await ledger.recordRepackCrash(marker.modelId ?? marker.filename); + else await ledger.recordCrash(toLoadCrash(marker)); } catch (e: any) { console.warn("[loadMarker] could not record the crashed load:", e?.message ?? e); } @@ -81,15 +90,20 @@ export function createLoadGuard(store: MarkerStore, ledger: CrashLedger, now: () return { ensureChecked, - async begin(meta: LoadMeta, previous: LoadMeta | null): Promise { + async begin(meta: LoadMeta, previous: LoadMeta | null, opts?: { repacked?: boolean }): Promise { await ensureChecked(); - const marker: LoadMarker = { ...meta, previous, at: now() }; + const marker: LoadMarker = { ...meta, previous, at: now(), ...(opts?.repacked ? { repacked: true } : {}) }; await store.write(JSON.stringify(marker)); }, async end(meta: LoadMeta, loaded: boolean): Promise { await store.clear(); if (loaded && meta.modelId) await ledger.recordSuccess(meta.modelId); }, + /** Whether this load should keep the mapped file because the last repacked load of the model killed the app. */ + async takeKeepMapped(meta: LoadMeta): Promise { + await ensureChecked(); + return ledger.takeKeepMapped(meta.modelId ?? meta.filename); + }, /** The last crash, once: the next call returns null. */ async consume(): Promise { await ensureChecked(); diff --git a/src/inference/memoryFit.test.ts b/src/inference/memoryFit.test.ts index 882856bf0..35f253f1f 100644 --- a/src/inference/memoryFit.test.ts +++ b/src/inference/memoryFit.test.ts @@ -1,6 +1,8 @@ import { describe, it, expect } from "vitest"; import { availableRamFrom, + shouldRepack, + isDecodeStall, describeFit, estimateMemoryFit, expertFractionOf, @@ -162,3 +164,41 @@ describe("availableRamFrom", () => { expect(availableRamFrom({ totalBytes: 12 * GiB, rssBytes: GiB, availBytes: 10 * GiB })).toBe(10 * GiB); }); }); + +describe("shouldRepack", () => { + const GiB = 1024 ** 3; + // Pixel 6a: 5.46 GiB total -> availableRamFrom budget ~2.96 GiB. + const budget = availableRamFrom({ totalBytes: 5.46 * GiB, rssBytes: 0, availBytes: 2.8 * GiB }); + + it("repacks the 4B on a 6 GB phone: single questions keep the fast prompt reading", () => { + expect(shouldRepack({ fileBytes: 2.33 * GiB, availableBytes: budget })).toBe(true); + }); + + it("repacks the 1.5B on the same phone", () => { + expect(shouldRepack({ fileBytes: 0.92 * GiB, availableBytes: budget })).toBe(true); + }); + + it("repacks the 4B on a 12 GB phone", () => { + const big = availableRamFrom({ totalBytes: 11.6 * GiB, rssBytes: 0, availBytes: 6 * GiB }); + expect(shouldRepack({ fileBytes: 2.33 * GiB, availableBytes: big })).toBe(true); + }); + + it("never repacks a mixture of experts that has to stream from storage (35B deep tier on 12 GB)", () => { + const big = availableRamFrom({ totalBytes: 11.6 * GiB, rssBytes: 0, availBytes: 6 * GiB }); + expect(shouldRepack({ fileBytes: 11 * GiB, availableBytes: big })).toBe(false); + }); + + it("keeps a dense model bigger than the budget mapped (7B on a 6 GB phone)", () => { + expect(shouldRepack({ fileBytes: 4.36 * GiB, availableBytes: budget })).toBe(false); + }); +}); + +describe("isDecodeStall", () => { + it("under 1 tok/s for 10 s or more", () => { + expect(isDecodeStall({ predictedTokens: 9, predictedMs: 120_000 })).toBe(true); + expect(isDecodeStall({ predictedTokens: 400, predictedMs: 90_000 })).toBe(false); + }); + it("not a short slow start (prompt still reading, one token)", () => { + expect(isDecodeStall({ predictedTokens: 1, predictedMs: 3_000 })).toBe(false); + }); +}); diff --git a/src/inference/memoryFit.ts b/src/inference/memoryFit.ts index bbccde277..715765014 100644 --- a/src/inference/memoryFit.ts +++ b/src/inference/memoryFit.ts @@ -222,6 +222,37 @@ export function estimateMemoryFit(i: FitInputs): MemoryFit { }; } +/** + * Whether llama.cpp may repack the weights for its fast ARM CPU kernels (llama.rn `no_extra_bufts: false`). + * + * Repacked weights are copies in the app's own (anonymous) memory, not the mmap'd file: Android can't drop + * and re-read them, only compress them into swap. A model whose file doesn't fit the memory budget can't + * work repacked at all (an 11 GB mixture of experts on a 12 GB phone would need its whole file in RAM instead + * of streaming its experts), so it stays the mapped file. + * + * Only that case, even when little memory is free right now. On a Pixel 6a with the 4B and 1.8-2.1 GB free + * (background apps open), the mapped file took 25 s to the first word against 10 s repacked, and the repacked + * load was not killed in three tries (eval:device standard set, 2026-10-06); a stricter rule (file + buffers + + * 0.75 GB) cost single questions 23 s instead of 9 s (2026-10-05). When a repacked load does get the app + * killed, or stalls, LlamaEngine keeps the mapped file for that model instead (loadMarker.ts, isDecodeStall). + */ +export function shouldRepack(f: Pick): boolean { + return f.fileBytes <= f.availableBytes; +} + +/** Decode below this rate, for at least STALL_MIN_MS, is a stall (normal on a 6 GB phone: 3.5-6 tok/s for a 4B). */ +export const STALL_TOK_PER_SEC = 1; +export const STALL_MIN_MS = 10_000; + +/** + * Whether an answer's writing stalled: with repacked weights in swap, every token waits for them to be + * decompressed (0.1 tok/s on a Pixel 6a with the 4B, ~1 answer in 6 on long conversations, 2026-10-04). + */ +export function isDecodeStall(t: { predictedTokens: number; predictedMs: number }): boolean { + if (t.predictedMs < STALL_MIN_MS) return false; + return (t.predictedTokens * 1000) / t.predictedMs < STALL_TOK_PER_SEC; +} + /** Kept for the system and the app you came from: a quarter of the phone's RAM, at least 2.5 GB. */ export const SYSTEM_RESERVE_FRACTION = 0.25; export const MIN_SYSTEM_RESERVE_BYTES = 2.5 * GiB; diff --git a/src/models/settings.ts b/src/models/settings.ts index ff125bf14..c949d2b80 100644 --- a/src/models/settings.ts +++ b/src/models/settings.ts @@ -46,6 +46,8 @@ interface Settings { loadCrashedIds?: string[]; /** The last load crash, until the chat shows it once (src/inference/loadGuard.ts consumeLoadCrash). */ pendingLoadCrash?: StoredLoadCrash | null; + /** Models whose repacked load killed the app: their next load keeps the mapped file (loadMarker.ts). */ + keepMappedIds?: string[]; /** Random id created the first time this install shares results (src/eval/shareResults.ts). */ installId?: string; /** The server's id for this phone's sharing key, once it has accepted the key (src/eval/shareResults.ts). */ @@ -565,6 +567,29 @@ export async function recordLoadSuccess(modelId: string): Promise { }); } +/** + * A load with repacked weights killed the app: the next load of that model keeps the mapped file instead + * (no repack copy in app memory), and the model stays usable. Not a CR-2 crash: if that load dies too, it is. + */ +export async function recordRepackCrash(modelId: string): Promise { + return serialized(async () => { + const s = await readSettings(); + s.keepMappedIds = [...new Set([...(s.keepMappedIds ?? []), modelId])]; + await writeSettings(s); + }); +} + +/** Whether this model's next load keeps the mapped file, once: the load after it decides from memory again. */ +export async function takeKeepMapped(modelId: string): Promise { + return serialized(async () => { + const s = await readSettings(); + if (!(s.keepMappedIds ?? []).includes(modelId)) return false; + s.keepMappedIds = (s.keepMappedIds ?? []).filter((id) => id !== modelId); + await writeSettings(s); + return true; + }); +} + /** The pending crash notice, once. */ export async function takePendingLoadCrash(): Promise { return serialized(async () => { diff --git a/src/routing/answerLength.test.ts b/src/routing/answerLength.test.ts new file mode 100644 index 000000000..b7501a4bc --- /dev/null +++ b/src/routing/answerLength.test.ts @@ -0,0 +1,72 @@ +import { describe, expect, it } from "vitest"; +import { ANSWER_TIME_BUDGET_S, MIN_ANSWER_WORDS, RECENT_MS, answerLength, currentSpeed, lengthInstruction } from "./answerLength"; + +describe("answerLength", () => { + it("lets a fast model use the user's max tokens", () => { + // Qwen2.5-1.5B on a Pixel 6a: ~15 tok/s, 90 s would allow 1,350 tokens. + expect(answerLength({ maxTokens: 512, tokPerSec: 15 })).toEqual({ tokens: 512, words: 310, limit: "max" }); + }); + + it("shortens the answer on a slow model so it ends within the time budget", () => { + // Qwen3-4B on the same phone: ~4 tok/s, cut at ~450 tokens by the 120 s timeout. + const r = answerLength({ maxTokens: 512, tokPerSec: 4 }); + expect(r).toEqual({ tokens: 4 * ANSWER_TIME_BUDGET_S, words: 220, limit: "time" }); + }); + + it("uses only the max tokens for a model never measured on this phone", () => { + expect(answerLength({ maxTokens: 512 })).toEqual({ tokens: 512, words: 310, limit: "max" }); + expect(answerLength({ maxTokens: 512, tokPerSec: 0 }).limit).toBe("max"); + }); + + it("never asks for fewer than the minimum words", () => { + expect(answerLength({ maxTokens: 512, tokPerSec: 0.5 }).words).toBe(MIN_ANSWER_WORDS); + }); +}); + +describe("answerLength on a hot phone", () => { + it("leaves room for the first word: a slow start gets a shorter answer", () => { + // Pixel 6a, answer 5 of a conversation: 2.5 tok/s and 31.6 s to the first word; with 90 s of + // writing it was cut at 120 s. 120 - 31.6 - 15 = 73.4 s of writing -> 183 tokens, done by ~105 s. + const r = answerLength({ maxTokens: 512, tokPerSec: 2.5, ttftMs: 31_600 }); + expect(r.tokens).toBe(183); + expect(r.limit).toBe("time"); + expect(31.6 + r.tokens / 2.5).toBeLessThan(120); + }); + + it("never plans less than the minimum writing time", () => { + expect(answerLength({ maxTokens: 512, tokPerSec: 4, ttftMs: 100_000 }).tokens).toBe(4 * 30); + }); + + it("caps writing at the budget when the first word is quick", () => { + expect(answerLength({ maxTokens: 512, tokPerSec: 4, ttftMs: 2_000 }).tokens).toBe(4 * ANSWER_TIME_BUDGET_S); + }); +}); + +describe("currentSpeed", () => { + const now = 10_000_000; + const rec = (createdAt: number, tokPerSec: number, ttftMs: number, extra = {}) => ({ modelId: "m", createdAt, tokPerSec, ttftMs, tokensGenerated: 200, outcome: "success", ...extra }); + + it("uses the latest recent answer of this model", () => { + const records = [rec(now - 60_000, 4.9, 8_000), rec(now - 10_000, 2.5, 31_600), rec(now - 5_000, 9, 1_000, { modelId: "other" })]; + expect(currentSpeed(records, "m", now, 4.2)).toEqual({ tokPerSec: 2.5, ttftMs: 31_600 }); + }); + + it("falls back to the typical speed when the latest answer is old", () => { + expect(currentSpeed([rec(now - RECENT_MS - 1, 2.5, 31_600)], "m", now, 4.2)).toEqual({ tokPerSec: 4.2 }); + }); + + it("skips failed and very short answers", () => { + const records = [rec(now - 60_000, 4.9, 8_000), rec(now - 1_000, 0.1, 40_000, { tokensGenerated: 9 }), rec(now - 500, 1, 1, { outcome: "failure" })]; + expect(currentSpeed(records, "m", now, 4.2)).toEqual({ tokPerSec: 4.9, ttftMs: 8_000 }); + }); + + it("returns nothing without a model", () => { + expect(currentSpeed([], undefined, now, 4.2)).toEqual({}); + }); +}); + +describe("lengthInstruction", () => { + it("states the target and asks for a complete answer", () => { + expect(lengthInstruction(220)).toBe("Keep the whole answer under about 220 words, and finish it."); + }); +}); diff --git a/src/routing/answerLength.ts b/src/routing/answerLength.ts new file mode 100644 index 000000000..be6f717da --- /dev/null +++ b/src/routing/answerLength.ts @@ -0,0 +1,85 @@ +/** + * How long an answer should be on this phone, so it ends instead of being cut. + * + * Two hard stops cut answers mid-sentence: the user's Max Output Tokens (512 by + * default; detailed answers from Qwen2.5-1.5B hit it) and the 120 s step + * timeout (Qwen3-4B on a Pixel 6a writes ~4 tok/s, so every detailed answer + * stopped at ~450 tokens, 2026-10-04). Raising them would trade a cut answer for + * minutes of waiting, so they stay; instead the model is told the length up + * front and plans a complete answer that fits. + * + * Length = min(max tokens, decode speed × writing time), as a word target with a + * margin for formatting and citations. Writing time = the step limit minus the + * time to the first word minus a safety margin, at most ANSWER_TIME_BUDGET_S. + * Speed and first-word time come from the latest answer when it is recent, since + * a phone that heats up writes slower (Pixel 6a, 2026-10-04: 4.9 → 2.5 tok/s over + * one conversation, and with the median speed two answers were still cut). + * A model never measured on this phone only gets the max-tokens target. + * + * Pure (no native imports), so it is unit-testable. + */ + +/** The step timeout the answer has to fit in, first word included (STEP_TIMEOUT_MS in executor.ts). */ +export const STEP_LIMIT_S = 120; +/** Kept free under the step limit: decode keeps slowing while the phone heats during the answer. */ +export const SAFETY_S = 15; +/** Most seconds of writing an answer may take. */ +export const ANSWER_TIME_BUDGET_S = 90; +/** Least: below this the answer would be a sentence, whatever the phone. */ +export const MIN_WRITE_S = 30; +/** A measurement older than this says little about the phone's current temperature. */ +export const RECENT_MS = 15 * 60_000; +/** English prose averages ~0.75 words per token; 0.8 of that leaves room for lists, citations and the closing line. */ +const WORDS_PER_TOKEN = 0.75 * 0.8; +/** Below this the target would read as "be terse" rather than "fit the phone"; such a phone gets this many. */ +export const MIN_ANSWER_WORDS = 60; + +export interface AnswerLength { + /** Tokens the answer can take on this phone. */ + tokens: number; + /** The word target given to the model. */ + words: number; + /** "time" when the phone's speed is the limit, "max" when the user's Max Output Tokens is. */ + limit: "time" | "max"; +} + +export function answerLength(i: { maxTokens: number; tokPerSec?: number; ttftMs?: number }): AnswerLength { + const writeS = Math.min(ANSWER_TIME_BUDGET_S, Math.max(MIN_WRITE_S, STEP_LIMIT_S - (i.ttftMs ?? 0) / 1000 - SAFETY_S)); + const byTime = i.tokPerSec && i.tokPerSec > 0 ? Math.floor(i.tokPerSec * writeS) : Infinity; + const tokens = Math.min(i.maxTokens, byTime); + const words = Math.max(MIN_ANSWER_WORDS, Math.round((tokens * WORDS_PER_TOKEN) / 10) * 10); + return { tokens, words, limit: byTime < i.maxTokens ? "time" : "max" }; +} + +export function lengthInstruction(words: number): string { + return `Keep the whole answer under about ${words} words, and finish it.`; +} + +export interface SpeedRecord { + modelId?: string; + createdAt?: number; + outcome?: string; + tokensGenerated?: number; + tokPerSec?: number; + ttftMs?: number; +} + +/** + * The speed to size the next answer with: the model's latest successful answer of at least 16 tokens + * when it is recent (with its time to the first word), else `typical` (the median over all answers). + */ +export function currentSpeed( + records: SpeedRecord[], + modelId: string | undefined, + now: number, + typical?: number +): { tokPerSec?: number; ttftMs?: number } { + if (!modelId) return {}; + const latest = records + .filter((r) => r.modelId === modelId && (!r.outcome || r.outcome === "success") && (r.tokensGenerated ?? 0) >= 16 && (r.tokPerSec ?? 0) > 0) + .sort((a, b) => (b.createdAt ?? 0) - (a.createdAt ?? 0))[0]; + if (latest && latest.createdAt !== undefined && now - latest.createdAt <= RECENT_MS) { + return { tokPerSec: latest.tokPerSec, ttftMs: latest.ttftMs }; + } + return { tokPerSec: typical }; +} diff --git a/src/routing/executor.test.ts b/src/routing/executor.test.ts index 326b7bc55..05c6b3ad3 100644 --- a/src/routing/executor.test.ts +++ b/src/routing/executor.test.ts @@ -15,6 +15,7 @@ const getModelInfoMock = vi.fn(() => ); let mockEmbeddedTemplate = false; const hasEmbeddedChatTemplateMock = vi.fn(() => mockEmbeddedTemplate); +let mockFirstTokenMs: number | null = null; vi.mock("../inference/LlamaEngine", () => ({ llamaEngine: { @@ -22,6 +23,7 @@ vi.mock("../inference/LlamaEngine", () => ({ generate: (opts: any) => generateMock(opts), getModelInfo: () => getModelInfoMock(), hasEmbeddedChatTemplate: () => hasEmbeddedChatTemplateMock(), + estimateFirstTokenMs: async () => mockFirstTokenMs, }, })); @@ -106,6 +108,48 @@ describe("executeRoutingPlan", () => { expect(call.messages).toBeUndefined(); }); + it("sizes the answer from this prompt's estimated first word, after retrieval", async () => { + const p = plan({ + steps: [{ id: "generate-0", type: "generate", modelId: "qwen-1.5b", required: true }], + selectedModelIds: ["qwen-1.5b"], + }); + // 4 tok/s; a 56 s first word leaves 120 - 56 - 15 = 49 s of writing -> 196 tokens -> ~120 words. + mockFirstTokenMs = 56_000; + await executeRoutingPlan(p, { query: "Compare them.", styleReminder: "Be thorough.", length: { maxTokens: 512, tokPerSec: 4, ttftMs: 11_500 } }, resolveModel); + const user = generateMock.mock.calls[0][0].messages.at(-1).content; + expect(user).toContain("Be thorough."); + expect(user).toContain("about 120 words"); + // No reading speed yet: the last answer's first word is the estimate (11.5 s -> 90 s of writing -> 360 tokens). + generateMock.mockClear(); + mockFirstTokenMs = null; + await executeRoutingPlan(p, { query: "Compare them.", length: { maxTokens: 512, tokPerSec: 4, ttftMs: 11_500 } }, resolveModel); + expect(generateMock.mock.calls[0][0].messages.at(-1).content).toContain("about 220 words"); + }); + + it("sizes the answer to the step's token cap when it is below the setting", async () => { + const p = plan({ + steps: [{ id: "generate-0", type: "generate", modelId: "qwen-1.5b", required: true, maxTokens: 200 }], + selectedModelIds: ["qwen-1.5b"], + }); + mockFirstTokenMs = null; + // 4 tok/s for 90 s would be 360 tokens (~220 words); the step stops at 200 tokens -> ~120 words. + await executeRoutingPlan(p, { query: "Compare them.", length: { maxTokens: 512, tokPerSec: 4, ttftMs: 11_500 } }, resolveModel); + expect(generateMock.mock.calls[0][0].messages.at(-1).content).toContain("about 120 words"); + expect(generateMock.mock.calls[0][0].nPredict).toBe(200); + }); + + it("sizes the answer with the speed of the model the router picked, not the base model's", async () => { + const p = plan({ + steps: [{ id: "generate-0", type: "generate", modelId: "qwen-1.5b", required: true }], + selectedModelIds: ["qwen-1.5b"], + }); + mockFirstTokenMs = null; + // Base model: 4 tok/s (-> ~220 words, above). The picked 1.5B: 2 tok/s, 11.5 s first word -> 180 tokens -> ~110 words. + const speedFor = (id: string) => (id === "qwen-1.5b" ? { tokPerSec: 2, ttftMs: 11_500 } : {}); + await executeRoutingPlan(p, { query: "Compare them.", length: { maxTokens: 512, tokPerSec: 4, ttftMs: 11_500, speedFor } }, resolveModel); + expect(generateMock.mock.calls[0][0].messages.at(-1).content).toContain("about 110 words"); + }); + it("Qwen (usesChatTemplate: true) receives a role-separated messages array instead of a prompt string", async () => { const p = plan({ steps: [{ id: "generate-0", type: "generate", modelId: "qwen-1.5b", required: true }], diff --git a/src/routing/executor.ts b/src/routing/executor.ts index e73a9c853..121d67f00 100644 --- a/src/routing/executor.ts +++ b/src/routing/executor.ts @@ -23,6 +23,8 @@ import { retrieve } from "../rag/retrieve"; import { assemblePrompt, assembleChatMessages, ConversationHistory, ANSWER_CONTEXT_CHUNKS } from "../rag/pure"; import { compressForAnswer } from "../rag/compress"; import { classifyTask } from "./classify"; +import { searchWithFollowUp } from "./followUp"; +import { answerLength, lengthInstruction } from "./answerLength"; import type { RetrievedChunk } from "../rag/retrieve.types"; import { RoutingPlan, RoutingStep } from "./router"; import { buildVerificationInput, parseVerificationVerdict, VERIFICATION_INSTRUCTION } from "./verify"; @@ -62,6 +64,13 @@ export interface PipelineInput { /** Personality.styleReminder of the selected tone. */ styleReminder?: string; history?: ConversationHistory; + /** + * Size the answer to the phone (answerLength.ts) once the sources are known: the time to the first word is + * estimated from this prompt's own size. `ttftMs` is the fallback when there is no reading speed yet. + * `speedFor` gives the speed of the model the router picked (it can differ from the one `tokPerSec` was + * measured on); without it, `tokPerSec` and `ttftMs` are used. + */ + length?: { maxTokens: number; tokPerSec?: number; ttftMs?: number; speedFor?: (modelId: string) => { tokPerSec?: number; ttftMs?: number } }; } export interface PipelineCallbacks { @@ -225,7 +234,9 @@ export async function executeRoutingPlan( switch (step.type) { case "retrieve": { const retrieveStart = performance.now(); - const compressed = compressForAnswer(input.query, await retrieve(input.query, ANSWER_CONTEXT_CHUNKS), classifyTask(input.query)); + // A follow-up ("What did he publish?") is searched with the previous question (followUp.ts). + const found = await searchWithFollowUp(input.query, input.history, (text) => retrieve(text, ANSWER_CONTEXT_CHUNKS)); + const compressed = compressForAnswer(found.searchedFor, found.chunks, classifyTask(input.query)); retrievalMs = performance.now() - retrieveStart; citations = compressed.chunks; contextTokensBefore = compressed.tokensBefore; @@ -268,16 +279,31 @@ export async function executeRoutingPlan( : !!generateModel?.usesChatTemplate; promptFormat = useTemplate ? "chat-template" : "plain"; + // The answer's length, from this prompt's size: a question with more sources or history reads longer + // before its first word, and leaves less of the step timeout for writing. + let styleReminder = input.styleReminder; + if (input.length) { + const draft = useTemplate + ? { messages: assembleChatMessages(input.query, citations, input.systemPrompt, input.history, input.styleReminder) } + : { prompt: assemblePrompt(input.query, citations, input.systemPrompt, input.history, input.styleReminder) }; + const estimated = await llamaEngine.estimateFirstTokenMs(draft); + const speed = input.length.speedFor && step.modelId ? input.length.speedFor(step.modelId) : input.length; + // The step's own cap is what generation stops at (nPredict below), when it is lower than the setting. + const maxTokens = Math.min(input.length.maxTokens, step.maxTokens ?? 512); + const len = answerLength({ maxTokens, tokPerSec: speed.tokPerSec, ttftMs: estimated ?? speed.ttftMs }); + styleReminder = [input.styleReminder, lengthInstruction(len.words)].filter(Boolean).join("\n"); + } + answer = useTemplate ? await llamaEngine.generate({ - messages: assembleChatMessages(input.query, citations, input.systemPrompt, input.history, input.styleReminder), + messages: assembleChatMessages(input.query, citations, input.systemPrompt, input.history, styleReminder), nPredict: step.maxTokens ?? 512, onToken: timedOnToken, timeoutMs: step.timeoutMs ?? STEP_TIMEOUT_MS, onTimeout: markTimedOut, }) : await llamaEngine.generate({ - prompt: assemblePrompt(input.query, citations, input.systemPrompt, input.history, input.styleReminder), + prompt: assemblePrompt(input.query, citations, input.systemPrompt, input.history, styleReminder), nPredict: step.maxTokens ?? 512, onToken: timedOnToken, timeoutMs: step.timeoutMs ?? STEP_TIMEOUT_MS, diff --git a/src/routing/followUp.test.ts b/src/routing/followUp.test.ts new file mode 100644 index 000000000..b1a66c618 --- /dev/null +++ b/src/routing/followUp.test.ts @@ -0,0 +1,91 @@ +import { describe, expect, it } from "vitest"; +import { followUpSearch, searchWithFollowUp } from "./followUp"; + +const after = (question: string) => ({ + turns: [ + { role: "user" as const, text: question }, + { role: "assistant" as const, text: "An answer about it." }, + ], +}); + +describe("followUpSearch", () => { + it("searches a question that points back together with the previous question", () => { + expect(followUpSearch("What did he publish?", after("Who proposed the theory of evolution by natural selection?"))).toEqual({ + direct: "Who proposed the theory of evolution by natural selection? What did he publish?", + previous: "Who proposed the theory of evolution by natural selection?", + }); + }); + + it("does the same in Portuguese", () => { + expect(followUpSearch("O que ele publicou?", after("Quem propôs a teoria da evolução?")).direct).toContain("Quem propôs"); + }); + + it("borrows the previous topic for a short question only as a fallback", () => { + expect(followUpSearch("What are the side effects?", after("How do vaccines work?"))).toEqual({ + fallback: "How do vaccines work? What are the side effects?", + }); + }); + + it("never borrows for a question that names its own subject", () => { + expect(followUpSearch("What is the capital of Australia?", after("Compare the French Revolution and the Industrial Revolution."))).toEqual({}); + }); + + it("leaves a longer self-contained question alone", () => { + expect(followUpSearch("How does the immune system respond to a new virus infection?", after("How do vaccines work?"))).toEqual({}); + }); + + it("never takes the current question as the previous one", () => { + const history = { + turns: [ + { role: "user" as const, text: "Who proposed the theory of evolution?" }, + { role: "assistant" as const, text: "Charles Darwin." }, + { role: "user" as const, text: "What did he publish?" }, + ], + }; + expect(followUpSearch("What did he publish?", history).direct).toBe("Who proposed the theory of evolution? What did he publish?"); + }); + + it("does nothing without an earlier question", () => { + expect(followUpSearch("What did he publish?", { turns: [] })).toEqual({}); + expect(followUpSearch("What did he publish?")).toEqual({}); + }); + + it("does not read 'the' or 'there' inside other words as pointing back", () => { + expect(followUpSearch("How does photosynthesis work in theory?", after("How do vaccines work?")).direct).toBeUndefined(); + }); +}); + +describe("searchWithFollowUp", () => { + const history = after("How do vaccines work?"); + + it("keeps what the question alone found", async () => { + const seen: string[] = []; + const r = await searchWithFollowUp("What are the side effects?", history, async (t) => (seen.push(t), ["chunk"])); + expect(r).toEqual({ chunks: ["chunk"], searchedFor: "What are the side effects?" }); + expect(seen).toHaveLength(1); + }); + + it("retries with the previous question when the short one finds nothing", async () => { + const seen: string[] = []; + const r = await searchWithFollowUp("What are the side effects?", history, async (t) => (seen.push(t), t.includes("vaccines") ? ["vaccine side effects"] : [])); + expect(r.chunks).toEqual(["vaccine side effects"]); + expect(r.searchedFor).toBe("How do vaccines work? What are the side effects?"); + expect(seen).toEqual(["What are the side effects?", "How do vaccines work? What are the side effects?"]); + }); + + it("searches a question that points back combined, once, when that finds sources", async () => { + const seen: string[] = []; + const r = await searchWithFollowUp("What did he publish?", after("Who proposed evolution?"), async (t) => (seen.push(t), ["darwin"])); + expect(seen).toEqual(["Who proposed evolution? What did he publish?"]); + expect(r.searchedFor).toBe("Who proposed evolution? What did he publish?"); + }); + + it("falls back to the previous question alone, still compressing against the combined words", async () => { + const seen: string[] = []; + const r = await searchWithFollowUp("What did he publish?", after("Who proposed evolution?"), async (t) => + (seen.push(t), t === "Who proposed evolution?" ? ["Evolution article"] : []) + ); + expect(seen).toEqual(["Who proposed evolution? What did he publish?", "Who proposed evolution?"]); + expect(r).toEqual({ chunks: ["Evolution article"], searchedFor: "Who proposed evolution? What did he publish?" }); + }); +}); diff --git a/src/routing/followUp.ts b/src/routing/followUp.ts new file mode 100644 index 000000000..389d21178 --- /dev/null +++ b/src/routing/followUp.ts @@ -0,0 +1,92 @@ +/** + * What to search the library for when the question is a follow-up. + * + * Retrieval saw only the question's own words, so a follow-up found nothing: + * "What did he publish?" after a Darwin answer, "What are the side effects?" + * after a vaccines answer (Pixel 6a, 2026-10-04, both models: no sources). The + * model still resolved "he" from the history and answered from memory, uncited, + * which is the weaker case for the bounty (Qwen3-4B 0.49 from memory vs 0.61 + * with passages, r4to's eval). + * + * Rule, from the conversation's previous question: + * - A question that points back ("he", "it", "that", "ele", "isso"…) is searched + * together with the previous question, and with the previous question alone + * when that finds nothing. + * - A short question with no name of its own ("What are the side effects?") is + * searched as written first, and with the previous question only when that + * finds nothing. A self-contained question that finds sources keeps them, and + * one with a name ("the capital of Australia") never borrows another topic. + * + * Pure (no native imports), like context.ts, so it is unit-testable. + */ +import type { ConversationHistory } from "../rag/pure"; +import { tokenizeTerms } from "./context"; + +/** Words that only make sense with an earlier subject (English and Portuguese). */ +const POINTS_BACK = + /(?:^|[^\p{L}])(he|she|it|its|they|them|their|theirs|him|his|her|hers|this|that|these|those|there|ele|ela|eles|elas|dele|dela|deles|delas|isso|isto|aquilo|nele|nela|esse|essa|esses|essas)(?![\p{L}])/iu; + +/** At most this many content words for a question to borrow the previous one's topic when it finds nothing. */ +export const SHORT_FOLLOW_UP_TERMS = 3; + +export interface FollowUpSearch { + /** Search this instead of the question (it points back). */ + direct?: string; + /** + * With `direct`: the previous question alone, when the combined text finds nothing. The keyword search + * wants most of the query's words in a source, and the combined query has many: "Who proposed the theory + * of evolution… What did he publish?" found 0 chunks, the Evolution article lacking "publish" (Pixel 6a). + */ + previous?: string; + /** Search this only if the question alone found nothing (short, no name of its own). */ + fallback?: string; +} + +/** The latest earlier question, never the current one (a history that already holds it would make it its own subject). */ +function previousQuestion(query: string, history?: ConversationHistory): string | null { + const turns = history?.turns ?? []; + const current = query.trim(); + for (let i = turns.length - 1; i >= 0; i--) { + const text = turns[i].text.trim(); + if (turns[i].role === "user" && text && text !== current) return text; + } + return null; +} + +/** A capitalized word after the first one: a name the question brings itself ("Australia", "EIP-1559"). */ +function namesItsOwnSubject(query: string): boolean { + const words = query.trim().split(/\s+/).slice(1); + return words.some((w) => /^[\p{Lu}0-9]/u.test(w.replace(/^["'(«“]/, ""))); +} + +export function followUpSearch(query: string, history?: ConversationHistory): FollowUpSearch { + const previous = previousQuestion(query, history); + if (!previous) return {}; + const combined = `${previous} ${query.trim()}`; + if (POINTS_BACK.test(query)) return { direct: combined, previous }; + if (tokenizeTerms(query).length <= SHORT_FOLLOW_UP_TERMS && !namesItsOwnSubject(query)) return { fallback: combined }; + return {}; +} + +/** + * Runs the search with the follow-up rule. `search` gets the text to search and + * returns its chunks; the result says which text found them, so the caller + * compresses against the same words. + */ +export async function searchWithFollowUp( + query: string, + history: ConversationHistory | undefined, + search: (text: string) => Promise +): Promise<{ chunks: T[]; searchedFor: string }> { + const plan = followUpSearch(query, history); + if (plan.direct) { + const chunks = await search(plan.direct); + if (chunks.length || !plan.previous) return { chunks, searchedFor: plan.direct }; + // Found by the earlier subject; compressed against the combined words, so the sentences closest to the + // follow-up ("publish") are the ones kept. + return { chunks: await search(plan.previous), searchedFor: plan.direct }; + } + const chunks = await search(query); + if (chunks.length || !plan.fallback) return { chunks, searchedFor: query }; + return { chunks: await search(plan.fallback), searchedFor: plan.fallback }; +} diff --git a/src/routing/historyBudget.test.ts b/src/routing/historyBudget.test.ts new file mode 100644 index 000000000..44eb90ea2 --- /dev/null +++ b/src/routing/historyBudget.test.ts @@ -0,0 +1,58 @@ +import { describe, expect, it } from "vitest"; +import { budgetHistory, leadSentences } from "./historyBudget"; + +const longAnswer = (topic: string) => + Array.from({ length: 40 }, (_, i) => `Sentence ${i + 1} about ${topic} explains one more detail of it.`).join(" "); + +describe("leadSentences", () => { + it("keeps short text as is", () => { + expect(leadSentences("Canberra is the capital.", 80)).toBe("Canberra is the capital."); + }); + + it("cuts a long answer to its opening sentences within the budget", () => { + const out = leadSentences(longAnswer("vaccines"), 40); + expect(out.startsWith("Sentence 1 about vaccines")).toBe(true); + expect(out.endsWith("…")).toBe(true); + expect(Math.ceil(out.length / 4)).toBeLessThanOrEqual(42); + }); + + it("cuts a single over-long sentence by characters", () => { + const out = leadSentences("word ".repeat(200).trim() + ".", 10); + expect(out.length).toBeLessThanOrEqual(42); + }); +}); + +describe("budgetHistory", () => { + it("keeps questions whole and shortens earlier answers", () => { + const turns = [ + { role: "user" as const, text: "How do vaccines work?" }, + { role: "assistant" as const, text: longAnswer("vaccines") }, + ]; + const out = budgetHistory(turns); + expect(out[0]).toEqual(turns[0]); + expect(out[1].text.length).toBeLessThan(turns[1].text.length); + }); + + it("drops the oldest turns once the budget is spent, newest first", () => { + const turns = ["vaccines", "evolution", "revolutions"].flatMap((t) => [ + { role: "user" as const, text: `Tell me about ${t}.` }, + { role: "assistant" as const, text: longAnswer(t) }, + ]); + const out = budgetHistory(turns, { budgetTokens: 90, answerTokens: 40 }); + expect(out[out.length - 1].text).toContain("revolutions"); + expect(out.some((t) => t.text.includes("vaccines"))).toBe(false); + expect(out.reduce((n, t) => n + Math.ceil(t.text.length / 4), 0)).toBeLessThanOrEqual(90); + }); + + it("always keeps the latest exchange, even over budget", () => { + const turns = [ + { role: "user" as const, text: "q ".repeat(300) }, + { role: "assistant" as const, text: longAnswer("x") }, + ]; + expect(budgetHistory(turns, { budgetTokens: 10 })).toHaveLength(2); + }); + + it("is a no-op for an empty history", () => { + expect(budgetHistory([])).toEqual([]); + }); +}); diff --git a/src/routing/historyBudget.ts b/src/routing/historyBudget.ts new file mode 100644 index 000000000..34682dc7c --- /dev/null +++ b/src/routing/historyBudget.ts @@ -0,0 +1,56 @@ +/** + * Conversation history that fits a token budget, for the answer prompt. + * + * The chat sent the last 6 messages verbatim, counted in messages. With the + * detailed personality each answer is ~500 tokens, so history alone grew to + * ~1,700 tokens and the first word of a follow-up took 20-48 s on a Pixel 6a + * (1.5B, 2026-10-03), against 1.6 s for the same question in a new chat. + * + * Rule: newest turns first, within `budgetTokens`. A question is kept whole; + * an earlier answer is cut to its opening sentences (`answerTokens`), since a + * follow-up mostly needs what the answer was about, and the full text is still + * on screen. The latest exchange is always kept, even over budget. + * + * Pure (no native imports), like context.ts, so it is unit-testable. + */ +import type { ConversationTurn } from "../rag/pure"; +import { approxTokens, splitSentences } from "./context"; + +export const HISTORY_BUDGET_TOKENS = 400; +export const HISTORY_ANSWER_TOKENS = 80; + +/** The opening sentences of `text` within `maxTokens`; at least the first sentence, cut by characters if it alone is too long. */ +export function leadSentences(text: string, maxTokens: number, count = approxTokens): string { + if (count(text) <= maxTokens) return text; + const sentences = splitSentences(text); + let out = ""; + for (const s of sentences) { + const next = out ? `${out} ${s}` : s; + if (count(next) > maxTokens) break; + out = next; + } + if (!out) out = `${(sentences[0] ?? text).slice(0, maxTokens * 4).trimEnd()}`; + return `${out} …`; +} + +export function budgetHistory( + turns: ConversationTurn[], + opts: { budgetTokens?: number; answerTokens?: number; countTokens?: (s: string) => number } = {} +): ConversationTurn[] { + const budget = opts.budgetTokens ?? HISTORY_BUDGET_TOKENS; + const answerTokens = opts.answerTokens ?? HISTORY_ANSWER_TOKENS; + const count = opts.countTokens ?? approxTokens; + + const kept: ConversationTurn[] = []; + let used = 0; + for (let i = turns.length - 1; i >= 0; i--) { + const t = turns[i]; + const text = t.role === "assistant" ? leadSentences(t.text, answerTokens, count) : t.text; + const cost = count(text); + // The latest exchange (one question and its answer) always goes in. + if (used + cost > budget && kept.length >= 2) break; + kept.unshift({ role: t.role, text }); + used += cost; + } + return kept; +} diff --git a/src/ui-android/ChatScreen.tsx b/src/ui-android/ChatScreen.tsx index deb7e4f4d..985a090b3 100644 --- a/src/ui-android/ChatScreen.tsx +++ b/src/ui-android/ChatScreen.tsx @@ -25,6 +25,10 @@ import { llamaEngine } from "../inference/LlamaEngine"; import { embeddingEngine } from "../rag/embed"; import { retrieve, assemblePrompt, RetrievedChunk, ConversationTurn } from "../rag/retrieve"; import { assembleChatMessages, ANSWER_CONTEXT_CHUNKS } from "../rag/pure"; +import { budgetHistory } from "../routing/historyBudget"; +import { searchWithFollowUp } from "../routing/followUp"; +import { answerLength, currentSpeed, lengthInstruction } from "../routing/answerLength"; +import { measuredSpeeds } from "../routing/depth"; import { compressForAnswer } from "../rag/compress"; import { classifyTask, isRetrievalIrrelevant } from "../routing/classify"; import type { TaskType } from "../routing/types"; @@ -82,7 +86,7 @@ import { splitThinking, stripThinking } from "../services/thinking"; import { cleanCitations } from "../services/citations"; import { SourceFootnotes } from "./components/SourceFootnotes"; import { recordQueryStats, trackPeakRss, startAppMemoryTracking, QueryStats } from "../services/telemetry"; -import { recordExecution } from "../services/executionTelemetry"; +import { listRecentExecutions, recordExecution } from "../services/executionTelemetry"; import { getMemoryInfo } from "ram-monitor"; import { useTheme, colors, typography } from "./theme"; import { spacing, radii } from "./theme/spacing"; @@ -135,6 +139,8 @@ export function ChatScreen({ const [loadStatus, setLoadStatus] = useState(t("chatScreen.initializingCore")); const [loadError, setLoadError] = useState(null); const [generating, setGenerating] = useState(false); + /** Set synchronously when a send starts, so a second tap can't slip in before `generating` updates. */ + const sendingRef = useRef(false); const [showPromptIdeas, setShowPromptIdeas] = useState(false); const [drawerOpen, setDrawerOpen] = useState(false); const [showAbout, setShowAbout] = useState(false); @@ -509,28 +515,40 @@ export function ChatScreen({ /** `text`: send this instead of the input box (a prompt idea). */ const send = useCallback(async (text?: string) => { const query = (text ?? input).trim(); - if (!query || generating) return; - - await cancelBackgroundTask(); - + // sendingRef, not only `generating`: the state flips after the awaits below, and taps during + // cancelBackgroundTask (a summary still reading its prompt, tens of seconds) each sent the + // question again, all with the same millisecond ids (UNIQUE constraint failed: chat_messages.id). + if (!query || generating || sendingRef.current) return; + sendingRef.current = true; setInput(""); setGenerating(true); - stopRequestedRef.current = false; let sessionId = activeSessionId; const isNewSession = !sessionId; - if (!sessionId) { - const session = await createSession(); - sessionId = session.id; - setActiveSessionId(sessionId); + // Unique even for two sends in the same millisecond. + const stamp = `${Date.now()}-${Math.random().toString(36).slice(2, 8)}`; + const userMsg: Message = { id: `${stamp}-u`, role: "user", text: query }; + const assistantId = `${stamp}-a`; + try { + await cancelBackgroundTask(); + stopRequestedRef.current = false; + if (!sessionId) { + const session = await createSession(); + sessionId = session.id; + setActiveSessionId(sessionId); + } + // Explicit id, matching the in-memory message id used for the FlatList + // key/React state — so a thumbs-up/down tap on this exact message can + // reference `item.id` directly as answer_feedback's foreign key later, + // rather than needing a separate id-mapping step. + await persistMessage(sessionId, "user", query, userMsg.id); + } catch (e: any) { + console.warn("[ChatScreen] send failed before generation:", e?.message ?? e); + sendingRef.current = false; + setGenerating(false); + setInput(query); + return; } - const userMsg: Message = { id: `${Date.now()}-u`, role: "user", text: query }; - const assistantId = `${Date.now()}-a`; - // Explicit id, matching the in-memory message id used for the FlatList - // key/React state — so a thumbs-up/down tap on this exact message can - // reference `item.id` directly as answer_feedback's foreign key later, - // rather than needing a separate id-mapping step. - await persistMessage(sessionId, "user", query, userMsg.id); setMessages((prev) => [...prev, userMsg, { id: assistantId, role: "assistant", text: "" }]); setProcessing({ messageId: assistantId, status: "retrieving" }); @@ -551,19 +569,35 @@ export function ChatScreen({ let assistantText = ""; try { - const [maxTokens, activePersonalityId, customPrompt] = await Promise.all([ + const [maxTokens, activePersonalityId, customPrompt, recent] = await Promise.all([ getMaxTokens(), getPersonalityId(), getCustomSystemPrompt(), + listRecentExecutions(100).catch(() => []), ]); const personality = getPersonality(activePersonalityId); const systemPrompt = activePersonalityId === "custom" ? customPrompt : personality.systemPrompt; - const styleReminder = activePersonalityId === "custom" ? undefined : personality.styleReminder; - - const priorMessages = messagesRef.current.filter((m) => m.text.length > 0); - const verbatimTurns: ConversationTurn[] = priorMessages - .slice(-VERBATIM_MESSAGE_COUNT) - .map((m) => ({ role: m.role, text: m.role === "assistant" ? stripThinking(m.text) : m.text })); + // A length this phone can write in time, so the answer ends instead of being cut (answerLength.ts). + const typical = measuredSpeeds(recent); + // Per model: the router can answer with another model than the base one. + const speedFor = (id: string) => currentSpeed(recent, id, Date.now(), typical.get(id)); + const speed = baseModelId ? speedFor(baseModelId) : {}; + const length = answerLength({ maxTokens, ...speed }); + const baseStyleReminder = activePersonalityId === "custom" ? undefined : personality.styleReminder; + // The router sizes the answer itself once the sources are known (executor.ts, `length`); the fixed-model + // fallback below still gets this up-front target. + const styleReminder = [baseStyleReminder, lengthInstruction(length.words)].filter(Boolean).join("\n"); + + // Not this send's own messages: setMessages above can reach messagesRef before this line, and the + // question then went into the history too (asked twice in the prompt, and a follow-up searched for + // itself: "What did he publish? What did he publish?", Pixel 6a). + const priorMessages = messagesRef.current.filter((m) => m.text.length > 0 && m.id !== userMsg.id && m.id !== assistantId); + // Within a token budget: earlier answers cut to their opening sentences (historyBudget.ts). + const verbatimTurns: ConversationTurn[] = budgetHistory( + priorMessages + .slice(-VERBATIM_MESSAGE_COUNT) + .map((m) => ({ role: m.role, text: m.role === "assistant" ? stripThinking(m.text) : m.text })) + ); const history = { summary: sessionSummaryRef.current, turns: verbatimTurns }; let firstToken = true; @@ -627,8 +661,10 @@ export function ChatScreen({ c = []; } else { setProcessing({ messageId: assistantId, status: "retrieving" }); - // Only the sentences that answer: a shorter prompt brings the first word sooner. - c = compressForAnswer(query, await retrieve(query, ANSWER_CONTEXT_CHUNKS), fixedTaskType).chunks; + // Only the sentences that answer: a shorter prompt brings the first word sooner. A follow-up + // ("What did he publish?") is searched with the previous question (followUp.ts). + const found = await searchWithFollowUp(query, history, (text) => retrieve(text, ANSWER_CONTEXT_CHUNKS)); + c = compressForAnswer(found.searchedFor, found.chunks, fixedTaskType).chunks; } setProcessing({ messageId: assistantId, status: "thinking" }); // Use the model's own chat template when its file ships one; the @@ -647,7 +683,7 @@ export function ChatScreen({ if (adaptiveRoutingEnabled) { try { const result = await runAdaptiveChat( - { query, systemPrompt, styleReminder, history }, + { query, systemPrompt, styleReminder: baseStyleReminder, history, length: { maxTokens, ...speed, speedFor } }, maxTokens, { onToken, @@ -799,10 +835,14 @@ export function ChatScreen({ const olderMessages = allMessages.slice(0, -VERBATIM_MESSAGE_COUNT); if (olderMessages.length > 0) { const sid = sessionId; - const turnsToSummarize: ConversationTurn[] = olderMessages.map((m) => ({ - role: m.role, - text: m.role === "assistant" ? stripThinking(m.text) : m.text, - })); + // Bounded like the answer's history: the whole detailed chat (thousands of tokens) took tens + // of seconds to read, and the next question waited for it (cancelBackgroundTask). + const turnsToSummarize: ConversationTurn[] = budgetHistory( + olderMessages.map((m) => ({ + role: m.role, + text: m.role === "assistant" ? stripThinking(m.text) : m.text, + })) + ); const previousSummary = sessionSummaryRef.current; backgroundTaskRef.current = summarizeConversation(turnsToSummarize, previousSummary) .then(async (summary) => { @@ -838,6 +878,7 @@ export function ChatScreen({ errorMessage: e?.message ?? String(e), }).catch(() => {}); } finally { + sendingRef.current = false; setGenerating(false); setStopping(false); setProcessing(null);