Skip to content
This repository was archived by the owner on Nov 12, 2026. It is now read-only.

Commit fc2c369

Browse files
techmoreclaude
andauthored
Docs: uzu and omarchy-ane evaluation on the M1 Pro (#640)
* Compositor: Command+Shift+3/4 screenshots also match Ctrl (ctrl:swap_lwin_lctl) (#624) Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> * Document uzu and omarchy-ane evaluation on the M1 Pro Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
1 parent ad496f4 commit fc2c369

1 file changed

Lines changed: 38 additions & 0 deletions

File tree

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# uzu and omarchy-ane evaluation — October 27, 2026
2+
3+
Target: M1 Pro (T6000, j314s), kernel 7.2.11-elementary-aurora72-wip4. Question: can Mirai's [uzu](https://github.com/trymirai/uzu) or [omarchy-ane](https://github.com/joshuaswarren/omarchy-ane) give faster local AI than the current MLX-on-Honeykrisp path?
4+
5+
## Verdict
6+
7+
Keep MLX on Honeykrisp (`mseries-ai-everyday`, Qwen3.5 4B 4-bit) as the GPU path. Do not add uzu to `rust/mseries-ai`. The only ANE item worth pursuing is the Parakeet dictation engine, for CPU and power, not speed.
8+
9+
## uzu
10+
11+
- MIT, v0.6.13 (2026-10-22), Rust workspace, edition 2024. `rust-toolchain.toml` pins `nightly`; running cargo in the checkout auto-installs a 2.8 GB nightly. It is **not required**: the CLI builds on stable 1.99 with `RUSTUP_TOOLCHAIN=stable cargo build --release -p cli --no-default-features --features backend-cpu` (3m15s, 2.3 GB target, 44 MB binary). A scratch `alsa.pc` is needed because `libasound2-dev` is not installed (rodio pulls alsa-sys).
12+
- Backends: `cpu` and `metal` only. The Linux aarch64 target is "in progress" in its README. CPU kernels are scalar reference code (no NEON).
13+
- GPU kernels: 51 hand-written `.metal` files (about 13k lines with headers), a macro layer (`VARIANTS`, `KERNEL`, `GROUPS`, `THREADS`, `SPECIALIZE`) and a `build.rs` that generates Rust bindings. A second GPU backend means rewriting every kernel.
14+
- Upstream: Vulkan scaffold PR #444 was closed unmerged (2026-05-26). Maintainers: WebGPU first (branches `webgpu`, `vulkan`), Vulkan later on top of it, no date.
15+
- Own model format via `lalamo`; registry downloads come from Mirai's service. The CLI has an OpenAI-compatible `server` subcommand.
16+
- Measured (`alibaba:qwen3.5:0.8b:mirai:mirai-m:4`, about 439 MB, SMS-reply prompt): time to first token 65 s, prefill 0.54 tokens/s, generation 0.36 tokens/s, 247 s total, 556 MB peak RSS.
17+
18+
## Metal on Linux
19+
20+
Metal does not exist on Linux, and no Metal-to-SPIR-V route was found in uzu's repo or tracker (SPIRV-Cross and naga only emit MSL; Apple's compiler is macOS-only; not searched beyond that). MLX on Honeykrisp Vulkan already runs on the Apple GPU: Vulkan 1.4, subgroup 32, fp16/int8/int dot, 16-bit and 8-bit storage, 32 KiB shared memory, no cooperative matrix, 15.8 GiB unified heap, private Mesa 26.3.0-devel.
21+
22+
## Current path, measured live (desktop busy, fan about 2300 rpm)
23+
24+
`mseries-ai-everyday`, Qwen3.5-4B-MLX-4bit: 35-token prompt, 0.8 s to first token warm (5 s cold); 620-token prompt, 4.0 s (about 154 tokens/s); generation about 8-10 tokens/s under load (counted by streamed chunks, approximate). Service memory 6.0 GB current, 6.25 GB peak. The quiet-machine receipt (`receipts/2026-10-16.md`) records pp512 221.7 and tg128 25.2 tokens/s. The first start attempt was refused by the admission policy (11 GiB available, 12.5 needed); a later one passed.
25+
26+
## ANE
27+
28+
- The driver is in the kernel: `CONFIG_DRM_ACCEL_ANE=m`, `ane.ko` and `ane_t6021.ko` in `drivers/accel/ane`, "Dual MIT/GPL", loaded at boot, `/dev/accel/accel0`, node `/soc/ane@285c04000` (`apple,t6000-ane`). omarchy-ane's `dkms.conf` excludes itself on such kernels, so no DKMS and no overlay are needed.
29+
- `omarchy-ane-check` (v0.4.9 copy, read-only): device tree, module, load and bind all pass. The one FAIL is `/dev/accel/accel0` being `root:render` 0660 (a udev rule overrides systemd's 0666); users in `render` are unaffected.
30+
- Earlier results (`mseries/dictation/STT-RESEARCH.md` addenda): smoke 20/20 bit-exact; Parakeet encoder matches with private Mesa 26.2.4; 1.98 s for 10.4 s audio versus 1.07 s for CPU whisper on 7 s. No speed win; benefit would be CPU and power (unmeasured).
31+
- omarchy-ane documents a staged Qwen3.8-2B decode on a 13-inch M1 (same H13 family): step p50 0.086 s fp16, 0.061 s int8, about 12-16 tokens/s. It is its own Python runtime with compiler-built packages not in the repo; not tried here.
32+
- Risk: a hung ANE submission needs a reboot, per upstream.
33+
34+
## Next steps (owner decides)
35+
36+
1. Measure CPU and power of the ANE dictation engine against CPU whisper.
37+
2. Optionally try the omarchy-ane Qwen decode runtime.
38+
3. Revisit uzu only if it gains a Vulkan/WebGPU backend or NEON CPU kernels.

0 commit comments

Comments
 (0)