feat(starry): sysbench big.LITTLE benchmark and RK3588 board harness

apps/starry/sysbench-board/COMPARISON.md

@@ -0,0 +1,68 @@
+# Linux vs StarryOS sysbench (OrangePi-5-Plus RK3588, 2026-07-20)
+
+Same board, same binary (glibc `sysbench 1.0.20`), same commands
+(`harness/sysbench-compare.sh`). StarryOS = this worktree's kernel with the
+ondemand DVFS governor + the axtask spawn/wake distribution fix. Linux = board
+Ubuntu 6.1.43-rockchip (native DVFS to 2.4 GHz). Two StarryOS columns show the OPP
+ceiling before and after "push the ceiling" (A76 1592→1725 MHz, A55 1372→1523 MHz,
+the calibrated safe max at 850→925/950 mV).
+
+> **⚠️ Reproducibility / provenance.** The 8-thread rows below (and "threads now
+> use all 8 cores" / "no brownout at threads=8") are a **historical research
+> measurement taken on a kernel built with `max_cpu_num = 8`** plus the
+> uncommitted DVFS-ceiling and ring-lever `cpufreq.rs` work on `worktree-sysbench`
+> (2026-07-20). They are **not reproducible from the config shipped in this PR**:
+> `build-aarch64-unknown-none-softfloat.toml` pins `max_cpu_num = 4` on purpose
+> (the board browns out / smp-8 boot is unproven at 8 cores on the test PSU, see
+> the config header + README "Gating risk"). The submitted `init.sh` therefore
+> schedules `--threads=8` on **4** online cores and reproduces the **SMP-4**
+> scaling curve, not the 8-core numbers. Treat the 8-thread column as an
+> archived result of that specific experimental kernel/config, not as an output
+> of this PR's default workflow. To reproduce the 8-core numbers you must build a
+> (currently unshipped, brownout-risk) `max_cpu_num = 8` kernel.
+
+## Results
+
+| metric                     | StarryOS @1592/1372 | StarryOS @1725/1523 | Linux   | Linux / StarryOS(top) |
+|----------------------------|---------------------|---------------------|---------|-----------------------|
+| cpu events/s, 1 thread     | 271                 | 300                 | 974     | 3.2x                  |
+| cpu events/s, 2 threads    | 938                 | 1471                | 1954    | 1.33x                 |
+| cpu events/s, 4 threads    | 2697                | ~3300               | 3894    | ~1.2x                 |
+| **cpu events/s, 8 threads**| 3668                | **3991**            | **5322**| **1.33x**             |
+| threads test, events (t=8) | 11339               | 12694               | 50595   | 4.0x                  |
+| mutex total time (t=8, s)  | 1.14                | 1.02                | 0.46    | 2.2x                  |
+| memory write MiB/s (t=8)   | 11573               | 13347               | 54984   | 4.1x                  |
+| memory read  MiB/s (t=8)   | 14161               | 15939               | 55599   | 3.5x                  |
+
+For scale: before this effort StarryOS was **flat at ~160 ev/s** regardless of
+thread count (~33x behind Linux at 8 threads). Governor + scheduler fix took it to
+3668, and pushing the OPP ceiling to **3991 — 1.33x behind Linux**, from 33x.
+
+## Reading it
+
+- **CPU throughput went from ~33x behind Linux to 1.33x.** Three fixes stacked:
+  the DVFS governor (per-core throughput ~2x, 816→1725 MHz on A76), the OPP-ceiling
+  push (voltage-calibrated safe max), and the axtask spawn/wake distribution fix
+  (flat → 13.6x thread scaling; threads spread across the machine — measured on the
+  `max_cpu_num=8` research build, see the provenance note; the shipped SMP-4 config
+  spreads across 4 cores).
+- **The residual CPU gap is frequency, not compute.** Both OSes reach ~96% of
+  their own big.LITTLE aggregate ceiling; per-core compute at equal freq is at
+  parity (StarryOS A76@816 == Linux A76@816, established earlier). StarryOS tops
+  A76 at 1725 MHz (the calibrated over-volted safe max via SCMI+PVTPLL) vs Linux
+  2.4 GHz — 1725/2400 = 0.72, which is the 8-thread gap. The SCMI/PVTPLL coupling
+  plus the PSU envelope prevent matching 2.4 GHz on the current voltage lever;
+  closing it further needs a different clock path or more PSU headroom.
+  StarryOS's boot core is also an A55, hence the larger 1-thread gap.
+- **Scheduler-heavy and memory workloads still lag** (threads 4.0x, mutex 2.2x,
+  memory ~3.5-4x). Separate levers: context-switch/futex overhead (the scheduler
+  effort) and the page-fault / first-touch memory path (roadmap #4), not addressed
+  here. Both narrowed slightly with the higher clocks.
+
+## Provenance
+Raw captures: `snapshots/linux-vs-starry-2026-07-20/` (Linux + StarryOS @1592) and
+the pushed-ceiling run in-line above. Linux baseline matches the earlier RESULTS.md
+run (974/1954/3894/5322). cpuprobe `mhz_pmc` (PMU cycle-counter oracle) confirmed
+the top OPPs on-board: A76 ~1725, A55 ~1520 MHz, exact, no brownout at threads=8
+on that `max_cpu_num=8` research build (the shipped board config caps at SMP-4 for
+PSU margin — see the provenance note above).

apps/starry/sysbench-board/GOVERNOR-VALIDATION.md

@@ -0,0 +1,75 @@
+# Ondemand DVFS governor — board validation (OrangePi-5-Plus RK3588, 2026-07-18)
+
+The `rk3588-cpufreq` driver's dynamic ondemand governor (see
+`drivers/ax-driver/src/soc/rockchip/cpufreq.rs` + the `cpufreq-gov` task in
+`os/StarryOS/kernel/src/entry.rs`) was validated on the board.
+
+## How freq is read
+
+`cpuprobe <cpu>` (harness) pins to a core, self-loads ~1 s, then times a fixed
+compute loop → `ips`. `pmc_ok=0` on StarryOS, so freq is derived from ips vs a
+same-binary baseline: **A55 58.26M ips @ 816 MHz, A76 77.69M ips @ 816 MHz**.
+cpuprobe's own ~1 s self-load means the 100 ms governor has already scaled the
+probed core's cluster **up** by the time it measures — so ips reads the
+governor-chosen (loaded) frequency.
+
+## Results
+
+| build | A55 cpu0–3 | A76 cpu4–7 | note |
+|---|---|---|---|
+| cluster-**average** busy (bug) | ~408 MHz | ~408 MHz | one busy core = 25%/50% of cluster, never crosses 80% up-threshold → never boosts; sits at floor |
+| **per-core** + 1608/1416 caps | — | ~1733 MHz | overshoot: ring=1608 @ 762.5 mV over-delivers (undervolt) |
+| **per-core** + 675 mV caps (final) | **~1018 MHz** (tgt 1008) | **~1186 MHz** (tgt 1200) | exact, voltage-safe |
+
+- **Up-scaling**: per-core scoring boosts a cluster from its busiest CPU (like
+  Linux schedutil/ondemand), so a single CPU-bound thread lifts its cluster.
+  After the fix, cpuprobe ips jumped **4.4×** (A76 37.7M → 165M) and sysbench
+  `cpu --threads=2` throughput **77 → 295 eps**.
+- **Down-scaling**: at idle every cluster decays to the 408 MHz floor (measured
+  directly in the average-logic run, where up-scaling never masked it).
+- **Exactness**: capping every OPP on the 675 mV rail — the only voltage
+  board-proven to make the PVTPLL deliver its SCMI target exactly — removed the
+  overshoot. Governor now scales clock 408↔1200 (A76) / 408↔1008 (A55) at fixed
+  675 mV; no undervolt, no PMIC voltage changes during scaling.
+- **PSU**: `threads=8` all-core ran to completion with **no brownout** (1200/1008
+  @ 675 mV draws less than the ~1490 MHz @ 800 mV overshoot the board already
+  survived).
+
+## Voltage calibration — higher OPPs unlocked (2026-07-20)
+
+An on-board calibration sweep (gated `CALIBRATE` const in `cpufreq.rs`, run from
+`init()` before the console handoff so its `CAL` lines reach serial; measures each
+`(ring, voltage)` point's delivered freq via the PMU cycle counter) resolved the
+>1200 MHz question. Prereq: the cycle counter is now enabled at boot
+(`components/axcpu/src/aarch64/init.rs`), which also makes `cpuprobe`'s `mhz_pmc`
+an exact oracle (`pmc_ok=1`).
+
+Key finding: the delivered freq is dominated by voltage; at any DT `(ring=F,
+V_nom(F))` pair the delivery *over*-shoots F (e.g. ring 1608 @ 762.5 mV → **1733
+MHz** measured, a ~125 mV undervolt). The safe lever is a **fixed low ring with
+rising voltage** — the delivered freq climbs while staying over-volted. Measured
+ladders (both A76 pairs identical):
+
+| A76 @ ring 1200 | 675→1189 | 725→1318 | 800→1491 | 850→**1592** | 925→1725 |
+| A55 @ ring 1008 | 675→1021 | 762.5→1212 | 800→1285 | 850→**1372** | 950→1523 |
+
+Every rung is over-volted (voltage ≥ the delivered freq's DT nominal; margin grows
+from ~0 mV at the base to +100 mV at the top). The governor ladders are now HYBRID:
+ring-scaled below 1200/1008 (idle floor), voltage-scaled above. New tops (capped at
+850 mV, a little above the board-proven 1490 MHz @ 800 mV, below the >1700 @ 925 mV
+brownout risk):
+
+- **A76 1200 → 1592 MHz (+33%)**, **A55 1008 → 1372 MHz (+36%)**.
+
+Board-validated 2026-07-20: under load `mhz_pmc` reads **exactly** 1592.x (A76) /
+1372.x (A55) — no overshoot; **threads=8 all-core held both with no brownout**;
+sysbench eps 201 → 271. Cross-checks (ring 1416 @ 850 → 1830, ring 1608 @ 925 →
+2126) undervolt, confirming fixed-ring voltage-scaling is the safe lever.
+
+## Known limitations / follow-ups
+
+- **`gov:` transition logs aren't capturable** post-boot (StarryOS kernel `info!`
+  stops reaching the serial once the shell owns the console; during boot the
+  cores are busy so no transition fires). cpuprobe freq is the evidence.
+- **sysbench eps is flat** t=2 vs t=8 (~201) — a separate StarryOS SMP-scaling
+  limitation, orthogonal to DVFS.

apps/starry/sysbench-board/README.md

@@ -0,0 +1,92 @@
+# sysbench (StarryOS on OrangePi-5-Plus board)
+
+Runs `sysbench` under StarryOS on the RK3588 board and produces the real
+performance + scaling numbers that QEMU cannot (see `../sysbench/README.md` — the
+QEMU TCG curve is flat by construction). Pairs with a native-Linux run of the
+**same binary on the same board** for an apples-to-apples comparison.
+
+This is the board sibling of the QEMU case `apps/starry/sysbench/`. It is a
+separate directory because a single app case cannot hold both `qemu-*` and
+`board-*` configs (the discovery rejects it).
+
+Selected as `-t sysbench-board`.
+
+## Why the deploy step exists
+
+`cargo xtask starry app board` deploys **only the StarryOS kernel**. Userspace
+binaries must already be on the board's ext4 rootfs (`mmcblk1p2`). StarryOS runs
+against that same rootfs, and it executes glibc dynamic binaries (cf. the
+`glibc-dynamic-smoke` app), so an `apt install sysbench` on the board Linux is
+exactly what StarryOS then runs. The board Linux is Ubuntu/`6.1.43-rockchip`, not
+Alpine — there is no `apk` here, unlike the QEMU case.
+
+## Procedure
+
+The board loop is **Linux (deploy + baseline) → power OFF → ostool → power ON
+(StarryOS run)**. Full board mechanics (ostool-server, `RUSTUP_TOOLCHAIN` trap,
+power-on cue, TFTP-vs-loady) live in the board-run notes; the essentials:
+
+1. **Board in Linux** — stage sysbench, then capture the Linux baseline.
+
+   **Deploy — option A (static binary, recommended, no board internet):** build a
+   fully static aarch64 sysbench on the host and scp it in. This runs identically
+   under Linux and StarryOS and avoids the shared-library-resolution risk that a
+   glibc-dynamic (apt) sysbench carries under StarryOS's loader.
+   ```bash
+   bash build-static-sysbench.sh          # -> ./sysbench-static-aarch64 (confirm `file` says statically linked)
+   scp sysbench-static-aarch64 orangepi@192.168.50.2:/tmp/sysbench
+   ssh orangepi@192.168.50.2 '
+     printf orangepi | sudo -S install -m755 /tmp/sysbench /usr/bin/sysbench
+     printf orangepi | sudo -S sync
+     /usr/bin/sysbench --version'
+   ```
+
+   **Deploy — option B (apt, only if the board has its own internet):**
+   ```bash
+   ssh orangepi@192.168.50.2 'bash -s' < deploy-sysbench.sh    # apt install + sync
+   ```
+
+   Then capture the native-Linux baseline (same binary, same board):
+   ```bash
+   ssh orangepi@192.168.50.2 'bash -s' < linux-baseline.sh | tee linux-baseline.out
+   ```
+   The `sync` after deploy is mandatory either way — the root mounts `commit=600`,
+   so an unsynced binary is invisible to StarryOS after the re-mount (`not found`).
+
+2. **Power the board OFF** (ostool power is a no-op; it must be off when ostool
+   launches, powered on at the "Waiting for remote board to power on…" cue).
+
+3. **Run StarryOS** (note `env -u RUSTUP_TOOLCHAIN` — the shell exports a stable
+   toolchain that breaks the `-Z` nightly build):
+   ```bash
+   env -u RUSTUP_TOOLCHAIN cargo xtask starry app board \
+     -t sysbench-board -b OrangePi-5-Plus --server localhost --port 2999
+   ```
+   Success sentinel: `SYSBENCH_BOARD_DONE`. Capture serial to a file and diff the
+   `CPU_THREADS=` / `THREADS_T4` / `MUTEX_T4` / `MEMORY_T4` lines against
+   `linux-baseline.out`.
+
+## Gating risk — multi-core StarryOS boot
+
+`max_cpu_num = 4` is set so sysbench can scale, but multi-core StarryOS boot on
+this board is unproven (the scheduler effort's smp8 boot hang; the board also
+browns out at 8 cores on the current PSU). If StarryOS hangs at boot with smp4:
+drop `max_cpu_num` to 1 in the build config for a functional single-core run and
+rely on the Linux-side scaling curve, **or** treat bringing smp4 up cleanly as
+part of the scheduler work — a clean smp4 sysbench scaling curve here is a strong
+result for that effort.
+
+Sharpest scheduler signals once multi-core boots: the `cpu` 1→4 scaling curve and
+the sync-heavy `threads` / `mutex` subtests, ideally with cluster affinity
+(A76 vs A55) as a follow-up.
+
+## Files
+
+| file | purpose |
+|------|---------|
+| `build-aarch64-unknown-none-softfloat.toml` | board StarryOS kernel (rockchip SoC/SD/eMMC, `max_cpu_num=4`) |
+| `init.sh` | StarryOS-side workload → `SYSBENCH_BOARD_DONE` |
+| `board-orangepi-5-plus.toml` | ostool board config (sentinel, fail regexes, timeout) |
+| `build-static-sysbench.sh` | host: build a static aarch64 sysbench (deploy option A) |
+| `deploy-sysbench.sh` | run on board Linux: `apt install sysbench` + `sync` (deploy option B) |
+| `linux-baseline.sh` | run on board Linux: native baseline for the same subtests |

apps/starry/sysbench-board/RESULTS.md

@@ -0,0 +1,105 @@
+# sysbench: StarryOS vs Linux on OrangePi-5-Plus (RK3588)
+
+Same `sysbench 1.0.20` glibc binary, same board, same ext4 rootfs. Native Linux
+(Orange Pi Jammy, `6.1.43-rockchip`) vs StarryOS booted via serial FIT loady.
+
+- Date: 2026-07-15
+- StarryOS: upstream `rcore-os/tgoskits` dev @ `dbbe0e065`, `max_cpu_num` 4 and 8
+- Raw logs: `tmp/board-results/` (linux-baseline.out, starry-smp4-final.log,
+  starry-boot-upstream-smp8.log, starry-smp8-cpuinfo2.log)
+
+## CPU (events/sec, higher = better)
+
+| threads | StarryOS smp4 | StarryOS smp8 | Linux | Linux / StarryOS |
+|--------:|--------------:|--------------:|------:|-----------------:|
+| 1 | 159.6 | 159.0 | 981 | ~6.2× |
+| 2 | 160.1 | 159.9 | 1949 | ~12× |
+| 4 | 160.5 | 160.4 | 3904 | ~24× |
+| 8 | — | — | 5333 | (Linux uses all 8) |
+
+Scaling 1→4: **StarryOS 1.01× (flat)** vs **Linux 3.98× (near-linear)**.
+
+**smp8 == smp4, exactly.** Doubling `max_cpu_num` from 4 to 8 (all cores up,
+`nproc=8`, no PSU brown-out) produced identical numbers — because with no load
+balancer, every thread stays on the boot core. mutex-4 = 7.44s and memory-4 =
+41.9 MiB/s were also identical at smp4 and smp8.
+
+## Other subtests
+
+| test | StarryOS smp4 | Linux |
+|------|--------------:|------:|
+| threads-4 (events) | 1088 | 49358 |
+| mutex-4 (total time, lower=better) | 7.45 s | 0.46 s |
+| memory-4 (MiB/s, higher=better) | 42 | 8333 |
+
+## Findings
+
+1. **smp4 boots cleanly on upstream.** An earlier `max_cpu_num=4` hang at the
+   post-mount fs sync was a *stale-base artifact* — reproduced only on the
+   `oscomp-posad` fork base (168 commits behind upstream, missing the axtask
+   SMP-wake #1495/#1426 and block-IRQ #1512 fixes). On upstream dev it boots,
+   mounts the ext4 root, and reaches the shell with `nproc=4`.
+
+2. **The load balancer is THE bottleneck — smp8 proves it.** smp4 and smp8 give
+   *identical* numbers. Bringing up all 8 cores (incl. the A76 big cores) buys
+   nothing, because with no load-balancing / work-stealing every sysbench thread
+   stays on the boot core. Flat 1→4 scaling + zero smp4→smp8 delta both point at
+   the same missing axtask load balancer. More cores ≠ more throughput until it
+   exists.
+
+3. **Compute is stuck on the A55 boot core.** Single-thread ~159 ev/s is
+   A55-class (the RK3588 boot core is a little core); Linux runs the same binary
+   on an A76 @ 2.4GHz (~981) → the ~6× single-thread gap. StarryOS's `/proc/cpuinfo`
+   reports a uniform `0xd05` (A55) for all cores — a reporting stub, not literal —
+   but the performance confirms work never reaches the A76 cores. This is the
+   big.LITTLE placement gap (cf. the "L1 pin-to-A76 = 3×" finding).
+
+4. **Memory bandwidth is pathologically low** (42 vs 8333 MiB/s) — beyond what
+   A55 + single-core explains; worth a separate look (allocation / fault-per-block
+   path).
+
+## MEASURED decomposition (harness, 2026-07-16) — supersedes the inferences above
+
+The harness (`harness/`, see HARNESS.md) measured directly what was previously
+inferred. Same board; StarryOS smp8 @ upstream. Linux reference: A55 0.199 ev/s/MHz
+(max 359 @ 1.8 GHz), A76 0.430 ev/s/MHz (max 979 @ 2.256 GHz).
+
+Per-core pinned sysbench on **StarryOS** (affinity `landed`==`req` on all 8 cores):
+
+| StarryOS core | type | sb ev/s | ⇒ frequency |
+|--|--|--:|--|
+| cpu0–3 | A55 | ~160 | ~800 MHz (≈ Linux A55@816 = 161) |
+| cpu4–7 | A76 | ~351 | ~816 MHz (≈ Linux A76@816 = 347) |
+
+**Confirmed, not inferred:**
+1. **All 8 cores run at ~800–816 MHz** — no DVFS (every core matches the Linux
+   @816 MHz point). The boot core is an A55.
+2. **Affinity WORKS and the A76 cores are reachable** — `taskset -c 4..7` lands on
+   an A76 (`landed`==`req`) and runs **2.2× faster** (351 vs 160). *This corrects the
+   earlier worry that affinity might be ignored.* The default scheduler just never
+   migrates work there.
+3. **Per-core compute is at parity** on BOTH clusters (A55 160≈161, A76 351≈347 at
+   the same clock) — no StarryOS execution penalty.
+4. **The 200× memory result is NOT broken memory.** `membw` warm bandwidth is
+   7.3 GB/s (A76) / 3.2 GB/s (A55); sysbench memory at **1M blocks = 2.9 GB/s**. The
+   42 MiB/s only appears at sysbench's default **1K blocks**, and `firsttouch` is
+   0.8–1.3 s on StarryOS vs 0.086 s on Linux ⇒ the cost is a **slow page-fault /
+   first-touch path** + per-small-op overhead, not DRAM bandwidth.
+
+**Decomposition (measured):** `159 (A55@~800MHz) × 2.25 (DVFS) × 2.72 (A55→A76 at
+max) × 4.0 (1→4 cores) ≈ 24×`. Three independent, addressable levers:
+- **DVFS / cpufreq** — cores stuck at the ~800 MHz U-Boot OPP.
+- **big.LITTLE placement** — affinity works, but nothing auto-migrates to A76.
+- **load balancing** — nothing auto-spreads across cores (unpinned stays on 1 A55).
+
+(cpuprobe's direct MIDR/PMCCNTR reads returned `midr_ok=0`/`pmc_ok=0` — StarryOS
+doesn't expose those at EL0 — so core-type/frequency came from the Linux MIDR map +
+the frequency curves rather than a direct register read; the cross-checks agree.)
+
+## Why this matters
+
+sysbench cleanly *quantifies* exactly the two levers the kernel-half optimization
+effort targets: **big.LITTLE placement** (get work onto A76) and the **load
+balancer** (distribute threads across cores). Combined, StarryOS is ~24× behind
+Linux at 4 threads today; both gaps are addressable and independently measurable
+with this same workload — a ready-made regression/ää progress metric.