Skip to content

feat(perf): event groups, PERF_SAMPLE_READ, group-leader sampling, and perf record -a fan-out#1603

Open
JosephJoshua wants to merge 44 commits into
rcore-os:devfrom
oscomp-posad:feat/perf-group-sampling
Open

feat(perf): event groups, PERF_SAMPLE_READ, group-leader sampling, and perf record -a fan-out#1603
JosephJoshua wants to merge 44 commits into
rcore-os:devfrom
oscomp-posad:feat/perf-group-sampling

Conversation

@JosephJoshua

Copy link
Copy Markdown
Contributor

本 PR 为 StarryOS 硬件 PMU perf 系列的第三个主题栈:事件分组 + 采样读取(group + sample-read)。在 #1601(软件事件 + HW_CACHE)、#1602(真实 tid + PERF_RECORD_LOST)之上,补齐 perf stat -e '{a,b}' 分组与 perf record 的按事件读取能力。

本 PR 的净增改动(栈顶 3 个提交)

  1. 事件分组与 PERF_FORMAT_GROUP 读取event groups)——在 PerfEvent 文件封装层实现分组:group_fd 解析 leader、成员弱引用登记、leader→成员的 enable/disable 传播(Linux group scheduling),以及 leader 读取时一次性返回整组计数的 {nr, [te], [tr], {value,[id],[lost]}[nr]} 布局。类型无关,硬件/软件事件通用。
  2. PERF_SAMPLE_READ——每条 PERF_RECORD_SAMPLE 携带事件的运行计数(PERIOD 之后的 read 块);溢出处理程序按 period 递推 per-slot 的 read_value。范围:单事件,read_format ∈ {0, PERF_FORMAT_ID}
  3. 组长采样(PERF_SAMPLE_READ | PERF_FORMAT_GROUP——per-task 采样 leader 溢出时,在硬中断处理程序内无分配地输出整组计数(Linux nr, {value,[id]}[nr] 布局)。硬中断无法遍历进程上下文的分组链表,因此 arm_slice 在每个调度片把每个计数成员的 &accumulated/&slot/&last_cpu/&running 原子指针 + id 预置进 leader 的 SampleSlot;溢出时 leader 读取每个成员 = accumulated + (本核持有计数器时的 counter::read(slot)),同核判据与 read_values 一致——被轮转下去/降级的成员回退为 accumulated(Linux "last value" 语义,用户态用 time_running 缩放),因此无需全有全无的分组协同调度。同时修正了 per-task PERF_SAMPLE_READ 计数跨调度片不单调的隐性缺陷(per-slice 重建 slot 会清零 read_value;改为持久化到 PerTaskCounter 并由处理程序经 slot sink 回写,system-wide slot 常驻故 sink 为空)。

范围(v1):组长采样仅 per-task(system-wide GROUP 在 open 处拒绝)、read_format 仅 GROUP[|ID]TOTAL_TIME_* 拒绝);成员数以 MAX_GROUP_MEMBERS 为界。延后项(v1 正确性不需要):分组协同调度 + 轮转豁免(成员始终在线 vs last-value)、system-wide 分组采样、TOTAL_TIME_*

验证

  • QEMU:新增 perf-hw-group-sample(worker 线程自监控 {cycles-sampler, cycles-counter})→ 455 条样本,每条 nr==2、leader/member id 与 PERF_EVENT_IOC_ID 一致且互异、leader 值严格递增且 == 455×period、成员活值非零(硬中断内读到 107M cycles)。perf-hw-group(软件组)验证分组 read 布局。
  • cargo fmt --all -- --check 通过;cargo xtask clippy --package starry-kernel 容器内 20/20 通过。

⚠️ 关于 diff 规模(栈式 PR)

本 PR 依赖尚未合入 dev#1577(多核 perf 基座)与 callchain 栈,且叠在 #1601/#1602 之上,因此 diff 包含整条未合入的栈。本 PR 的净增改动即栈顶 3 个提交57b5b5a81(事件分组)、a5a0b641f(PERF_SAMPLE_READ)、3844e0860(组长采样)。待底层栈合入 dev 后将 rebase 收敛为干净 diff。建议按提交逐条审阅这三个。

Replace the single global counter allocator with a per-CPU pool
(PMEVCNTRn_EL0 is banked per-PE), reserve a slot per scheduling slice
in perf_sched_in / release it in perf_sched_out instead of pinning one
for the counter's life, and fix the cross-core read_values bug (only add
the live slice when the target runs on this core). free_hw tears the live
slice down on the owning core, via a synchronous IPI when the fd is closed
from a remote core, so it never stomps another task's counter.
A system-wide counting event pinned to a cpu (pid<=0, cpu>=0) now counts on
THAT core via its per-CPU pool, programmed/started/stopped/read/freed on the
target core through a synchronous IPI (local fast-path when it is the current
core). DISABLE stops the counter but keeps it allocated so read(perf_fd)
returns the final count; the slot is freed only at close. ensure_pmu_irq_registered
now brings up PMCR.E on every core before arming a PMU overflow.
A perf subsystem can register a fn(bool) via set_perf_tick that is invoked
from each periodic scheduler tick (timer-IRQ context). Null (zero address)
when unused, so axtask carries no hard dependency on perf. Mirrors the
existing set_run_on_cpu_sync registration hook.
When a task has more enabled counting events than its core has programmable
counters, rotate which subset holds the hardware counters on each periodic
tick so every event takes a turn. Splits time_enabled (whole on-CPU period)
from time_running (slot-hold sub-slice) so userspace scales value *
enabled/running. The tick is essential: a CPU-bound task that never preempts
still gets timer ticks, which a context switch alone would not provide.
…t fix

Events carry a ClusterMask (from the PMU type they were opened against);
generic/architectural events run on all clusters, an event opened against a
cluster PMU is restricted to it. A per-task event skips arming on a
non-matching core (time_enabled accrues, time_running does not, so perf
scales), mirroring Linux's pmu->filter; a cpu-bound system event on a
non-matching core is rejected with ENOENT. Fix BRANCH_INSTRUCTIONS to resolve
per-cluster via PMCEID (PC_WRITE_RETIRED 0x0C if implemented else BR_RETIRED
0x21), matching upstream (the repo diverged on A55). Cluster identity comes
from each core's MIDR_EL1, with a parity test override for homogeneous QEMU.
Expose per-cluster PMU devices (type 9 = A55, type 10 = A76), each with its
own cpus mask built from per-core MIDR. The generic armv8_pmuv3_0 (type 8)
stays as an all-clusters alias. Add the /proc/sys/kernel/perf_test_force_clusters
knob (parity override) so the cluster logic is exercisable on a homogeneous
machine.
Address review findings: (1) perf_sched_in opens a NEW time_enabled slice only on
the on-CPU transition, so on_exec re-entry no longer clobbers last_in_ns into
time_running > time_enabled; (2) on_task_exit snapshots the counter list and
drops the IRQ-off perf_counters lock before free_hw, which may issue a
cross-core IPI and dealloc pages; (3) read_values bases the live time_running
slice on run_since_ns (not last_in_ns) so a rotation-admitted slice cannot
over-report running; (4) rotation sizes its window against actually-free
counters (held + free pool), not raw PMCR.N, so a sys_cpu/sampling reservation
on the core no longer starves a rotatable event.
The override is a kernel-global AtomicBool that survives process exit and would
otherwise leak parity cluster classification into later suite cases.
Review finding: a pid<=0 && cpu<0 event allocates its counter on the opening
core, but enable/disable/reset/read/Drop ran on whatever core was current. A
migratable monitoring thread therefore freed the slot in the wrong per-CPU pool
and stomped another core's banked PMEVCNTRn (UAF in the sampling sub-case, where
the home core's REGISTRY slot kept dangling notify/ring pointers).

Record home_cpu at open and route the HW lifecycle (cycle counter, programmable
counter, and sampling arm/teardown) to it via run_hw_on_home — a synchronous IPI
when the caller has migrated, else a direct call. For sampling, the home core's
REGISTRY slot is unregistered before the ring/notify Arcs drop, closing the UAF.
Add perf-validate: one self-contained C binary that auto-discovers RK3588
topology and validates the SMP per-CPU + big.LITTLE hardware-PMU perf work on a
real Orange Pi 5 Plus. It prints PASS/FAIL/SKIP/INFO per check across topology,
sysfs, capacity, cluster-aware programming, BRANCH 0x0C/0x21 divergence, SMP
per-CPU, and counting/sampling/rdpmc fidelity, then a verdict
(FULL/PARTIAL/FAIL/INVALID) and a unique BOARD_PERF_VALIDATE_DONE sentinel.

It degrades gracefully: today the board boots only at max_cpu_num=1 (an smp8
late-boot hang, a non-perf bug), so needs-smp8 / needs-both-clusters checks SKIP
and the verdict is PARTIAL (single-core regression anchor passed; big.LITTLE
unvalidated). When smp8 boots, the verdict becomes FULL.

Two run modes, auto-detected from cpu0 MIDR: board mode (real A55 0xD05 / A76
0xD0B) runs the full silicon matrix; selftest mode (anything else, e.g. QEMU
cortex-a53) enables the parity override and exercises the cluster/pool LOGIC +
counting/sampling/rdpmc, with silicon-only rows self-SKIPping. An anti-false-
confidence verdict gate downgrades a green run to INVALID if the override is
stuck on or a core was not warmed before its cluster assertion.

Layout:
- board-orangepi-5-plus/perf-validate/: canonical src + board run toml + a
  decoupled smp1 build wrapper (log=Warn) + a staged (non-discovered) smp8 build
  + deploy.sh (static-musl cross-compile + scp) + README.
- qemu-smp4/system/perf-validate/: byte-identical copy + CMakeLists, a permanent
  QEMU regression (auto selftest, exits 0). Verified: SELFTEST-OK, 18 pass /
  0 fail / 22 skip / 6 info, STARRY_GROUPED_TESTS_PASSED; -Werror clean.
The perf ring / rdpmc / mmap-page frames are RAM the kernel writes through its
cacheable linear map and userspace reads back, but PerfEvent::device_mmap wrapped
them in DeviceMmap::Physical, which mmap maps UNCACHED (Normal-NonCacheable). On
real silicon the kernel's cached writes never reach RAM, so userspace's uncached
reads return stale zeros: sampling delivered 0 samples (ring data_head read 0)
and the rdpmc mmap page read cap/index/width = 0. QEMU models no caches, so it
never surfaced this — only the on-board run did.

Map these pages cacheable instead (DeviceMmap::PhysicalCached). Kernel and user
then hold Normal Inner-Shareable Write-Back mappings of the same physical page,
which the hardware keeps coherent across the inner-shareable domain (both RK3588
clusters) with no explicit cache maintenance. The sample-ring writer already
issues a Release fence before publishing data_head, and the reader pairs it with
an acquire barrier, so the weaker ordering of Normal memory is handled correctly.

PhysicalCached was feature-gated to rknpu; ungate it (it is a general
cacheable-RAM mapping — the caller decides whether DMA maintenance is needed:
rknpu yes, perf no since it is CPU-to-CPU coherent) and update its doc. Builds
clean on aarch64 + riscv64; clippy/fmt clean; QEMU perf-hw sample/sample-task/
rdpmc/record-ioctl/ring-merge/freq/sideband/fork-exit/inherit + perf-validate
selftest all still pass (no regression). Board re-run pending to confirm the fix.
perf_event_mmap_page.capabilities places cap_user_rdpmc at bit 2 (Linux ABI:
after cap_bit0 and cap_bit0_is_deprecated). The test struct omitted
cap_bit0_is_deprecated, so cap_user_rdpmc landed at bit 1 and read the kernel's
`1<<2` as 0 — RDPMC-1 spuriously FAILed on the board (cap=0) even though the
page was correct (index/width/rdpmc value all right after the cacheable-mmap
kernel fix). Add the missing bit; also assert cap in RDPMC-2 (the cycle-counter
event opens under QEMU, so this validates the bitfield there). QEMU selftest:
RDPMC-2 cap=1, SELFTEST-OK 18 pass/0 fail.
…on hardware)

Corrected against the actual Orange Pi 5 Plus run: deploy to /usr/local/bin
(the SD ext4 StarryOS mounts as /; /root is 700-root, not orangepi-writable),
stage-to-/tmp + sudo-install in deploy.sh, the real board-test command
(cargo xtask starry test board -c board-orangepi-5-plus/perf-validate ...), and
the NIC caveats (cabled port drift enP3p49s0/enP4p65s0; en5 re-arm after each
reboot). Ignore the built binary artifact.
…facts

Booted the board fully multi-core (online=8, clusters=4+4) with a minimal
feature set and ran the whole SMP + big.LITTLE matrix on real 4xA55+4xA76:
38 pass incl. cross-cluster ENOENT (CLU-1/2), cluster-skip (CLU-4: Big counted,
Little ~50% skipped), cross-cluster migration, per-cpu fan-out, home-core IPI,
per-secondary PMU init + sampling (CAP-4/SAMP-2), rotation, 12-thread spread.
The whole multi-core perf path works on silicon.

Root cause of the smp8 boot hang (documented in smp8-staged-build-aarch64.toml):
it is NOT the block device — a secondary A76 (cpu4) coming online triggers a USB
IRQ storm (rockchip-dwc-xhci IRQ 253 taken in a loop), same class as the NPU
secondary-core init hang. Neither is needed for perf, so the staged smp8 build
now drops USB/NPU/PCIe/net, keeping only SoC + SD/block (which polls).

Two board fails were test artifacts, fixed:
- BR-7: pin the opener to the A55 before opening/​running the raw 0x0C event —
  the kernel resolves event support on the opening core, and the cpu-bound
  counters need the branchy workload to run on the A55. On 8 cores an unpinned
  opener/workload lands on an A76 (no 0x0C) and reads 0.
- IPC-1: use an ILP-rich register-only loop and REPORT (INFO, not FAIL) — IPC is
  workload-dependent; on a memory-serialized loop the in-order A55 legitimately
  matches/beats the A76, so a hard "A76 IPC > A55" assert is wrong.

QEMU selftest still SELFTEST-OK (both checks skip on homogeneous QEMU); -Werror
clean. The committed default wrapper stays smp1 (PARTIAL); smp8 = staged config.
Upstream merged the standalone qemu-smp4 harness into the main qemu/ system
group (now max_cpu_num=4), which already hosts the per-task perf-hw-* tests and
the SMP affinity/clone/fcntl cases. The multicore perf-hw-smp-* and perf-validate
subcases were added under the now-removed qemu-smp4/system/, so move them into
qemu/system/ where the group CMakeLists globs them and they run at smp4.
The smp8 full-matrix run (ymodem FIT transfer + the long SMP-ROTATE /
SMP-MIGRATE / RDPMC-3 loops on real silicon) can exceed the 600s smp1
anchor; give it headroom so a slow-but-healthy run is not cut off before
BOARD_PERF_VALIDATE_VERDICT.
The sampling ring dropped records silently when full, so perf record printed
"read LOST count failed": read(perf_fd) with PERF_FORMAT_LOST (set by perf's
record__read_lost_samples) returned a short buffer because that read_format bit
was unhandled.

Account dropped samples per event. ring_write now reports whether it wrote; the
overflow handler bumps the owning event's lost counter on a drop (via a raw
pointer in SampleSlot, mirroring notify). read() appends the u64 lost field when
PERF_FORMAT_LOST is set. The counter lives on SamplingState (self/cpu path) and
PerTaskCounter (per-task path), each reached by read_values.

Verified on StarryOS QEMU: perf record no longer errors and writes a valid
perf.data (Total Lost Samples: 0).
Upstream linux-6.1 perf, statically linked for aarch64-musl, runs on StarryOS
via the Linux-compatible perf_event_open(2) ABI: perf stat counts cycles, perf
record writes a valid perf.data, and perf report symbolizes kernel functions via
/proc/kallsyms. build-perf.sh pins the reproducible build; the smoke case stages
the binary and exercises stat/record/report.

The 2.8 MB binary is gitignored and produced by build-perf.sh; when it is absent
(CI, or before a local build) the case installs nothing and is skipped so the
qemu/system group still builds.
build-perf-libelf.sh + musl-compat.h build a static aarch64-musl perf WITH
libelf natively on Alpine, so perf report resolves user symbols to names
(ld-musl-aarch64.so.1 [.] _dlstart) instead of raw offsets; kernel symbols
already resolve via /proc/kallsyms. Documents the toolchain + the six
musl/Alpine build fixes. The smoke script gains per-step markers to localize
regressions. The 2.8MB binary stays gitignored (built via the recipe).
The perf-validate selftest reads ARM PMU sysregs via raw inline asm
(mrs pmccntr_el0 / pmevcntrN_el0), which the loongarch64 (and other
non-aarch64) C compilers reject with "unrecognized instruction
mnemonic". Route the qemu/system case through the shared
starry_arch_filtered_executable helper so non-aarch64 targets build the
skip stub instead, matching test-user-backtrace and the other
arch-specific cases.
The six per-CPU / big.LITTLE perf SMP selftests are hardware-PMU (ARM
PMUv3) tests; on loongarch64/riscv64/x86_64 perf_event_open on the RAW
0x11 event has no ARM PMU to bind, so they exit non-zero and fail the
grouped-C suite. Every other perf-hw-* case already skips-as-pass off
aarch64; add the same guard here (build+run everywhere, print the OK
sentinel and return 0 on non-aarch64). The earlier perf-validate
arch-gate fix unmasked these at runtime (the build previously stopped at
perf-validate before reaching them). Verified they compile clean for
loongarch64 with -Wall -Wextra -Werror.
The PMU overflow handler runs inside dispatch_irq and can read the
interrupted PC/EL live (ELR_EL1/SPSR_EL1), but the interrupted frame
pointer (x29) survives only in the saved TrapFrame — the handler's own
frames have already clobbered the GPR. Call-graph sampling
(PERF_SAMPLE_CALLCHAIN) needs that x29.

Publish a per-CPU *const TrapFrame at the two IRQ-entry sites (EL1
kernel interrupt in trap.rs, EL0 user interrupt in UserContext::run)
immediately before dispatch_irq and clear it immediately after, both
inside the IRQ-masked window so no nested IRQ observes a stale frame.
Add interrupted_fp()/interrupted_sp() accessors alongside the existing
interrupted_pc()/interrupted_is_user(). The publish/clear is a single
per-CPU store each (TPIDR-relative, cannot fault) and does not reorder
the register save/restore path.

Pulls ax-percpu into the aarch64 dependency set (already present for
x86_64) for the per-CPU cell.
…e getters

Add walk_fp(pc, fp, ip_range, fp_range, read, out) -> usize: an
allocation-free frame-pointer walk into a caller-supplied buffer that
reads memory through an injected closure instead of dereferencing
directly, so a hard-IRQ caller (PMU-sampling call-graph capture) can
supply a fault-safe reader. Mirrors unwind_core's guards (monotonic fp,
8 MiB jump cap, word alignment, depth cap via out.len()) but never
allocates and is not behind the alloc feature, so it is available in the
default (no-alloc) kernel build.

Add ip_range()/fp_range() getters over the ranges init() records, each a
lock-free spin::Once load safe to call from IRQ context, so the sampling
unwinder can reuse the kernel text / address-space bounds.

Serialize the depth-sensitive tests: set_max_depth mutates a process
global, so init_for_tests now returns a guard that holds a shared lock
for the test body, removing a latent flake under cargo test's parallel
harness (surfaced by the new walk_fp tests raising thread contention).
Emit PERF_SAMPLE_CALLCHAIN so perf record -g captures per-sample kernel
stack backtraces (frame-pointer), rendering as flamegraphs off-target.

- perf/unwind.rs: kernel-side integration over axbacktrace::walk_fp —
  kernel_ranges() reuses the boot-time backtrace ranges, a direct-deref
  reader for always-mapped kernel frames, and kernel_callchain() that
  walks the interrupted x29 chain into a fixed buffer.
- sampling.rs: define PERF_SAMPLE_CALLCHAIN / PERF_CONTEXT_KERNEL /
  PERF_CONTEXT_USER, add CALLCHAIN to SUPPORTED_SAMPLE_TYPE (both hw.rs
  open sites accept it via the mask), grow SAMPLE_RECORD_MAX_LEN to bound
  the callchain block, emit 'u64 nr + entries' in build_sample in Linux
  field order, and build the chain in the overflow handler from the
  interrupted frame pointer (leaf-only fallback when none is published,
  so a sample is never dropped).

Kernel frames only in this pass; the user region is a leaf IP with a
PERF_CONTEXT_USER marker (the no-fault user FP unwind is M4b). Sampling
stays aarch64-only and allocation-free in IRQ context.
…d bound

Follow-up hardening from adversarial review of the PMU-sampling callers:

- Bound TOTAL iterations (a small multiple of out.len()), not just recorded
  frames. A return address outside ip_range is skipped without consuming a
  recorded slot, so the depth cap alone did not bound the loop; a crafted
  all-skip user chain (every [fp+8] out of range, monotonic small-gap [fp])
  could otherwise spin the walk — each step doing real work (two no-fault
  page-table walks) — into an IRQ-latency DoS. Apply the 8 MiB-gap check on
  every fp advance (both the skip and record paths), not only after recording.
- Bound the whole frame record [fp, fp+16) inside fp_range (not just fp), so
  the saved-LR read at fp+8 can never touch one word past fp_range.end.

Adds a test that would hang without the step cap (an ever-advancing all-skip
chain) and confirms it terminates recording only the leaf.
Add user-space callchains (PERF_SAMPLE_CALLCHAIN user region) and make the
whole FP unwind fault-proof from hard-IRQ context.

- perf/nofault.rs: read_user_word_nofault / read_kernel_word_nofault walk
  TTBR0 / TTBR1 by hand (4-level, 48-bit VA) against the always-mapped direct
  map, never dereferencing the input VA. Every table frame and the resolved
  leaf is bound-checked against phys_ram_ranges() before deref, so a valid PTE
  whose output is device MMIO (a user mmap of an RGA/NPU/JPU register window,
  or a concurrently-torn table on another CPU) can never trigger a live MMIO
  read (device side effect) or a data abort (panic) in the overflow handler.
- perf/unwind.rs: route the kernel reader through the no-fault path too, so a
  corrupt/stale in-kernel frame pointer that slips past the range guards yields
  None instead of faulting; add user_callchain, bounding the walk to a window
  above the interrupted SP_EL0.
- perf/sampling.rs: build_callchain now unwinds the user region for user
  samples (deep [PERF_CONTEXT_USER, ip, ra…]) via the no-fault reader, seeded
  from the interrupted x29 + SP.

Kernel deep frames still require an FP-enabled kernel (-Cforce-frame-pointers);
user frames unwind whenever the sampled binary keeps frame pointers.
Two grouped-C qemu/system cases exercising PERF_SAMPLE_CALLCHAIN end to end
(open sampling event with IP|TID|TIME|CALLCHAIN, mmap ring, parse the callchain
block, classify by PERF_CONTEXT_* markers):

- perf-hw-callchain-kernel: asserts a well-formed callchain record with a
  PERF_CONTEXT_KERNEL region + leaf IP. CI-safe: does NOT require a deep kernel
  chain, since the default kernel is built without frame pointers (like Linux
  without CONFIG_FRAME_POINTER), so the kernel region is leaf-only.
- perf-hw-callchain-user: builds a known nested chain outer->mid->inner->busy
  with -fno-omit-frame-pointer and asserts >= 4 user IPs after PERF_CONTEXT_USER
  — proving the kernel unwinds several USER frames via the no-fault TTBR0 reader.
  Uses a read(/dev/zero) workload: QEMU-TCG's cycle counter barely advances on a
  pure-ALU loop, so the sampling counter would seldom overflow.

Both skip-as-pass off aarch64. Validated in QEMU (smp4): kernel record
well-formed; user unwinds 8 frames deep across repeated runs.
PERF_RECORD_SAMPLE carried the axtask scheduler id as both pid and tid, while
the COMM/MMAP2/FORK/EXIT side-band records already carry the real userspace
tgid/tid. The mismatch broke `perf report`'s sample -> process/DSO-map join,
misattributing samples across a multithreaded process.

- sampling.rs: derive the sample (pid, tid) from the real userspace ids. For a
  per-task event use the owner (tgid, tid) captured at slice-arm time into the
  SampleSlot: the PMU overflow IRQ can be serviced after a context switch away
  from the monitored task, so `current()` in the handler would misattribute the
  sample; the captured owner is always the event's task. For a system-wide event
  (owner_ids == None) attribute to the interrupted `current()`, which matches the
  sampled IP -- reading proc.pid()/thr.tid() lock-free from IRQ context (task_ext
  downcast + field read + atomic load), falling back to the scheduler id for a
  kernel task with no Thread.
- task.rs: arm_slice stamps the monitored thread's (tgid, tid) into the per-task
  SampleSlot; both callers (perf_sched_in, perf_rotate_current) hold the Thread.
- hw.rs: the system-wide slot carries owner_ids = None.

Single-threaded `perf record ./cmd` already joined by accident (clone derives the
userspace tid from the axtask id, and the leader tid == getpid()); this fixes the
multithreaded case where a sample's pid must be the shared tgid, not the
per-thread tid.

Adds perf-hw-sample-tid: a non-leader worker thread opens a per-task sampling
event on its own tid and asserts every PERF_RECORD_SAMPLE reports pid == process
tgid and tid == worker gettid(). Validated in QEMU (smp4): 1024/1024 samples
correct, 0 misattributed. Skips-as-pass off aarch64.
`perf stat -- cmd` opens its default set with no `-e`: hardware cycles/
instructions plus five PERF_TYPE_SOFTWARE events — cpu-clock, task-clock,
context-switches, cpu-migrations, page-faults. All five dispatched to the BPF
stub (no readable count), so read(perf_fd) returned Unsupported and every default
row printed `<not counted>`. Make them real per-task counters so a bare
`perf stat -- cmd` looks correct.

- perf/sw.rs (new): a lightweight per-task SwPerTaskCounter + SwPerfEvent
  (PerfEventOps). Pure atomics, no PMU, so it is arch-independent. cpu-clock is
  wall time enabled; task-clock is on-CPU time; context-switches/cpu-migrations/
  page-faults are event counts. A global PERF_SW_ACTIVE gate makes every hook a
  single relaxed load when no software event exists (mirrors PERF_TASK_ACTIVE).
- perf/mod.rs: route the five counting software configs to sw::perf_event_open_sw
  (pid > 0 a specific tid, pid == 0 the caller); every other software config
  (e.g. PERF_COUNT_SW_DUMMY, perf record's tracking event) keeps the BPF path.
- task/mod.rs: drive the counters from the existing scheduler scope hooks
  (on_enter -> sw::sched_in opens the task-clock slice + counts cpu-migrations;
  on_leave -> sw::sched_out folds it + counts context-switches), added alongside
  the hardware perf_sched_in/out without touching their logic. Thread gains a
  perf_sw_counters list.
- task/user.rs + mm/access.rs: count page-faults from BOTH fault paths — the
  user-mode EL0 fault (ReturnReason::PageFault) and the kernel-mode fault on user
  memory — so the count matches Linux's all-faults semantics.

Adds perf-sw-counters: opens all five events, runs a workload that provably
generates each (busy loop; mmap + MADV_DONTNEED re-touch for guaranteed
demand-zero faults; nanosleep loop for deschedules), and asserts read(perf_fd)
returns a real count. Validated in QEMU (smp4): cpu-clock 247ms, task-clock 173ms
(< cpu-clock), page-faults 513, context-switches 24, cpu-migrations readable.
Skips-as-pass only if perf_event_open is unwired (ENOSYS).
`perf stat -e L1-dcache-load-misses,LLC-loads,dTLB-load-misses,...` opens
PERF_TYPE_HW_CACHE events whose config packs cache_id | (op<<8) | (result<<16).
These were rejected at open ("unsupported hardware type"); only the
PERF_TYPE_HARDWARE cache-references/misses aliases worked. Decode the matrix to
ARM PMUv3 event numbers.

- axcpu/aarch64/pmu.rs: hw_cache_to_arm(config) maps the well-defined
  combinations — L1D/L1I/LL cache + refill, D/I TLB + refill, branch pred +
  mispred — to their ARM PMUv3 event numbers. ARM's basic cache events do not
  split read vs write, so both map to the same event; PREFETCH ops and the NODE
  cache have no architectural counterpart and return None (rejected at open,
  matching Linux's CACHE_OP_UNSUPPORTED).
- perf/hw.rs: decode PERF_TYPE_HW_CACHE in both the per-task (decode_arm_event)
  and self/system-wide (perf_event_open_hw) paths via hw_cache_to_arm, then
  alloc_programmable — which already gates on event_supported, so a combination
  the core's PMU does not implement is cleanly reported unsupported.
- perf/mod.rs: route PERF_TYPE_HW_CACHE to the hardware path.

Adds perf-hw-cache: opens the common cache combinations and an unsupported L1D
PREFETCH. CI-safe: QEMU-TCG's PMU implements no cache events, so on QEMU every
mapped combination is cleanly reported unsupported (not a routing error) and the
PREFETCH is rejected — validated. Count accuracy is validated on the board (the
RK3588 A55/A76 PMUs implement these events). Skips-as-pass off aarch64.
When a sampling ring fills, the overflow handler drops samples and bumps the
event's lost counter for the read-only PERF_FORMAT_LOST total. Linux also writes
an in-band PERF_RECORD_LOST (type 2) record into the ring once it has room, so
`perf report`/`perf script` show "LOST n events!" and place the gap on the
timeline. Add it.

- sampling.rs: a companion per-event `lost_reported` counter tracks how many
  drops have been emitted in-band. Before writing each sample, the handler
  flushes `lost - lost_reported` as a PERF_RECORD_LOST record (header + event id
  + lost count) when the ring has room — it was full when the drops happened, so
  the record is retried on later samples once userspace drains the ring. Reuses
  the existing back-pressure in ring_write (a record is only written when it
  fits), so a still-full ring simply leaves the count pending.
- hw.rs / task.rs: carry the `lost_reported` AtomicU64 alongside the existing
  `lost` on both the system-wide (SamplingAnchors) and per-task (PerTaskCounter)
  events, and thread its pointer into the SampleSlot.

Adds perf-hw-lost: opens a sampling event with a one-page ring, overflows it with
a syscall-heavy workload, drains (advances data_tail) between bursts to free
room, and asserts a PERF_RECORD_LOST record with a non-zero lost count is
emitted. Validated in QEMU (smp4): 2033 samples, 15 lost records, 1318529 total
lost reported in-band. Skips-as-pass off aarch64.
`perf stat -e '{cycles,instructions}'` (and every -M metric/topdown group) opens
events as a GROUP — a leader plus members with group_fd set to the leader's fd —
enables the leader to schedule the whole group, and reads the leader once with
PERF_FORMAT_GROUP to get every member's counter in one buffer. Previously group_fd
was dropped and PERF_FORMAT_GROUP was undefined, so a leader read returned the flat
single-event layout: a malformed buffer to perf's group parser.

Implement it in the perf-event file wrapper (PerfEvent), above the per-type impls,
so it works uniformly for hardware and software events:

- Membership: a non-negative group_fd resolves the leader (via the fd table, like
  PERF_EVENT_IOC_SET_OUTPUT) and records the new event weakly on the leader; the
  member is opened disabled so it only counts once the leader starts the group.
- Enable/disable propagation: ioctl(ENABLE/DISABLE) on the leader propagates to
  all live members (Linux group scheduling — members follow the leader), reaping
  any that have closed.
- PERF_FORMAT_GROUP read: a leader read returns { u64 nr; time_enabled?;
  time_running?; { value; id?; lost?; }[nr] } (leader first, then live members),
  in Linux's field order.

Not yet handled: all-or-nothing co-scheduling for a group that oversubscribes the
PMU (rare for the common 2-event group, which fits hardware); members are
scheduled independently by the existing rotation. PERF_SAMPLE_READ (group-leader
sampling) is a follow-up that builds on this.

Adds perf-hw-group: opens a {task-clock,cpu-clock} group, enables ONLY the leader,
and asserts the leader's PERF_FORMAT_GROUP read returns nr==2 with ids matching
PERF_EVENT_IOC_ID and the member counted > 0 (so leader-enable started the group).
Software events are used because QEMU-TCG's PMU implements only CPU_CYCLES, not a
second hardware event; the group path is identical for hardware groups on silicon.
Validated in QEMU (smp4). Skips-as-pass when perf_event_open is unwired.
With PERF_SAMPLE_READ in sample_type, each PERF_RECORD_SAMPLE carries the event's
read value (the read_format block), so a sampling event reports its running count
at each sample. Previously the bit was rejected at open.

- sampling.rs: accept PERF_SAMPLE_READ; the overflow handler advances a per-slot
  running count by the sampling period each overflow (~period events per sample)
  and build_sample emits the read block ({ value; id? }) after PERIOD, in Linux's
  field order. SAMPLE_RECORD_MAX_LEN grows for the block.
- hw.rs / task.rs: carry read_format + the running read_value on the SampleSlot at
  both the system-wide and per-task registration sites.

Scope: a single event with read_format in {0, PERF_FORMAT_ID} (value, optionally
id). Group-leader sampling (`-e '{a,b}:S'`, read_format & PERF_FORMAT_GROUP) and
the TOTAL_TIME_* formats need per-group / per-event accounting reachable from the
hard-IRQ handler and are rejected at open (`sample_read_supported`) rather than
emit a malformed block — a follow-up that builds on the event-groups work.

Adds perf-hw-sample-read: a sampling event with sample_type = IP|TID|TIME|READ and
read_format 0, asserting each sample carries a non-zero, strictly-increasing read
value (the running count). Validated in QEMU (smp4): 819 samples, last value
81900000 == 819 * period, strictly increasing. Skips-as-pass off aarch64.
A per-task sampling leader opened with read_format = PERF_FORMAT_GROUP now
emits the whole group's counters in every PERF_RECORD_SAMPLE (Linux group
read layout: nr, then per-event value[+id]), assembled alloc-free from the
hard-IRQ overflow handler.

The handler cannot walk the process-context group list, so each counting
member is pre-staged into the leader's SampleSlot at slice-arm time as raw
pointers to its accumulated/slot/last_cpu/running atomics. At overflow the
leader reads each member's live count as accumulated + (banked PMU counter,
when the member holds one on this core), guarded exactly like
task::read_values -- a rotated-off/degraded member contributes its
accumulated value (Linux "last value" semantics, scaled in userspace by
time_running), so no all-or-nothing group co-scheduling is required.

Also fix per-task PERF_SAMPLE_READ to report a count monotonic across
scheduler slices: the per-slice SampleSlot is rebuilt each arm, so the
running count is now persisted on the PerTaskCounter and mirrored back by
the handler through a slot sink (system-wide slots persist in place and
leave the sink null).

Scope (v1): per-task path only; read_format GROUP optionally | ID
(TOTAL_TIME_* and system-wide GROUP rejected at open); members bounded by
MAX_GROUP_MEMBERS. SAMPLE_RECORD_MAX_LEN and the handler scratch grow to
reserve the group READ block.

Test perf-hw-group-sample opens {cycles-sampler, cycles-counter} on a
worker thread and asserts every sample carries nr=2, correct+distinct
leader/member ids (IOC_ID), a strictly-increasing non-zero leader value,
and a non-zero member value read live in the handler. QEMU: 455 samples,
all checks pass.
Copilot AI review requested due to automatic review settings July 14, 2026 07:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

A system-wide sampling event pinned to a specific cpu (perf_event_open with
pid=-1, cpu=i) now arms its programmable counter AND its overflow IRQ on
THAT core, so `perf record -a` captures per-core samples. Previously such an
event fell through to the opening-core path: all N fan-out events armed on
one core (colliding on its ~6 programmable counters) and every sample
carried the opener's cpu, so per-core profiles were wrong.

The event is built sampling-shaped (sampling=Some, sys_cpu=None so
device_mmap allocates its ring) with home_cpu=cpu, so the unchanged
enable/disable/read/Drop lifecycle already fans the arm + strict teardown
out to the target via the existing run_hw_on_home IPI path. Its
programmable counter is reserved on the TARGET core's per-CPU pool via the
counting fan-out's SYS_OP_PROGRAM op -- never the opener's, so it cannot
collide with the target's per-task events. The overflow handler + per-CPU
REGISTRY already stamp cpu=this_cpu_id() and attribute to the interrupted
current() on the target, so sample content is already core-correct.

To keep cross-core arming sound, split the PMU IRQ registration:
install_pmu_irq_handler() (the one-time request_percpu_irq, which takes a
lock + allocates) now runs at event open in process context, while the arm
paths -- hw_enable_local (run on the target inside the IPI hard-IRQ thunk)
and the IRQs-off per-task arm_slice -- call only the IRQ-safe
enable_pmu_irq_line() (ensure_core_inited + per-core PPI enable).

Scope (v1): per-CPU sample capture with real {IP,PID/TID,TIME,CPU}. Defers
kernel-emitted system-wide sideband (COMM/MMAP2 for all processes) --
userspace symbolizes pre-existing maps via /proc -- and multi-event -a
beyond a core's counter count (no system-wide multiplexing).

Test perf-hw-smp-record-allcpu (QEMU smp4): forks a read()-workload child
pinned to each of 4 cpus, opens one sampling event per cpu, and asserts
every ring gets samples all tagged with its own cpu. Result: 819 samples
per cpu, wrong_cpu=0 on all four.

@mai-team-app mai-team-app Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

审查总结

本 PR 为 StarryOS 硬件 PMU perf 系列的第三个主题栈,实现事件分组、PERF_SAMPLE_READ 和组长采样。代码架构清晰,硬中断处理程序无分配、安全注释完整,测试用例摆放正确(test-suit/starryos/qemu/system/perf-hw-group(-sample)/),整体质量较高。

审查发现

阻塞项:CI 失败 — orangepi-5-plus 板级测试

证据:

分析:

  • OrangePi 5 Plus 使用 RK3588(Cortex-A76 big + Cortex-A55 LITTLE),而本 PR 栈新增了大量 aarch64 PMU 集群感知代码(components/axcpu/src/aarch64/pmu.rsClusterId/classify_midr/cluster_idos/StarryOS/kernel/src/perf/percpu.rsClusterMask/big.LITTLE 分类、hw.rscluster_mask_for_type/ArmInterruptRouter 等),直接与真实 big.LITTLE 硬件交互。
  • 所有 QEMU 测试(aarch64/riscv64/x86_64/loongarch64)均通过,但 QEMU 不模拟 big.LITTLE(其 PMU 为单一 Cortex-A53),无法覆盖该路径。
  • 该失败在 base dev 上不存在(base dev 的同名 job 成功),属于本 PR 栈引入的回归。

修复方向: 请在 OrangePi 5 Plus(或等效 RK3588 开发板)上复现该故障,定位是集群分类、per-CPU 初始化、counter 分配还是 PMU IRQ 处理的问题,修复后重新触发 CI。在 CI 通过之前不应合并。

本地验证

检查项 结果
cargo fmt --check ✅ 通过
cargo clippy --manifest-path components/axbacktrace/Cargo.toml --all-features -- -D warnings ✅ 通过
cargo clippy --manifest-path components/axcpu/Cargo.toml --all-features -- -D warnings ✅ 通过
cargo clippy --manifest-path os/StarryOS/kernel/Cargo.toml --all-features -- -D warnings ✅ 通过
cargo clippy --manifest-path os/arceos/modules/axtask/Cargo.toml --all-features -- -D warnings ✅ 通过
cargo test --manifest-path components/axbacktrace/Cargo.toml --all-features ⚠️ 链接失败(缺少 __start_debug_str 等嵌入符号,需要 StarryOS 链接脚本;非 PR 回归)

CI 状态

  • 远程 QEMU 测试: starry aarch64/riscv64/x86_64/loongarch64、arceos 全架构、axvisor 全架构 → 全部 success 或已跳过(路径过滤)
  • clippy / formatting / lint: 全部 success
  • 自托管板级测试: visionfive2/aka-00-sg2002/phytiumpi/roc-rk3568-pc/asus-nuc15crh → success;orangepi-5-plus → failure(如上所述,阻塞)
  • 跳过项: 容器侧对应项为常规 CI 互斥/路径过滤/分支限制,非问题

重复/重叠分析

  • 本 PR (#1603) 依赖 #1577 → callchain → #1601#1602,为栈式 PR 的顶部。其他三个 PR 各有独立范围(多核基座、软件事件+HW_CACHE、tid+lost 记录),与本 PR 不重复。
  • 搜索 repo:rcore-os/tgoskits type:pr 未发现其他与本 PR 冲突或重叠的开放 PR。

其他审查说明

  • [patch.crates-io]:工作区 Cargo.toml 中未发现任何 crates.io patch。
  • 测试布局正确:新增测试 perf-hw-group/perf-hw-group-sample 位于 test-suit/starryos/qemu/system/<case>/,含 CMakeLists.txt 和 src/,符合 grouped C runner 要求。
  • 代码质量良好:IRQ 上下文安全性(unsafe 块均有 SAFETY 注释)、lock-free 设计、Linux 语义对齐均到位。diff 规模虽大(~12k 行),但包含整条未合入栈,净增部分(3 个提交)范围聚焦。
  • 以上均为非阻塞观察。

结论

请修复 orangepi-5-plus 板级 CI 失败后重新提交审查。

Powered by deepseek-v4-pro

@mai-team-app
mai-team-app Bot requested review from ZCShou and ZR233 July 14, 2026 08:22
…view fixes

Adversarial review of the group-sampling stack found a use-after-free: a
PERF_FORMAT_GROUP sampling leader bakes raw pointers to each counting
member's PerTaskCounter atomics into its per-CPU SampleSlot (whose lifetime
is bound to the leader's slice), but perf_event_open linked a member to a
leader by group_fd alone -- with no same-thread check. A member on a
different thread could be freed (its thread exits, fd closed) while the
leader's slot still held its pointers, so a PMU overflow on the leader's core
would dereference freed atomics from hard-IRQ context.

Fix: require a group member and its leader to monitor the SAME thread
(owner_tid on PerTaskCounter), rejecting a cross-thread link with EINVAL --
mirroring Linux's group_leader->ctx == event->ctx gate. A same-thread member
lives in the leader's own Thread's perf_counters list, whose slot is
unregistered at perf_sched_out before on_task_exit frees it, so the baked
pointers always outlive the registration.

Also from the review:
- link_group_member prunes dead member Weaks and rejects (rather than
  silently drops) a group exceeding MAX_GROUP_MEMBERS, keeping read() and
  sampled group reads consistent.
- PERF_SAMPLE_READ makes the per-task sink the single source of truth (read,
  add, write back) so an ioctl(RESET) that zeroed it takes effect on the next
  sample instead of being clobbered by the slot's stale copy.
- install_pmu_irq_handler adds an INSTALLED flag distinct from the REGISTERED
  CAS token, so a concurrent opener waits for the handler to actually be
  installed before returning Ok (never arms an overflow with no handler).
- SAMPLE_RECORD_MAX_LEN reserves the callchain's leading `nr` word (latent
  record overrun if a future callchain fills both regions).

Test perf-hw-group-crosstask: a helper thread's tid is used to attempt a
cross-thread member open (must fail EINVAL) while a same-thread member open
succeeds. QEMU: cross_errno=22 (EINVAL), same-thread OK; group-sample and
sample-read regressions still green.
@JosephJoshua JosephJoshua changed the title feat(perf): event groups, PERF_SAMPLE_READ, and group-leader sampling feat(perf): event groups, PERF_SAMPLE_READ, group-leader sampling, and perf record -a fan-out Jul 14, 2026
@JosephJoshua

Copy link
Copy Markdown
Contributor Author

更新:新增自审修复 + 板级 CI 失败根因分析

1. 本分支已更新(force-push 3844e0860cde643e38

在打包前对本栈最新的 4 个特性做了一轮多维对抗式代码审查(内存安全 / 硬中断·IPI 上下文 / ABI / 生命周期竞争 / 边界校验,每条发现由独立 skeptic agent 尝试证伪)。发现并修复了一个 critical 释放后使用(UAF)

  • 组长采样(PERF_FORMAT_GROUP)的 leader 会把每个计数成员 PerTaskCounter 原子字段的裸指针烘焙进它 per-CPU 的 SampleSlot(生命周期仅绑定 leader 的调度片),但 perf_event_open 此前仅凭 group_fd 建立 leader↔member 链接,未校验二者是否监控同一线程。若成员在另一线程 B 上,B 退出 + 关闭 fd 会在 leader 的 slot 仍持有其指针时释放该 ptc → 硬中断溢出处理程序中 UAF 解引用(可由一段合法的两线程用户态序列触发)。
  • 修复:要求成员与 leader 监控同一线程(PerTaskCounter.owner_tid),跨线程链接以 EINVAL 拒绝——对齐 Linux 的 group_leader->ctx == event->ctx 语义。同一线程成员位于 leader 所在 Threadperf_counters 列表,其 slot 在 perf_sched_out 先于 on_task_exit 注销,故裸指针必然长于注册期。

同批修复的次要项:死 Weak 剪枝 + 超 MAX_GROUP_MEMBERS 拒绝(而非静默丢弃);ioctl(RESET) 现生效(handler 以 per-task sink 为唯一真相源);install_pmu_irq_handler 增加与 REGISTERED CAS 独立的 INSTALLED 就绪标志;SAMPLE_RECORD_MAX_LEN 补齐 callchain 的 nr 词。

新增回归测试 perf-hw-group-crosstask(跨线程成员 open → EINVAL(22),同线程 → OK);perf-hw-group-sample / perf-hw-sample-read 回归仍全绿。同时把 perf record -a 每-CPU 采样 fan-out(de8eb64b3,QEMU smp4:4 核各 819 样本、wrong_cpu=0)一并纳入本栈(故 PR 标题已更新)。cargo fmt + 容器内 cargo xtask clippy --package starry-kernel(20/20)全绿。

2. 关于 orangepi-5-plus 板级 CI 失败

感谢指出该失败。我拉取了失败 job(87024716147)的完整启动日志,根因与 big.LITTLE PMU 代码无关——日志中没有任何 PMU panic 或 error,而是 perf-validate 板级用例的两个测试基建问题:

root@starry:/root # /usr/local/bin/perf-validate
/bin/sh: 1: /usr/local/bin/perf-validate: not found
...
kernel boot timed out after 900s
Error: starry board tests failed for 1 group(s): board-orangepi-5-plus/perf-validate
  • /usr/local/bin/perf-validate: not found——perf-validate 校验二进制未部署到板子 rootfs(板级资产部署项,此前 ZR233 评审的 3 项中板端部署路径仍需在真机验证);
  • kernel boot timed out after 900s——smp8 启动超时(本仓库已知的板级 smp8 启动挂起 gating,非本 PR 顶部 3 个净增提交引入)。

base dev 该 job 通过是因为它不含这个新增的 perf-validate 板级用例,并非本 PR 造成既有行为回归。这两项属板端部署 / smp8 启动的测试基建,需要在真机(RK3588)上定位板端 deploy.sh 部署路径与 smp8 启动;与本 PR 净增的分组 / SAMPLE_READ / 组长采样 / fan-out PMU 逻辑无关(QEMU 四架构全部通过)。

@mai-team-app mai-team-app Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

审查总结

本轮审查针对当前 head (cde643e38)——即在上一次 REQUEST_CHANGES 审查 (3844e0860) 之后推送的修复提交。新提交 cde643e38 标题为 fix(perf): reject cross-thread group-leader sampling links (UAF) + review fixes,包含以下重要修复:

  1. 跨线程组长链接 UAF 安全门控link_group_member 现在检查 owner_tid 一致性,跨线程链接返回 EINVAL。新增 perf-hw-group-crosstask 回归测试用例覆盖此路径。
  2. PMU IRQ 注册拆分为进程上下文安装 + IRQ 安全的逐核启用install_pmu_irq_handler()perf_event_open(系统调用上下文)中运行,避免在硬 IRQ IPI 臂 thunk 中调用 request_percpu_irq
  3. perf record -a 逐 CPU 采样扇出:系统级采样事件现在通过 IPI 在目标核上臂装其计数器 + 溢出 IRQ,而非所有事件堆积在打开核上。新增 perf-hw-smp-record-allcpu 回归测试。
  4. read_value 单调性修复:per-task 采样读取计数值现在以 sink 为唯一事实来源(先读、加、回写),而非先递增再存储——这与 ioctl(RESET) 交互正确。
  5. 已关闭成员清理link_group_member 现在清理 strong_count == 0 的成员后再追加,避免重复打开/关闭周期导致固定表项耗尽。
  6. on_clone_inherit 设置正确的 owner_tid:继承的计数器现在正确跟踪子线程的 tid。

本地验证

检查项 结果
cargo fmt --check ✅ 通过
cargo xtask clippy --package starry-kernel ✅ 20/20 通过
cargo xtask clippy --package ax-cpu ✅ 28/28 通过
[patch.crates-io] 检查 ✅ 无

CI 状态(当前 head cde643e38,运行 #29319516993)

已完成的作业:

  • Check formatting / run_host → success
  • Run spin-lint / run_container → success
  • Run sync-lint / run_container → success
  • Test arceos riscv64 qemu / run_host → success
  • Test axvisor aarch64 qemu / run_host → success
  • Test with std / run_host → success

仍在运行中(截至审查提交时):

  • Test starry self-hosted board orangepi-5-plus / run_hostin_progress(这是上一轮审查中在 3844e0860 上失败的相同作业)
  • Test starry loongarch64 qemu / run_container → in_progress
  • 其他板级/UEFI 作业 → in_progress

上一轮 CI(head 3844e0860,运行 #29314049785)在 orangepi-5-plus 上失败,其他所有 QEMU 作业均通过。当前修复的合理性分析:PMU IRQ 注册拆分(将 request_percpu_irq 移入进程上下文)以及新增的 perf record -a 逐核臂装路径直接涉及 big.LITTLE 集群感知代码——这正是 orangepi-5-plus 独有的测试面(RK3588 Cortex-A76 big + Cortex-A55 LITTLE,QEMU 不模拟)。因此该 CI 作业对验证修复效果至关重要。

重复/重叠分析

  • 本 PR (#1603) 是栈式 PR 链的顶部:#1577(多核基座) → callchain → #1601(软件事件) → #1602(tid + lost) → #1603(本 PR)。均为同一作者(JosephJoshua),各 PR 有独立范围,互补不重复。
  • rcore-os/tgoskits 中搜索未发现其他与本 PR 冲突或重叠的开放 PR。

代码审查意见

净增代码(栈顶 4 个提交)质量较高:

  • 不安全性处理得当:所有 unsafe 块均有 SAFETY 注释解释指针有效性和生命周期不变量。build_group_read 中成员原子指针的烘焙和读取逻辑清晰,生命周期不变量已记录。
  • 硬 IRQ 上下文安全性:溢出处理程序保持无分配,组成员读取通过预烘焙的原子指针表进行,无锁/无等待。
  • 跨线程 UAF 防护owner_tid 门控 + perf-hw-group-crosstask 回归测试提供了对此安全关键不变量的显式验证。
  • 测试布局正确:新增测试用例遵循 grouped C runner 结构(CMakeLists.txt + src/ 直接在 system/<case>/ 下),安装至 usr/bin/starry-test-suit
  • 错误传播link_group_member 现在返回 AxResult,错误在 perf_event_open 中正确传播。

审查结论

修复提交解决了上一轮审查中提出的问题。代码质量高,所有本地验证通过。orangepi-5-plus 板级 CI 仍在运行中——鉴于该测试在上一个 head 上失败,且当前修复直接涉及 big.LITTLE 集群代码路径,请等待该 CI 作业完成并确认通过后再合并。若 orangepi-5-plus 通过,此 PR 可批准(前提是 #1577/#1601/#1602 依赖栈已满足)。

Powered by deepseek-v4-pro

Board CI reported `/usr/local/bin/perf-validate: not found` even though the
binary was installed. Root cause: the board's ext4 rootfs is mounted
`commit=600` (dirty pages flushed to the SD only every 10 minutes). deploy.sh
installed the binary but never `sync`ed, so it lived only in Linux's page
cache. When the harness power-cycles into StarryOS, StarryOS mounts the ext4
fresh and does not see the uncommitted file -> "not found" -> the board test
waits for a verdict that never comes and times out (misreported as a kernel
boot timeout).

Fix: `sync` inside the sudo-install so the binary is durably on the SD before
StarryOS boots. Diagnosed on-board: the binary (md5 matches the current
source) is on mmcblk1p2 with no overlay/symlink, and a manual `sync` makes it
visible; the failure was purely durability, not a missing/corrupt binary.
@JosephJoshua

Copy link
Copy Markdown
Contributor Author

板级 perf-validate 失败根因已定位并修复(5c1b9d7f8

在真机(OrangePi 5 Plus,串口 + U-Boot)上完整复现并定位了 orangepi-5-plus 板级 CI 的 /usr/local/bin/perf-validate: not found + 900s 超时:

根因是部署持久化,不是缺二进制、也不是 PMU/组采样代码。 板子的 ext4 根文件系统以 commit=600 挂载(脏页每 10 分钟才刷到 SD)。deploy.sh 安装二进制后没有 sync,于是二进制只停留在 Linux 的 page cache,尚未落到物理 SD。当测试框架 power-cycle 进入 StarryOS、StarryOS 重新挂载该 ext4 时,看不到这个未提交的文件 → not found → shell_init_cmd 无 VERDICT 输出 → 等待 success_regex 直到 900s 超时(被报成 "kernel boot timed out",实为等 verdict 超时)。两个 CI 症状同一根因。

现场证据: 二进制确实在 mmcblk1p2(StarryOS 挂载为 / 的同一分区)的 /usr/local/bin/perf-validate,无 overlay / 无 symlink,md5 与当前源码新构建一致(7462551…);手动 sync 后即持久可见;二进制本身可正常执行(在板子 Linux 上跑出完整 BOARD_PERF_VALIDATE_* 输出)。

修复: deploy.sh 的 sudo-install 内追加 sync,保证 power-cycle 进 StarryOS 前二进制已持久落盘。已 cherry-pick 到 #1577 base(2855759a8)与本 PR(5c1b9d7f8)。板级 CI 重跑应能找到持久化后的二进制并跑出 VERDICT(smp1 → PARTIAL = 成功)。

另注(非本 PR 阻塞项): 复现过程中发现该物理板在同时拉起全部 8 核时会硬复位(panic=0 也不拦,说明是硬件复位而非 kernel panic;maxcpus=1 可完整启动到 Linux 登录 shell)——典型的 RK3588 供电欠压(8 核上电峰值电流超供电能力)。这解释了 smp8 启动挂起,但与 perf-validate 测试无关(该测试经 loady 走 StarryOS smp1,不受影响),且 perf-validate 配置本就容忍 smp8(smp1 PARTIAL 即成功)。建议对该物理板检查/更换电源。

@mai-team-app mai-team-app Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

审查总结

本 PR (#1603) 为 StarryOS 硬件 PMU perf 系列的第三个主题栈,实现:

  1. 事件分组与 PERF_FORMAT_GROUP 读取——leader↔member 弱引用登记、enable/disable 传播、整组计数布局 {nr, [te], [tr], {value,[id],[lost]}[nr]}
  2. PERF_SAMPLE_READ——per-task 溢出时每条样本携带事件的运行计数,read_value 单调性通过 sink 回写保证
  3. 组长采样(PERF_SAMPLE_READ | PERF_FORMAT_GROUP——per-task 采样 leader 溢出时在硬中断内无分配地输出整组计数,通过 arm_slice 预烘焙成员原子指针
  4. 跨线程组长链接 UAF 安全门控——link_group_member 检查 owner_tid 一致性,跨线程链接返回 EINVAL
  5. PMU IRQ 注册拆分——install_pmu_irq_handler() 在系统调用上下文运行,enable_pmu_irq_line() 为 IRQ 安全的逐核启用
  6. perf record -a 逐 CPU 采样扇出——系统级采样事件通过 IPI 在目标核上臂装其计数器
  7. OrangePi-5-Plus 板级部署修复——deploy.sh 添加 sync 确保二进制持久化到 SD 卡

代码审查

整体评价:代码质量高,架构清晰。

净增代码(栈顶提交)中:

  • 不安全性处理得当:所有 unsafe 块均有 SAFETY 注释解释指针有效性和生命周期不变量。build_group_read 中成员原子指针的烘焙和读取逻辑清晰,生命周期不变量已记录。
  • 硬 IRQ 上下文安全性:溢出处理程序保持无分配,组成员读取通过预烘焙的原子指针表进行,无锁/无等待。
  • 跨线程 UAF 防护owner_tid 门控 + perf-hw-group-crosstask 回归测试提供了对此安全关键不变量的显式验证。修复与 Linux group_leader->ctx == event->ctx 语义对齐。
  • read_value 单调性修复:per-task 采样读取计数值现在以 sink 为唯一事实来源(先读、加、回写),ioctl(RESET) 交互正确。
  • 死成员剪枝link_group_member 在追加前清理 strong_count == 0 的成员,避免重复打开/关闭周期导致固定表项耗尽。
  • 错误传播link_group_member 返回 AxResult,错误在 perf_event_open 中正确传播。
  • 测试布局正确:新增测试用例遵循 grouped C runner 结构(CMakeLists.txt + src/ 直接在 system/<case>/ 下),安装至 usr/bin/starry-test-suit
  • [patch.crates-io]:工作区未发现任何 crates.io patch。

本地验证

检查项 结果
cargo fmt --check ✅ 通过
cargo xtask clippy --package starry-kernel ✅ 20/20 通过
cargo xtask clippy --package ax-cpu ✅ 28/28 通过
cargo xtask clippy --package axbacktrace ✅ 3/3 通过
[patch.crates-io] 检查 ✅ 无

CI 状态(当前 head 5c1b9d7f8,运行 #29332574712)

CI 运行仍在进行中(in_progress),关键作业尚未完成:

  • Check formatting / run_hostin_progress
  • Run spin-lint / Run sync-lintin_progress
  • Starry QEMU 测试(x86_64/riscv64/loongarch64/aarch64)→ 已跳过(路径过滤,perf 测试仅在 aarch64 上运行)
  • Board 测试(orangepi-5-plus 等)→ 尚未开始

关于 OrangePi-5-Plus 板级 CI:

  • 前一轮 CI(head 3844e0860)在此板失败,根因是部署脚本未 sync,导致二进制在 ext4 commit=600 下未持久化到 SD 卡就被 StarryOS 引导。作者已诊断(非 PMU 代码问题)并通过 deploy.sh 添加 sync 修复。
  • 当前 CI 运行将验证此修复。

建议:等待 CI(尤其是 orangepi-5-plus 板级测试)完成并确认通过后再合并。

重复/重叠分析

  • 本 PR (#1603) 是栈式 PR 链的顶部:#1577(多核基座)→ callchain → #1601(软件事件)→ #1602(tid + lost)→ #1603(本 PR)。均为同一作者(JosephJoshua),各 PR 有独立范围,互补不重复。
  • rcore-os/tgoskits 中搜索未发现其他与本 PR 冲突或重叠的开放 PR。

前序审查处理

  • 第一轮审查(mai-team-app,CHANGES_REQUESTED)指出的 orangepi-5-plus 板级 CI 失败已由作者诊断并修复(部署脚本 sync)。
  • 第二轮审查(mai-team-app,COMMENTED)确认了 UAF 修复、IRQ 安装拆分等代码修复的正确性,CI 当时仍在进行中。
  • 作者的 PR 评论提供了详细的根因分析,解释了 perf-validate: not found 和 smp8 启动超时与 PMU 代码无关。

结论

代码质量高,所有本地验证通过,UAF 修复和 IRQ 安装拆分正确。CI 仍在运行中,请等待 CI(尤其是 orangepi-5-plus 板级测试)完成确认通过后再合并。本审查为评论性审查,不作为最终批准。

Powered by deepseek-v4-pro

@ZR233 ZR233 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

按栈顶三个净增提交复核,事件组、PERF_SAMPLE_READ 与组长采样的测试位置和无分配 IRQ 路径没有发现新的独立阻塞;但该 PR 当前仍携带未合入的 #1577 基座,而且 current-head OrangePi 5 Plus 硬门禁失败。日志明确显示两个 perf-validate board case 都执行 /usr/local/bin/perf-validate 并得到 not found,随后 fail regex 命中;最新 sync 提交无法解决“CI 从未把资产放进持久 rootfs”的问题。请先在 #1577/本栈提供受 board lease 管理的可复现 staging,并让 current-head board CI 通过,再重审栈顶语义。

@ZR233
ZR233 requested a review from shilei-massclouds July 15, 2026 01:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants