-
Notifications
You must be signed in to change notification settings - Fork 126
docs(starry): add macOS HVF self-build app #984
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from 48 commits
Commits
Show all changes
60 commits
Select commit
Hold shift + click to select a range
710a83b
docs(starry): add macOS HVF self-build app
40b23a4
docs(starry): clarify HVF self-build reproduction
6fd6832
test(starry): relax loongarch64 rust hello qemu limits
60dcaf6
test(starry): keep rust hello loongarch config unchanged
9fa496d
fix(starry): address macOS selfbuild review issues
217b0c4
test(starry): relax smoke qemu timeouts
93f25ac
test(starry): fix futex clone-thread readiness race
a3952e2
fix(starry): make macOS selfbuild app reproducible
0985033
docs(starry): clarify macOS selfbuild rootfs artifact
e19aeb8
docs(starry): document macOS selfbuild rootfs construction
955e2f3
fix(starry): make macOS rootfs build reproducible
31e7a4c
fix(starry): refresh macOS selfbuild reproduction
a5f60b8
fix(starry): refresh macOS selfbuild reproduction
3babde4
fix(starry): restore fast macOS selfbuild defaults
b76e61b
fix(starry): guard macOS selfbuild reproduction profile
25ebba6
docs(starry): add macOS selfbuild reproduce entry
6d29c12
fix(starry): align macOS selfbuild crate guard
00d0302
fix(starry): stabilize macOS selfbuild reproduction
2ff8ba3
fix(starry): make macOS selfbuild reproducible
141e042
docs(starry): clarify macOS selfbuild core count
9500ec0
fix(starry): make macos self-build reproducible
Alayfolk64 45f2f72
fix: fix the cargo path
Alayfolk64 1ea1dfd
Merge pull request #1 from 1301182193/fix/starry-macos-selfbuild
yks23 7382a2d
merge upstream dev into macOS selfbuild PR
3a60686
fix(starry): align macOS selfbuild merge
ddf8712
fix(axbuild): resolve starry external rootfs path
ebcfc4d
fix(axbuild): propagate explicit rootfs errors
ba5fbb4
fix(starry): gate uprobe state behind ebpf feature
f05d278
fix(starry): allow ebpf-only helpers without ebpf
d55d11d
fix(starry): gate c variadic feature
8cf0a06
fix(starry): render someboot linker script
1a2a4b6
fix(starry): avoid grouped test false failures
8deb8b2
fix(arceos): use single-threaded tcg for riscv smp tests
cc68efc
fix(starry): keep ebpf probe out of default suite
1a4dae9
fix(starry): avoid loongarch futex wake count race
e1be77f
fix(axvisor): set roc rk3568 uboot load addresses
d7ebdc4
fix(starry): relax futex wake count telemetry
b4419b8
fix(axvisor): keep roc fit load address from uboot
9145f40
fix(axvisor): reduce roc linux guest memory
d84fe97
fix(axbuild): silence unused arceos cargo parameter
f8a5d2a
fix(axvisor): extend vmx smoke timeout
ed2a3a1
fix(axtask): keep sleep blocked until deadline
9d5da91
fix(starry): expose kprobe module for kprobe test
ea5d24b
fix(starry): relax concurrent tcp payload wait
42e952e
fix(starry): enable kprobe dependency for test feature
6cf6652
fix(starry): keep kretprobe stack for kprobe test
e5e5ce5
fix(starry): stabilize concurrent tcp payload check
1df700f
fix(starry): silence kprobe test-only dead code
4adf118
fix(starry): keep selfbuild options app-local
7251e1a
test(starry): skip ebpf basics without bpf syscall
0e6e2b0
Merge remote-tracking branch 'upstream/dev' into HEAD
d31ad23
fix(arceos): mark primary cpu ipi-ready
f25ae20
fix(starry): address selfbuild review blockers
0c24db9
test(starry): stabilize futex bitset wake check
82afd6b
test(starry): treat futex wake count as telemetry
e485fa4
ci: retry pr checks
e048742
Merge remote-tracking branch 'upstream/dev' into HEAD
4b6088f
fix(starry): declare ebpf kmod feature
a0379c2
ci: retry flaky vmx check
ebea1c4
Merge remote-tracking branch 'upstream/dev' into app/starry-macos-sel…
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,270 @@ | ||
| # StarryOS macOS HVF Self-Build | ||
|
|
||
| This case is an Apple Silicon macOS reproduction workflow. The host runs QEMU | ||
| with HVF, boots an AArch64 StarryOS SMP guest, and runs guest `cargo build` to | ||
| build StarryOS again inside StarryOS. | ||
|
|
||
| The normal reproduction path builds the rootfs locally on macOS and does not use | ||
| Docker. `build_rootfs.sh` creates the AArch64 Alpine guest rootfs, installs the | ||
| pinned guest Rust/Cargo toolchain, prefetches the offline Cargo cache, and | ||
| injects the current TGOSKits source tree. | ||
|
|
||
| ## What This Demonstrates | ||
|
|
||
| - host: Apple Silicon macOS; | ||
| - accelerator: QEMU HVF; | ||
| - guest kernel: StarryOS AArch64 SMP kernel; | ||
| - guest workload: StarryOS guest runs Cargo and builds the `starryos` binary; | ||
| - stable build shape: 8-vCPU guest with eight Cargo jobs (`SMP=8 JOBS=8`); | ||
| - pass marker: `===STARRY-MACOS-SELFBUILD-PASS jobs=<N> elapsed=<seconds>===`. | ||
|
|
||
| The default reproduction proves that the StarryOS SMP guest can complete a | ||
| self-build while booted with 8 vCPUs and running Cargo with eight jobs. | ||
|
|
||
| Using AArch64/HVF keeps the guest ISA aligned with the Mac host CPU. This avoids | ||
| the cross-ISA TCG cost of RISC-V-on-macOS experiments and makes the self-build | ||
| result practical to reproduce on a laptop. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| ```bash | ||
| brew install qemu e2fsprogs zig llvm | ||
| ``` | ||
|
|
||
| The scripts use: | ||
|
|
||
| - `qemu-system-aarch64`; | ||
| - `debugfs` from Homebrew `e2fsprogs`; | ||
| - `zig` or an `aarch64-linux-musl-gcc` cross compiler for the seed kernel; | ||
| - `llvm-nm` and `llvm-objdump`, with Homebrew `llvm` used as fallback. | ||
|
|
||
| The generated self-build rootfs contains the guest Rust/Cargo toolchain, | ||
| offline Cargo dependencies, and a TGOSKits source tarball. `check_rootfs.sh` | ||
| verifies the minimum paths. | ||
|
|
||
| ## Reproduce From a Fresh Clone | ||
|
|
||
| Clone and enter the branch under test: | ||
|
|
||
| ```bash | ||
| git clone https://github.com/yks23/tgoskits.git | ||
| cd tgoskits | ||
| git checkout app/starry-macos-selfbuild | ||
| ``` | ||
|
|
||
| Run the complete reproduction on macOS: | ||
|
|
||
| ```bash | ||
| RUST_DIST_SERVER=https://rsproxy.cn \ | ||
| STARRY_CARGO_REGISTRY_INDEX=sparse+https://rsproxy.cn/index/ \ | ||
| apps/starry/macos-selfbuild/reproduce.sh | ||
| ``` | ||
|
|
||
| The mirror variables are optional. Omit them to use the official Rust and | ||
| crates.io endpoints, or set them to closer mirrors when `static.rust-lang.org` | ||
| or `index.crates.io` is slow. | ||
|
|
||
| `reproduce.sh` runs the three required steps: build or refresh the rootfs, build | ||
| the seed kernel, and launch the QEMU HVF self-build. On a base M1 with 8 GB | ||
| memory, close other heavy applications before running the 8-vCPU case; if it is | ||
| memory pressured, first verify the setup with: | ||
|
|
||
| ```bash | ||
| SMP=4 JOBS=4 MEM=3072M \ | ||
| RUST_DIST_SERVER=https://rsproxy.cn \ | ||
| STARRY_CARGO_REGISTRY_INDEX=sparse+https://rsproxy.cn/index/ \ | ||
| apps/starry/macos-selfbuild/reproduce.sh | ||
| ``` | ||
|
|
||
| The rootfs build downloads Alpine AArch64 APKs and Rust nightly components, then | ||
| runs `cargo fetch` on the host to populate the guest offline cache. The | ||
| `cargo fetch` phase may print only `Updating crates.io index` for several | ||
| minutes while the sparse registry cache is being populated. | ||
|
|
||
| For reproducibility, the default native rootfs payload is pinned to Alpine | ||
| `v3.23` and Rust `nightly-2026-05-28`. Do not switch `ALPINE_BRANCH=edge` for | ||
| the graded reproduction: Alpine edge currently provides a newer Cargo/Rust | ||
| package set and can change the build-std schedule substantially. | ||
|
|
||
| The expected output ends with: | ||
|
|
||
| ```text | ||
| STARRY_MACOS_SELFBUILD_ROOTFS_OK | ||
| ``` | ||
|
|
||
| After every `git pull`, branch switch, or source edit, refresh the source | ||
| payload embedded in the rootfs before running QEMU: | ||
|
|
||
| ```bash | ||
| apps/starry/macos-selfbuild/prepare_rootfs.sh | ||
| ``` | ||
|
|
||
| `run_selfbuild.sh` checks `/opt/tgoskits-src.meta` by default and exits early if | ||
| the rootfs was built from a different commit. This avoids accidentally running | ||
| an old rootfs for an hour. Set `REQUIRE_FRESH_ROOTFS=0` only for deliberate | ||
| stale-rootfs experiments. | ||
|
|
||
| If a prebuilt rootfs artifact is supplied, it can still be placed at the | ||
| standard path instead of rebuilding locally. Use either a local downloaded file: | ||
|
|
||
| ```bash | ||
| apps/starry/macos-selfbuild/fetch_rootfs.sh \ | ||
| --input /path/to/rootfs-aarch64-hvf-selfbuild.img | ||
| ``` | ||
|
|
||
| or a direct artifact URL: | ||
|
|
||
| ```bash | ||
| ROOTFS_URL='https://.../rootfs-aarch64-hvf-selfbuild.img' \ | ||
| apps/starry/macos-selfbuild/fetch_rootfs.sh | ||
| ``` | ||
|
|
||
| Build the seed StarryOS kernel on macOS: | ||
|
|
||
| ```bash | ||
| apps/starry/macos-selfbuild/build_kernel.sh | ||
| ``` | ||
|
|
||
| Run the complete 8-vCPU, eight-Cargo-job self-build: | ||
|
|
||
| ```bash | ||
| KERNEL=target/aarch64-unknown-none-softfloat/release/starryos.bin \ | ||
| ROOTFS=tmp/axbuild/rootfs/rootfs-aarch64-hvf-selfbuild.img \ | ||
| SMP=8 \ | ||
| JOBS=8 \ | ||
| RAYON_NUM_THREADS=1 \ | ||
| RUSTC_THREADS=2 \ | ||
| SOURCE_TMPFS=1 \ | ||
| QEMU_TIMEOUT_SEC=10800 \ | ||
| apps/starry/macos-selfbuild/run_selfbuild.sh | ||
| ``` | ||
|
|
||
| A successful run prints: | ||
|
|
||
| ```text | ||
| ===STARRY-MACOS-SELFBUILD-QEMU-AARCH64-PROFILE expected_crates~420=== | ||
| ===STARRY-MACOS-SELFBUILD-PASS jobs=8 elapsed=<seconds>=== | ||
| ===STARRY-MACOS-SELFBUILD-RUN-END rc=0=== | ||
| ``` | ||
|
|
||
| The qemu-aarch64 reproducible profile should show: | ||
|
|
||
| ```text | ||
| rustc_threads=2 | ||
| features=plat-dyn,ax-driver/virtio-blk,ax-driver/virtio-net,ax-driver/virtio-gpu,ax-driver/virtio-input,ax-driver/virtio-socket,starry-kernel/input,starry-kernel/vsock | ||
| ``` | ||
|
|
||
| The guest refuses a run whose `RUSTC_THREADS` differs from `2` unless | ||
| `ALLOW_SLOW_SELFBUILD=1` is explicitly set. This keeps the graded run on the | ||
| same qemu-aarch64 profile that was debugged for reproducibility. | ||
|
|
||
| The host runner also refuses unexpectedly large Cargo totals by default. The | ||
| current qemu-aarch64 profile is expected to report about `420` Cargo units. A much larger | ||
| total usually means a stale rootfs or slow feature set is being used; refresh the | ||
| rootfs from the current checkout and rerun the command above. | ||
|
|
||
| Logs are written under: | ||
|
|
||
| ```text | ||
| target/starry-macos-selfbuild/logs/ | ||
| ``` | ||
|
|
||
| The runner copies the input rootfs into | ||
| `target/starry-macos-selfbuild/rootfs/`, injects the guest runner scripts, and | ||
| runs QEMU against that working copy, so the input artifact is not modified. Set | ||
| `QEMU_SNAPSHOT=1` only when an explicit throw-away QEMU disk overlay is desired. | ||
|
|
||
| ## Quick Boot Check | ||
|
|
||
| To verify the rootfs and kernel reach the StarryOS guest shell without starting | ||
| the Cargo build: | ||
|
|
||
| ```bash | ||
| KERNEL=target/aarch64-unknown-none-softfloat/release/starryos.bin \ | ||
| ROOTFS=tmp/axbuild/rootfs/rootfs-aarch64-hvf-selfbuild.img \ | ||
| SMP=8 \ | ||
| BOOT_ONLY=1 \ | ||
| QEMU_TIMEOUT_SEC=180 \ | ||
| apps/starry/macos-selfbuild/run_selfbuild.sh | ||
| ``` | ||
|
|
||
| `BOOT_ONLY=1` is only a smoke test. Leave it unset for the full self-build. | ||
|
|
||
| ## Rootfs Contract | ||
|
|
||
| The generated rootfs is intentionally kept outside git because it is large. It | ||
| must contain: | ||
|
|
||
| ```text | ||
| /usr/bin/cargo | ||
| /opt/rust-nightly/bin/rustc | ||
| /opt/rust-nightly/lib/rustlib/src/rust/library/Cargo.lock | ||
| /usr/bin/aarch64-linux-musl-gcc | ||
| /opt/rustc-nightly-sysroot | ||
| /opt/rustdoc-nightly-sysroot | ||
| /opt/tgoskits/Cargo.toml or /opt/tgoskits-src.tar | ||
| /root/.cargo/registry/index | ||
| /root/.cargo/registry/cache | ||
| ``` | ||
|
|
||
| Check an already placed rootfs with: | ||
|
|
||
| ```bash | ||
| apps/starry/macos-selfbuild/check_rootfs.sh \ | ||
| tmp/axbuild/rootfs/rootfs-aarch64-hvf-selfbuild.img | ||
| ``` | ||
|
|
||
| If a prebuilt toolchain rootfs is available and only the source tree needs to be | ||
| refreshed, inject the current checkout without rebuilding the toolchain: | ||
|
|
||
| ```bash | ||
| apps/starry/macos-selfbuild/prepare_rootfs.sh \ | ||
| --base-rootfs tmp/axbuild/rootfs/rootfs-aarch64-hvf-toolchain.img \ | ||
| --output-rootfs tmp/axbuild/rootfs/rootfs-aarch64-hvf-selfbuild.img | ||
| ``` | ||
|
|
||
| `prepare_rootfs.sh` writes `/opt/tgoskits-src.tar` and | ||
| `/opt/tgoskits-src.meta`. The host runner checks this metadata before booting | ||
| QEMU, and the guest checks it again when `TGOSKITS_COMMIT` is supplied. | ||
|
|
||
| ## Important Knobs | ||
|
|
||
| | Variable | Default | Meaning | | ||
| | --- | --- | --- | | ||
| | `SMP` | `8` | QEMU vCPU count, passed to `-smp`. | | ||
| | `JOBS` | `SMP` | Guest Cargo job count. | | ||
| | `RAYON_NUM_THREADS` | `1` | Rayon worker limit for guest build scripts. | | ||
| | `RUSTC_THREADS` | `2` | Guest `-Zthreads=<N>` value for the reproducible qemu-aarch64 profile. | | ||
| | `SOURCE_TMPFS` | `1` | Copy source into `/tmp` before building. | | ||
| | `QEMU_TIMEOUT_SEC` | `7200` | Host timeout; use `0` to disable. | | ||
| | `QEMU_ACCEL` | `hvf` | QEMU accelerator string. | | ||
| | `QEMU_MACHINE` | `virt,gic-version=3` | QEMU machine string. | | ||
| | `QEMU_CPU` | `host` | QEMU CPU model for HVF. | | ||
| | `BOOT_ONLY` | `0` | Stop after the shell prompt instead of starting Cargo. | | ||
| | `BUILD_TARGET` | `aarch64-unknown-none-softfloat` | Guest Cargo target. | | ||
| | `BUILD_PACKAGE` | `starryos` | Cargo package to build. | | ||
| | `BUILD_BIN` | `starryos` | Cargo binary to build; set `none` for library package diagnostics. | | ||
| | `FEATURES` | `plat-dyn,ax-driver/virtio-blk,ax-driver/virtio-net,ax-driver/virtio-gpu,ax-driver/virtio-input,ax-driver/virtio-socket,starry-kernel/input,starry-kernel/vsock` | QEMU AArch64 feature profile used by the reproducible self-build; set it to an empty string for single-crate diagnostics. | | ||
| | `REQUIRE_FRESH_ROOTFS` | `1` | Refuse a rootfs whose embedded source commit does not match the checkout. | | ||
| | `ALLOW_SLOW_SELFBUILD` | `0` | Permit the slow full-device feature profile only for explicit experiments. | | ||
| | `GUEST_MONITOR_INTERVAL_SEC` | `60` | Print guest `ps` snapshots while Cargo runs; set `0` to disable. | | ||
| | `EXTRA_RUSTFLAGS` | empty | Extra guest Rust flags for local experiments. | | ||
|
|
||
| ## Rootfs Rebuild Details | ||
|
|
||
| ```bash | ||
| apps/starry/macos-selfbuild/build_rootfs.sh | ||
| ``` | ||
|
|
||
| That helper creates an AArch64 Alpine payload natively on macOS, installs the | ||
| pinned guest Rust toolchain and offline Cargo cache, and injects it with | ||
| `debugfs`. It writes: | ||
|
|
||
| ```text | ||
| tmp/axbuild/rootfs/rootfs-aarch64-hvf-toolchain.img | ||
| tmp/axbuild/rootfs/rootfs-aarch64-hvf-selfbuild.img | ||
| ``` | ||
|
|
||
| The script still accepts `--payload` or `ROOTFS_PAYLOAD_URL` for an externally | ||
| prepared payload tarball. `--build-payload-with-docker` is kept only as an | ||
| explicit maintainer fallback and is not used by the macOS reproduction path. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.