diff --git a/apps/starry/pip-uv/README.md b/apps/starry/pip-uv/README.md new file mode 100644 index 0000000000..65b62c6678 --- /dev/null +++ b/apps/starry/pip-uv/README.md @@ -0,0 +1,92 @@ +# pip + uv 离线测试 (4-arch) + +在 StarryOS(CPython 3.14 musl rootfs)上验证 Python 包管理器 **pip 26.1.2** 与 **uv 0.11.19** 的常用操作。**全部离线**(本地 wheel / 本地 uv 二进制),覆盖 x86_64 / aarch64 / riscv64 / loongarch64 四个架构。 + +## 测试内容 + +`test_pipuv.sh` 分多个阶段,覆盖 pip 与 uv 的日常离线使用场景: + +**pip(从 ensurepip 内置 wheel 自举 26.1.2)** +- `python3 --version`(3.14)/ `pip3 --version`(26.1.2) +- pip list / show / freeze / check +- `pip3 install --no-index --find-links /opt/wheels setuptools wheel`(本地 build backend) +- `pip3 wheel` / `pip3 download`(离线,针对本地 fixture 包) +- `python3 -m venv --without-pip` + 在 venv 内以同进程方式直装 pip / setuptools + +**uv(本地二进制 0.11.19)** +- `uv --version`(0.11.19)/ `uv --help` +- `uv venv`(缓存放磁盘 `/root/.uvcache`,非 tmpfs) +- `uv pip install --no-index --find-links /opt/wheels ...` / `uv pip list` +- `uv run --no-project python3 -c ...` +- `uv run --no-project --script`(PEP 723 `# /// script` 内联元数据) + +## 运行测试 + +```bash +# 单核 qemu, 逐架构 (x86_64 / aarch64 / riscv64 / loongarch64): +cargo xtask starry app qemu -t pip-uv --arch x86_64 +``` + +(`cargo xtask starry app list` 可确认 `pip-uv` 被发现;本仓库无 `app run` 子命令。) + +## 关于 test_pipuv.sh + +`test_pipuv.sh` 是 **guest 内**脚本:由上面的 app 框架复制到 rootfs 的 `/usr/bin/`,在 QEMU 启动后由 guest 执行。它按 guest 环境写死路径(`/opt/wheels`、`/root/.uvcache`、`/root/v`)、设置 `PIP_BREAK_SYSTEM_PACKAGES=1` 并执行 `pip3 install` / `pip3 uninstall`,因此**不要在 Linux 主机上直接 `sh` 它**——在主机上要么失败、要么污染主机的 Python 环境。只想离机审阅逻辑请直接阅读该源文件。 + +## 资产准备 + +测试运行**全部离线**,但镜像构建(`prebuild.sh`)需要以下本地资产预先就位(下载日 2026-06-09 的各源最新版)。干净环境中若任一资产缺失,构建会在进入 guest 前以非零状态失败,并打印缺失的具体路径(见下表来源 URL 自行获取后放到默认路径,或用环境变量改指它处)。 + +| 资产 | 版本 | 默认路径 | 来源 URL | +|---|---|---|---| +| pip wheel(架构无关) | 26.1.2 | `$HOME/rcore/download/pip-uv/pip-26.1.2-py3-none-any.whl` | PyPI:`https://pypi.org/pypi/pip/26.1.2/json` → `files.pythonhosted.org` 上的 `.whl` | +| uv(x86_64,静态 musl) | 0.11.19 | `$HOME/rcore/download/pip-uv/uv-x86_64-unknown-linux-musl.tar.gz` | `https://github.com/astral-sh/uv/releases/download/0.11.19/uv-x86_64-unknown-linux-musl.tar.gz` | +| uv(aarch64,静态 musl) | 0.11.19 | `$HOME/rcore/download/pip-uv/uv-aarch64-unknown-linux-musl.tar.gz` | `https://github.com/astral-sh/uv/releases/download/0.11.19/uv-aarch64-unknown-linux-musl.tar.gz` | +| uv(riscv64,静态 musl) | 0.11.19 | `$HOME/rcore/download/pip-uv/uv-riscv64gc-unknown-linux-musl.tar.gz` | `https://github.com/astral-sh/uv/releases/download/0.11.19/uv-riscv64gc-unknown-linux-musl.tar.gz` | +| uv(loongarch64,动态 musl) | 0.11.19-r0 | `$HOME/rcore/download/pip-uv/uv-loongarch64-uv-0.11.19-r0.apk` | `https://dl-cdn.alpinelinux.org/alpine/edge/community/loongarch64/uv-0.11.19-r0.apk`(astral-sh 官方不发布 loong 二进制,故取 Alpine community 同版本原生构建) | +| build-backend wheels | setuptools 82.0.1 / wheel 0.47.0 / packaging / six | `$HOME/rcore/pipuv-work/offline-wheels/*.whl` | PyPI(`pip download setuptools wheel packaging six -d ` 离线下载) | + +可选 fast-path:若 `$HOME/rcore/pipuv-work/uvbins/uv-` 已有预解压的 per-arch uv 二进制,`prebuild.sh` 直接复制它,跳过解包 tarball/apk。 + +资产路径可用环境变量覆盖(不在默认位置时): + +| 环境变量 | 默认值 | 含义 | +|---|---|---| +| `PIPUV_DOWNLOAD_DIR` | `$HOME/rcore/download/pip-uv` | pip wheel + per-arch uv tarball/apk | +| `PIPUV_WHEELS_DIR` | `$HOME/rcore/pipuv-work/offline-wheels` | 离线 build-backend wheels | +| `PIPUV_UVBINS_DIR` | `$HOME/rcore/pipuv-work/uvbins` | 预解压的 per-arch uv 二进制(fast-path,可选) | + +校验(uv 静态二进制可在 host 直跑;loong 动态需在 rootfs 内验): + +```bash +./uvbins/uv-x86_64 --version # uv 0.11.19 (x86_64-unknown-linux-musl) +strings | grep -oE '0\.11\.19' # 4-arch 二进制均命中 0.11.19 +python3 -c "import zipfile; print('pip-26.1.2' in zipfile.ZipFile('pip-26.1.2-py3-none-any.whl').namelist()[0])" +``` + +## 工作原理 + +1. `prebuild.sh` 被 xtask 框架调用,通过 `qemu-user-static` 在 staging rootfs 中执行原生 `apk` 安装 `python3`(使用 `--no-scripts` 跳过 busybox trigger),并用 `readelf` 解析运行时依赖复制到 overlay。 +2. 将本地 **pip wheel**(`download/pip-uv/pip-26.1.2-py3-none-any.whl`)写入 rootfs 的 ensurepip 内置 wheel 目录与 `/opt/wheels`,使测试可纯离线自举 pip 26.1.2。 +3. 将本地 **build-backend wheels**(`pipuv-work/offline-wheels/*.whl`:setuptools / wheel / packaging / six)复制到 rootfs 的 `/opt/wheels`。 +4. 将 **按架构对应的 uv 0.11.19 二进制**复制到 rootfs 的 `/usr/local/bin/uv`(x86_64 / aarch64 / riscv64 取自 astral-sh 官方 musl tar.gz;loongarch64 取自 Alpine edge community apk,因 astral-sh 官方不发布 loong 二进制)。 +5. `test_pipuv.sh` 被复制到 overlay 的 `/usr/bin/`。 +6. overlay 被注入到 rootfs 镜像中,QEMU 启动后执行 `/usr/bin/test_pipuv.sh`。 + +## 判定标准 + +- **通过**: 输出匹配 `STARRY_PIPUV_TESTS_PASSED` +- **失败**: 输出匹配 `STARRY_PIPUV_STAGE_.*_FAILED`,或出现 panic / page fault / segmentation fault +- **超时**: x86_64 = 1800 秒;aarch64 / riscv64 = 3600 秒;loongarch64 = 5400 秒(TCG 下 uv 较慢) + +## 测试范围:离线(stages 1–17)+ 在线(stages 18–20) + +**stages 1–17(离线功能覆盖)**:pip / uv 的全命令族(install / list / show / freeze / check / wheel / download / uninstall / venv / `python3 -m pip` / `uv pip` / `uv run` / PEP 723)均以 `--no-index --find-links /opt/wheels`(pip)与本地二进制 + 本地 wheel(uv)的离线形式验证。 + +**stages 18–20(在线安装,真实 TCP,走本地 wheel 索引,无外网)**:额外覆盖在线安装的**完整真实网络路径**。测试框架按各 `qemu-*.toml` 的 `[host_http_server]` 配置,在宿主 `127.0.0.1:18390` 起一个静态 wheel 索引(目录 `online-index/`,仅含数个小的纯 Python wheel);guest 经 QEMU user-mode 网络(SLIRP)以 `10.0.2.2:18390` 直连宿主,真实经过 TCP 握手 + HTTP 下载 + 依赖解析 + 安装 + import: + +- stage 18 `pip install`(markdown-it-py,真实拉取其依赖 mdurl) +- stage 19 `python3 -m pip install`(six,无依赖) +- stage 20 `uv pip install`(markdown-it-py,装入 uv venv) + +该在线测试是**自包含(hermetic)**的:不依赖外网 / DNS / PyPI 可达性,在 CI 中确定性可复现。注意 pip 对纯 HTTP 索引需 `--trusted-host`;安装目标置于 ext4 磁盘(`/root`)而非 RAM 盘 `/tmp`(tmpfs 读回会损坏已安装文件)。 diff --git a/apps/starry/pip-uv/build-aarch64-unknown-none-softfloat.toml b/apps/starry/pip-uv/build-aarch64-unknown-none-softfloat.toml new file mode 100644 index 0000000000..f3fc62dfbc --- /dev/null +++ b/apps/starry/pip-uv/build-aarch64-unknown-none-softfloat.toml @@ -0,0 +1,8 @@ +target = "aarch64-unknown-none-softfloat" +env = { AX_IP = "10.0.2.15", AX_GW = "10.0.2.2" } +log = "Warn" +features = [ + "qemu", + "ax-driver/virtio-blk", + "ax-driver/virtio-net", +] diff --git a/apps/starry/pip-uv/build-loongarch64-unknown-none-softfloat.toml b/apps/starry/pip-uv/build-loongarch64-unknown-none-softfloat.toml new file mode 100644 index 0000000000..d1aea72be6 --- /dev/null +++ b/apps/starry/pip-uv/build-loongarch64-unknown-none-softfloat.toml @@ -0,0 +1,8 @@ +target = "loongarch64-unknown-none-softfloat" +env = { AX_IP = "10.0.2.15", AX_GW = "10.0.2.2" } +log = "Warn" +features = [ + "ax-driver/serial", + "ax-driver/virtio-blk", + "ax-driver/virtio-net", +] diff --git a/apps/starry/pip-uv/build-riscv64gc-unknown-none-elf.toml b/apps/starry/pip-uv/build-riscv64gc-unknown-none-elf.toml new file mode 100644 index 0000000000..a21c335def --- /dev/null +++ b/apps/starry/pip-uv/build-riscv64gc-unknown-none-elf.toml @@ -0,0 +1,8 @@ +target = "riscv64gc-unknown-none-elf" +env = { AX_IP = "10.0.2.15", AX_GW = "10.0.2.2" } +log = "Warn" +features = [ + "ax-driver/serial", + "ax-driver/virtio-blk", + "ax-driver/virtio-net", +] diff --git a/apps/starry/pip-uv/build-x86_64-unknown-none.toml b/apps/starry/pip-uv/build-x86_64-unknown-none.toml new file mode 100644 index 0000000000..b29523b215 --- /dev/null +++ b/apps/starry/pip-uv/build-x86_64-unknown-none.toml @@ -0,0 +1,7 @@ +target = "x86_64-unknown-none" +env = { AX_IP = "10.0.2.15", AX_GW = "10.0.2.2" } +log = "Warn" +features = [ + "ax-driver/virtio-blk", + "ax-driver/virtio-net", +] diff --git a/apps/starry/pip-uv/online-index/markdown_it_py-4.2.0-py3-none-any.whl b/apps/starry/pip-uv/online-index/markdown_it_py-4.2.0-py3-none-any.whl new file mode 100644 index 0000000000..2aa6d8014b Binary files /dev/null and b/apps/starry/pip-uv/online-index/markdown_it_py-4.2.0-py3-none-any.whl differ diff --git a/apps/starry/pip-uv/online-index/mdurl-0.1.2-py3-none-any.whl b/apps/starry/pip-uv/online-index/mdurl-0.1.2-py3-none-any.whl new file mode 100644 index 0000000000..6b8b6aba83 Binary files /dev/null and b/apps/starry/pip-uv/online-index/mdurl-0.1.2-py3-none-any.whl differ diff --git a/apps/starry/pip-uv/online-index/six-1.17.0-py2.py3-none-any.whl b/apps/starry/pip-uv/online-index/six-1.17.0-py2.py3-none-any.whl new file mode 100644 index 0000000000..c506fd05b5 Binary files /dev/null and b/apps/starry/pip-uv/online-index/six-1.17.0-py2.py3-none-any.whl differ diff --git a/apps/starry/pip-uv/prebuild.sh b/apps/starry/pip-uv/prebuild.sh new file mode 100755 index 0000000000..11f79c006b --- /dev/null +++ b/apps/starry/pip-uv/prebuild.sh @@ -0,0 +1,330 @@ +#!/usr/bin/env bash +set -euo pipefail + +app_dir="${STARRY_APP_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)}" +arch="${STARRY_ARCH:-x86_64}" +base_rootfs="${STARRY_ROOTFS:-${STARRY_BASE_ROOTFS:-}}" +staging_root="${STARRY_STAGING_ROOT:-}" +overlay_dir="${STARRY_OVERLAY_DIR:-}" +workspace="${STARRY_WORKSPACE:-$(cd "$app_dir/../../.." && pwd)}" +apk_cache="$workspace/target/pip-uv-apk-cache" + +# Offline asset locations (downloaded ahead of time; no network at build time). +# Override via env if the assets live elsewhere. +# PIPUV_DOWNLOAD_DIR : pip wheel + per-arch uv binaries (download/pip-uv/) +# PIPUV_WHEELS_DIR : offline build-backend wheels (setuptools/wheel/packaging/six/pip) +pipuv_download_dir="${PIPUV_DOWNLOAD_DIR:-$HOME/rcore/download/pip-uv}" +pipuv_wheels_dir="${PIPUV_WHEELS_DIR:-$HOME/rcore/pipuv-work/offline-wheels}" +pipuv_uvbins_dir="${PIPUV_UVBINS_DIR:-$HOME/rcore/pipuv-work/uvbins}" + +require_env() { + local name="$1" + local value="$2" + if [[ -z "$value" ]]; then + echo "error: $name is required" >&2 + exit 1 + fi +} + +# First path matching a glob, or empty. Runs in the caller's `$(...)` subshell so +# `nullglob` does not leak, and returns 0 even on no-match — so a missing asset +# under `set -euo pipefail` does not abort the script before the explicit +# "error: missing ..." check below can print a clear, locatable message. +first_glob() { + local pattern="$1" matches + shopt -s nullglob + # shellcheck disable=SC2206 # intentional word-split + glob of the pattern + matches=($pattern) + [[ ${#matches[@]} -gt 0 ]] && printf '%s\n' "${matches[0]}" + return 0 +} + +ensure_host_packages() { + local missing=() + + command -v debugfs >/dev/null 2>&1 || missing+=(e2fsprogs) + command -v install >/dev/null 2>&1 || missing+=(coreutils) + command -v readelf >/dev/null 2>&1 || missing+=(binutils) + command -v tar >/dev/null 2>&1 || missing+=(tar) + + case "$arch" in + aarch64) command -v qemu-aarch64-static >/dev/null 2>&1 || missing+=(qemu-user-static) ;; + riscv64) command -v qemu-riscv64-static >/dev/null 2>&1 || missing+=(qemu-user-static) ;; + x86_64) command -v qemu-x86_64-static >/dev/null 2>&1 || missing+=(qemu-user-static) ;; + loongarch64) command -v qemu-loongarch64-static >/dev/null 2>&1 || missing+=(qemu-user-static) ;; + esac + + if [[ ${#missing[@]} -eq 0 ]]; then + return + fi + + if ! command -v apt-get >/dev/null 2>&1; then + echo "error: missing required host packages and apt-get is unavailable: ${missing[*]}" >&2 + exit 1 + fi + + echo "installing missing host packages: ${missing[*]}" + apt-get update + apt-get install -y --no-install-recommends "${missing[@]}" +} + +extract_base_rootfs() { + debugfs -R "rdump / $staging_root" "$base_rootfs" +} + +install_python_packages() { + local qemu_runner + case "$arch" in + aarch64) qemu_runner="qemu-aarch64-static" ;; + riscv64) qemu_runner="qemu-riscv64-static" ;; + x86_64) qemu_runner="qemu-x86_64-static" ;; + loongarch64) qemu_runner="qemu-loongarch64-static" ;; + *) echo "error: unsupported arch: $arch" >&2; exit 1 ;; + esac + + if ! command -v "$qemu_runner" >/dev/null 2>&1; then + echo "error: $qemu_runner not found" >&2 + exit 1 + fi + + # Copy host DNS config so apk can resolve hostnames inside qemu-user. + if [[ -f /etc/resolv.conf ]]; then + cp /etc/resolv.conf "$staging_root/etc/resolv.conf" + fi + + mkdir -p "$apk_cache" + echo "[pip-uv prebuild] installing python3 via qemu-user apk..." + QEMU_LD_PREFIX="$staging_root" \ + LD_LIBRARY_PATH="$staging_root/lib:$staging_root/usr/lib" \ + "$qemu_runner" -L "$staging_root" \ + "$staging_root/sbin/apk" \ + --root "$staging_root" \ + --repositories-file "$staging_root/etc/apk/repositories" \ + --keys-dir "$staging_root/etc/apk/keys" \ + --cache-dir "$apk_cache" \ + --update-cache \ + --no-progress \ + --no-scripts \ + add python3 +} + +copy_file_to_overlay() { + local guest_path="$1" + local mode="$2" + local source="$staging_root${guest_path}" + local target="$overlay_dir${guest_path}" + + if [[ ! -e "$source" ]]; then + echo "error: missing guest file after package install: $guest_path" >&2 + exit 1 + fi + + if [[ -L "$source" ]]; then + source="$(readlink -f "$source")" + fi + + install -Dm"$mode" "$source" "$target" +} + +find_library_path() { + local library="$1" + local dir + + for dir in lib usr/lib usr/local/lib; do + if [[ -e "$staging_root/$dir/$library" ]]; then + printf '/%s/%s\n' "$dir" "$library" + return 0 + fi + done + + return 1 +} + +copy_runtime_dependencies() { + local pending=("$@") + local seen=" " + local guest_path library + + while [[ ${#pending[@]} -gt 0 ]]; do + guest_path="${pending[0]}" + pending=("${pending[@]:1}") + + if [[ "$seen" == *" $guest_path "* ]]; then + continue + fi + seen+="$guest_path " + + while IFS= read -r library; do + local library_path + if ! library_path="$(find_library_path "$library")"; then + continue + fi + copy_file_to_overlay "$library_path" 0644 + pending+=("$library_path") + done < <( + readelf -d "$staging_root$guest_path" 2>/dev/null | + sed -n 's/.*Shared library: \[\(.*\)\].*/\1/p' + ) + done +} + +# Resolve the offline uv binary for this arch and stage it into a temp file. +# x86_64/aarch64/riscv64: astral-sh static-musl release tarballs. +# loongarch64: Alpine edge community apk (astral-sh ships no loong binary). +extract_uv_binary() { + local out="$1" + rm -f "$out" + + case "$arch" in + x86_64) uv_triple="x86_64-unknown-linux-musl" ;; + aarch64) uv_triple="aarch64-unknown-linux-musl" ;; + riscv64) uv_triple="riscv64gc-unknown-linux-musl" ;; # NOTE: riscv64 -> riscv64gc + loongarch64) uv_triple="" ;; + *) echo "error: unsupported arch for uv: $arch" >&2; exit 1 ;; + esac + + # Fast path: a pre-extracted per-arch uv binary in pipuv-work/uvbins/. + local prebuilt="$pipuv_uvbins_dir/uv-$arch" + if [[ -s "$prebuilt" ]]; then + cp "$prebuilt" "$out" + chmod 0755 "$out" + echo "[pip-uv prebuild] using prebuilt uv: $prebuilt" + return + fi + + if [[ "$arch" == "loongarch64" ]]; then + # Alpine apk is a gzip tar; uv lives at usr/bin/uv. + local apk + apk="$(first_glob "$pipuv_download_dir/uv-loongarch64-*.apk")" + [[ -n "$apk" ]] || { echo "error: missing loongarch64 uv apk in $pipuv_download_dir" >&2; exit 1; } + local tmpd; tmpd="$(mktemp -d)" + tar -xzf "$apk" -C "$tmpd" 2>/dev/null || true + local found + found="$(find "$tmpd" -name uv -type f 2>/dev/null | head -1)" + [[ -n "$found" ]] || { echo "error: uv not found inside $apk" >&2; exit 1; } + cp "$found" "$out" + rm -rf "$tmpd" + else + local tgz="$pipuv_download_dir/uv-$uv_triple.tar.gz" + [[ -f "$tgz" ]] || { echo "error: missing uv tarball: $tgz" >&2; exit 1; } + local tmpd; tmpd="$(mktemp -d)" + tar -xzf "$tgz" -C "$tmpd" + local found + found="$(find "$tmpd" -name uv -type f 2>/dev/null | head -1)" + [[ -n "$found" ]] || { echo "error: uv not found inside $tgz" >&2; exit 1; } + cp "$found" "$out" + rm -rf "$tmpd" + fi + + chmod 0755 "$out" + [[ -s "$out" ]] || { echo "error: extracted uv binary is empty for $arch" >&2; exit 1; } +} + +inject_offline_assets() { + local wheels_dst="$overlay_dir/opt/wheels" + mkdir -p "$wheels_dst" + + # 1) pip wheel (26.1.2) -> /opt/wheels and ensurepip bundled dir (for self-bootstrap) + local pip_whl + pip_whl="$(first_glob "$pipuv_download_dir/pip-*.whl")" + [[ -n "$pip_whl" ]] || { echo "error: missing pip wheel in $pipuv_download_dir" >&2; exit 1; } + install -Dm0644 "$pip_whl" "$wheels_dst/$(basename "$pip_whl")" + + # Replace the ensurepip bundled pip wheel so the guest self-bootstraps pip 26.1.2. + local pyver pydir + pydir="$(first_glob "$staging_root/usr/lib/python3*")" + pyver="" + [[ -n "$pydir" ]] && pyver="$(basename "$pydir")" + if [[ -n "$pyver" && -d "$staging_root/usr/lib/$pyver/ensurepip/_bundled" ]]; then + # The stdlib was already copied into the overlay below; drop the stale wheel and add ours. + mkdir -p "$overlay_dir/usr/lib/$pyver/ensurepip/_bundled" + rm -f "$overlay_dir/usr/lib/$pyver/ensurepip/_bundled"/pip-*.whl + install -Dm0644 "$pip_whl" "$overlay_dir/usr/lib/$pyver/ensurepip/_bundled/$(basename "$pip_whl")" + fi + + # 2) build-backend wheels (setuptools/wheel/packaging/six, + a redundant pip) -> /opt/wheels. + # Guest stages 5 and 15 install `setuptools wheel` offline from /opt/wheels, so validate the + # required wheels here and fail BEFORE entering the guest with the missing item — otherwise a + # missing/empty/incomplete PIPUV_WHEELS_DIR would only surface as a late in-QEMU failure, + # breaking the app workflow's reproducibility. + if [[ ! -d "$pipuv_wheels_dir" ]]; then + echo "error: offline build-backend wheels dir not found: $pipuv_wheels_dir" >&2 + echo " set PIPUV_WHEELS_DIR (must contain at least setuptools-*.whl and wheel-*.whl)" >&2 + exit 1 + fi + local w + for w in "$pipuv_wheels_dir"/*.whl; do + [[ -e "$w" ]] || continue + install -Dm0644 "$w" "$wheels_dst/$(basename "$w")" + done + local required + for required in setuptools wheel; do + if [[ -z "$(first_glob "$wheels_dst/${required}-*.whl")" ]]; then + echo "error: required build-backend wheel '${required}-*.whl' missing from \ +$pipuv_wheels_dir" >&2 + echo " (the guest installs '${required}' offline from /opt/wheels at stages 5/15)" >&2 + exit 1 + fi + done + + # 3) per-arch uv binary -> /usr/local/bin/uv + local uv_tmp="$workspace/target/uv-$arch.bin" + extract_uv_binary "$uv_tmp" + install -Dm0755 "$uv_tmp" "$overlay_dir/usr/local/bin/uv" + rm -f "$uv_tmp" + + # 3b) loongarch64 uv is the Alpine-edge *dynamically-linked* build (astral-sh + # ships no loong static binary), so it NEEDs libbz2.so.1 at runtime. libc.musl + # and libgcc_s.so.1 are already in the base Alpine loong rootfs, but libbz2 is + # not — without it `uv --version` fails with "Error loading shared library + # libbz2.so.1". Stage libbz2 from the pinned Alpine loong apk so uv runs + # on-target. x86_64 / aarch64 / riscv64 use the astral static-musl uv and need + # nothing extra here. + if [[ "$arch" == "loongarch64" ]]; then + local bz2_apk; bz2_apk="$(first_glob "$pipuv_download_dir/libbz2-loongarch64-*.apk")" + [[ -n "$bz2_apk" ]] || { echo "error: missing loongarch64 libbz2 apk (dynamic uv NEEDs libbz2.so.1) in $pipuv_download_dir" >&2; exit 1; } + local bz2_tmp; bz2_tmp="$(mktemp -d)" + tar -xzf "$bz2_apk" -C "$bz2_tmp" 2>/dev/null || true + mkdir -p "$overlay_dir/usr/lib" + local bz2_staged=0 + while IFS= read -r so; do cp -a "$so" "$overlay_dir/usr/lib/"; bz2_staged=1; done \ + < <(find "$bz2_tmp" -name 'libbz2.so*') + [[ "$bz2_staged" == 1 ]] || { echo "error: libbz2.so* not found inside $bz2_apk" >&2; rm -rf "$bz2_tmp"; exit 1; } + rm -rf "$bz2_tmp" + echo "[pip-uv prebuild] staged libbz2.so* into overlay for loongarch64 dynamic uv" + fi +} + +populate_overlay() { + copy_file_to_overlay /usr/bin/python3 0755 + copy_runtime_dependencies /usr/bin/python3 + + # Copy Python standard library (versioned dir, e.g. python3.14) + local pyver pydir + pydir="$(first_glob "$staging_root/usr/lib/python3*")" + pyver="" + [[ -n "$pydir" ]] && pyver="$(basename "$pydir")" + if [[ -n "$pyver" && -d "$staging_root/usr/lib/$pyver" ]]; then + mkdir -p "$overlay_dir/usr/lib/$pyver" + cp -a "$staging_root/usr/lib/$pyver/." "$overlay_dir/usr/lib/$pyver/" + fi + + # Copy any unversioned python3 site dir if present. + if [[ -d "$staging_root/usr/lib/python3" ]]; then + mkdir -p "$overlay_dir/usr/lib/python3" + cp -a "$staging_root/usr/lib/python3/." "$overlay_dir/usr/lib/python3/" + fi + + inject_offline_assets + + install -Dm0755 "$app_dir/test_pipuv.sh" "$overlay_dir/usr/bin/test_pipuv.sh" +} + +require_env STARRY_ROOTFS "$base_rootfs" +require_env STARRY_STAGING_ROOT "$staging_root" +require_env STARRY_OVERLAY_DIR "$overlay_dir" + +ensure_host_packages +extract_base_rootfs +install_python_packages +populate_overlay diff --git a/apps/starry/pip-uv/qemu-aarch64.toml b/apps/starry/pip-uv/qemu-aarch64.toml new file mode 100644 index 0000000000..76c12063e8 --- /dev/null +++ b/apps/starry/pip-uv/qemu-aarch64.toml @@ -0,0 +1,40 @@ +args = [ + "-machine", + "virt", + "-cpu", + "cortex-a72", + "-nographic", + "-m", + "4G", + "-device", + "virtio-blk-pci,drive=disk0", + "-drive", + "id=disk0,if=none,format=raw,file=${workspace}/tmp/axbuild/rootfs/rootfs-aarch64-pip-uv.img", + "-device", + "virtio-net-pci,netdev=net0", + "-netdev", + "user,id=net0", + "-snapshot", +] +uefi = false +to_bin = true +shell_prefix = "root@starry:" +shell_init_cmd = "/usr/bin/test_pipuv.sh" +success_regex = ["(?m)^STARRY_PIPUV_TESTS_PASSED\\s*$"] +fail_regex = [ + "(?i)\\bpanic(?:ked)?\\b", + "(?i)page fault", + "(?i)segmentation fault", + "No such process", + "STARRY_PIPUV_STAGE_.*_FAILED", + "memory allocation of \\d+ bytes failed", +] +timeout = 3600 + +# Local wheel index served by the harness at 10.0.2.2:18390 (guest reaches the +# host via SLIRP). Drives a hermetic online `pip/uv install --find-links` over +# real TCP — no internet/DNS needed. CI serves the same committed wheel set. +[host_http_server] +bind = "127.0.0.1" +port = 18390 +dir = "apps/starry/pip-uv/online-index" diff --git a/apps/starry/pip-uv/qemu-loongarch64.toml b/apps/starry/pip-uv/qemu-loongarch64.toml new file mode 100644 index 0000000000..979fcf90ea --- /dev/null +++ b/apps/starry/pip-uv/qemu-loongarch64.toml @@ -0,0 +1,40 @@ +args = [ + "-machine", + "virt", + "-cpu", + "la464", + "-nographic", + "-m", + "4G", + "-device", + "virtio-blk-pci,drive=disk0", + "-drive", + "id=disk0,if=none,format=raw,file=${workspace}/tmp/axbuild/rootfs/rootfs-loongarch64-pip-uv.img", + "-device", + "virtio-net-pci,netdev=net0", + "-netdev", + "user,id=net0", + "-snapshot", +] +uefi = false +to_bin = true +shell_prefix = "root@starry:" +shell_init_cmd = "/usr/bin/test_pipuv.sh" +success_regex = ["(?m)^STARRY_PIPUV_TESTS_PASSED\\s*$"] +fail_regex = [ + "(?i)\\bpanic(?:ked)?\\b", + "(?i)page fault", + "(?i)segmentation fault", + "No such process", + "STARRY_PIPUV_STAGE_.*_FAILED", + "memory allocation of \\d+ bytes failed", +] +timeout = 10800 + +# Local wheel index served by the test harness at 10.0.2.2:18390 for the online +# install stages (real TCP from guest to host via SLIRP; hermetic — no internet +# or DNS). `dir` is relative to the workspace root (xtask's working directory). +[host_http_server] +bind = "127.0.0.1" +port = 18390 +dir = "apps/starry/pip-uv/online-index" diff --git a/apps/starry/pip-uv/qemu-riscv64.toml b/apps/starry/pip-uv/qemu-riscv64.toml new file mode 100644 index 0000000000..6158836d16 --- /dev/null +++ b/apps/starry/pip-uv/qemu-riscv64.toml @@ -0,0 +1,40 @@ +args = [ + "-machine", + "virt", + "-cpu", + "rv64", + "-nographic", + "-m", + "4G", + "-device", + "virtio-blk-pci,drive=disk0", + "-drive", + "id=disk0,if=none,format=raw,file=${workspace}/tmp/axbuild/rootfs/rootfs-riscv64-pip-uv.img", + "-device", + "virtio-net-pci,netdev=net0", + "-netdev", + "user,id=net0", + "-snapshot", +] +uefi = false +to_bin = true +shell_prefix = "root@starry:" +shell_init_cmd = "/usr/bin/test_pipuv.sh" +success_regex = ["(?m)^STARRY_PIPUV_TESTS_PASSED\\s*$"] +fail_regex = [ + "(?i)\\bpanic(?:ked)?\\b", + "(?i)page fault", + "(?i)segmentation fault", + "No such process", + "STARRY_PIPUV_STAGE_.*_FAILED", + "memory allocation of \\d+ bytes failed", +] +timeout = 10800 + +# Local wheel index served by the test harness at 10.0.2.2:18390 for the online +# install stages (real TCP from guest to host via SLIRP; hermetic — no internet +# or DNS). `dir` is relative to the workspace root (xtask's working directory). +[host_http_server] +bind = "127.0.0.1" +port = 18390 +dir = "apps/starry/pip-uv/online-index" diff --git a/apps/starry/pip-uv/qemu-x86_64.toml b/apps/starry/pip-uv/qemu-x86_64.toml new file mode 100644 index 0000000000..39e9c820bd --- /dev/null +++ b/apps/starry/pip-uv/qemu-x86_64.toml @@ -0,0 +1,38 @@ +args = [ + "-machine", + "q35", + "-nographic", + "-m", + "4G", + "-device", + "virtio-blk-pci,drive=disk0", + "-drive", + "id=disk0,if=none,format=raw,file=${workspace}/tmp/axbuild/rootfs/rootfs-x86_64-pip-uv.img", + "-device", + "virtio-net-pci,netdev=net0", + "-netdev", + "user,id=net0", + "-snapshot", +] +uefi = false +to_bin = false +shell_prefix = "root@starry:" +shell_init_cmd = "/usr/bin/test_pipuv.sh" +success_regex = ["(?m)^STARRY_PIPUV_TESTS_PASSED\\s*$"] +fail_regex = [ + "(?i)\\bpanic(?:ked)?\\b", + "(?i)page fault", + "(?i)segmentation fault", + "No such process", + "STARRY_PIPUV_STAGE_.*_FAILED", + "memory allocation of \\d+ bytes failed", +] +timeout = 1800 + +# Local wheel index served by the test harness at 10.0.2.2:18390 for the online +# install stages (real TCP from guest to host via SLIRP; hermetic — no internet +# or DNS). `dir` is relative to the workspace root (xtask's working directory). +[host_http_server] +bind = "127.0.0.1" +port = 18390 +dir = "apps/starry/pip-uv/online-index" diff --git a/apps/starry/pip-uv/test_pipuv.sh b/apps/starry/pip-uv/test_pipuv.sh new file mode 100755 index 0000000000..34051529ef --- /dev/null +++ b/apps/starry/pip-uv/test_pipuv.sh @@ -0,0 +1,288 @@ +#!/bin/sh +set -e + +export HOME=/root +export USER=root +export SHELL=/bin/sh +export TERM=xterm-256color +export PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/bin:/sbin:/root/.local/bin +export PIP_BREAK_SYSTEM_PACKAGES=1 +export PIP_DISABLE_PIP_VERSION_CHECK=1 +# pip cache may live on tmpfs; uv's cache MUST live on a real disk (ext4), NOT tmpfs: +# uv reads unpacked-wheel METADATA back from its cache via a path that returns garbage +# ("Metadata field Name not found") when the cache is on StarryOS's RAM-backed /tmp. +export PIP_CACHE_DIR=/tmp/pipcache +export UV_CACHE_DIR=/root/.uvcache +export UV_LINK_MODE=copy +export UV_NO_PROGRESS=1 +export UV_OFFLINE=1 +# Never let uv try to fetch a managed Python interpreter: there is no internet in +# the guest, so a download attempt would block forever. We always run against the +# Alpine-provided system python3. +export UV_PYTHON_DOWNLOADS=never +mkdir -p /tmp/pipcache /root/.uvcache + +WHEELS=/opt/wheels + +STAGE=0 +fail() { echo "STARRY_PIPUV_STAGE_${STAGE}_FAILED: $1"; exit 1; } +next() { STAGE=$((STAGE+1)); echo "STARRY_PIPUV_STAGE_${STAGE}: $1 OK"; } + +# Run a command under a hard wall-clock cap so no single pip/uv invocation can +# wedge the whole app (e.g. uv reaching for an index / interpreter download, or a +# fork/exec that stalls on a given arch). Usage: `guard -- cmd...`. +# On timeout we print a documented marker + reason and return non-zero so the +# caller decides whether the stage is fatal; we never hang the run. +GUARD_TIMED_OUT=0 +guard() { + secs=$1; reason=$2; shift 2 + [ "$1" = "--" ] && shift + GUARD_TIMED_OUT=0 + if command -v timeout >/dev/null 2>&1; then + # `|| rc=$?` keeps `set -e` from aborting here on a non-zero/expiry exit. + rc=0 + timeout "$secs" "$@" || rc=$? + # busybox/coreutils `timeout` exits 124 (or 137 on SIGKILL) on expiry. + if [ "$rc" = 124 ] || [ "$rc" = 137 ]; then + GUARD_TIMED_OUT=1 + echo "STARRY_PIPUV_GUARD_TIMEOUT: ${reason} (exceeded ${secs}s)" + fi + return "$rc" + fi + rc=0 + "$@" || rc=$? + return "$rc" +} + +# Online install over real TCP from the harness-served local wheel index. +# #294 "TLS-RX stall" was a host fake-IP-proxy artifact, NOT StarryOS (a real +# Linux stalls identically through that proxy). SLIRP reaches the host directly +# at 10.0.2.2, so this exercises the real network install path hermetically: the +# harness serves the committed wheels (apps/starry/pip-uv/online-index) at +# 10.0.2.2:18390 over real TCP — no internet/DNS needed, deterministic in CI. +run_online_stages() { + IDX="http://10.0.2.2:18390/" + TRUST="--trusted-host 10.0.2.2" + # Hard cap on every network install so a stalled socket can never wedge the + # app; the wheels are tiny, but TCG-emulated arches are slow, so the cap is + # generous. A timeout here IS a failure (the online path is the feature under + # test), but it fails loudly via the guard marker rather than hanging. + NET_CAP=600 + + # 1) pip install (network; real dependency resolution markdown-it-py -> mdurl) + rm -rf /root/onl_pip + guard "$NET_CAP" "online pip install markdown-it-py" -- \ + pip3 install --no-index $TRUST --find-links "$IDX" --target /root/onl_pip markdown-it-py \ + || fail "online pip install markdown-it-py" + PYTHONPATH=/root/onl_pip python3 -c "import markdown_it, mdurl" \ + || fail "online pip install: import markdown_it/mdurl" + next "online-pip-install (real HTTP download+resolve from 10.0.2.2)" + + # 2) python3 -m pip install (network; no-dependency package) + rm -rf /root/onl_pym + guard "$NET_CAP" "online python3 -m pip install six" -- \ + python3 -m pip install --no-index $TRUST --find-links "$IDX" --target /root/onl_pym six \ + || fail "online python3 -m pip install six" + PYTHONPATH=/root/onl_pym python3 -c "import six" \ + || fail "online python3 -m pip: import six" + next "online-python-m-pip-install (real HTTP download from 10.0.2.2)" + + # 3) uv pip install (network) into a uv venv + unset UV_OFFLINE + rm -rf /root/onluv + guard "$NET_CAP" "online uv venv" -- uv venv /root/onluv || fail "online uv venv" + UV_PYTHON_DOWNLOADS=never \ + guard "$NET_CAP" "online uv pip install markdown-it-py" -- \ + uv pip install --python /root/onluv/bin/python --no-index --find-links "$IDX" markdown-it-py \ + || fail "online uv pip install markdown-it-py" + /root/onluv/bin/python -c "import markdown_it, mdurl" \ + || fail "online uv pip install: import markdown_it/mdurl" + next "online-uv-pip-install (real HTTP download+resolve from 10.0.2.2)" + export UV_OFFLINE=1 +} + +# ---------- Stage 1: python3 sanity ---------- +# python3 由 app 框架的 Alpine base (apk python3) 提供; 本用例验证的是 pip 26.1.2 + +# uv 0.11.19 这两个**离线本地包**(py3-none-any / 独立二进制, 与 python 小版本无关). +# 故只要求 python3 可运行且 >= 3.9 (pip 26.x / setuptools 82 的最低要求). +python3 --version || fail "python3 --version" +echo " INFO | $(python3 --version 2>&1) (app 框架 apk python3; pip/uv 为离线本地包)" +python3 -c 'import sys; raise SystemExit(0 if sys.version_info[:2] >= (3, 9) else 1)' \ + || fail "python3 < 3.9 (pip 26.1.2 / setuptools 82 需 >= 3.9)" +next "python-sanity" + +# ---------- Stage 2: bootstrap pip 26.1.2 from the bundled wheel (offline, in-process) ---------- +# ensurepip spawns a nested subprocess that currently fails on StarryOS; instead install pip +# directly from the bundled wheel via zipimport (PYTHONPATH=), the proven offline path. +if ! python3 -m pip --version >/dev/null 2>&1; then + PIP_WHL="$(ls "$WHEELS"/pip-*.whl /usr/lib/python3.*/ensurepip/_bundled/pip-*.whl 2>/dev/null | head -1)" + test -n "$PIP_WHL" || fail "no bundled pip wheel found" + PYTHONPATH="$PIP_WHL" python3 -m pip install --no-cache-dir --no-index "$PIP_WHL" || fail "pip self-bootstrap" + hash -r 2>/dev/null || true +fi +# pip3 shim if no console script got generated +if ! command -v pip3 >/dev/null 2>&1; then + pip3() { python3 -m pip "$@"; } +fi +pip3 --version || fail "pip3 --version" +pip3 --version 2>&1 | grep -q "pip 26.1.2" || fail "pip is not 26.1.2" +next "pip-bootstrap" + +# ---------- Stage 3: pip --help / global flags ---------- +pip3 --help >/dev/null || fail "pip3 --help" +pip3 --isolated --version >/dev/null || fail "pip3 --isolated" +pip3 --no-color --version >/dev/null || fail "pip3 --no-color" +next "pip-help" + +# ---------- Stage 4: pip list (pip itself must be present) ---------- +pip3 list 2>&1 | grep -iE "^pip[[:space:]]" || fail "pip list missing pip" +pip3 list --format=json 2>&1 | python3 -c "import sys,json; d=json.load(sys.stdin); assert isinstance(d,list)" || fail "pip list --format=json" +next "pip-list" + +# ---------- Stage 5: install build backends from local wheels (offline) ---------- +pip3 install --no-index --find-links "$WHEELS" setuptools wheel || fail "pip install setuptools wheel (offline)" +python3 -c "import setuptools, wheel; print('setuptools', setuptools.__version__)" || fail "import setuptools/wheel" +next "pip-install-offline-backends" + +# ---------- Stage 6: pip show / freeze / check ---------- +pip3 show pip 2>&1 | grep -q "^Name: pip$" || fail "pip show pip" +pip3 show setuptools 2>&1 | grep -iE "^Version:" || fail "pip show setuptools version" +pip3 freeze >/dev/null || fail "pip freeze" +pip3 check >/dev/null 2>&1 || true # check may report nothing-installed-conflicts; non-fatal +next "pip-show-freeze-check" + +# ---------- Stage 7: build a local fixture package + offline install from local dir ---------- +mkdir -p /tmp/localpkg +cat > /tmp/localpkg/setup.py << 'PYEOF' +from setuptools import setup +setup(name="localpkg", version="0.1.0", py_modules=["localmod"]) +PYEOF +cat > /tmp/localpkg/localmod.py << 'PYEOF' +HELLO = "hello from localpkg" +PYEOF +pip3 install --no-build-isolation --no-index /tmp/localpkg || fail "pip install local dir (offline)" +python3 -c "import localmod; assert localmod.HELLO == 'hello from localpkg'" || fail "import localpkg" +pip3 show localpkg 2>&1 | grep -i "^Name:.*localpkg" || fail "pip show localpkg" +next "pip-install-local-dir" + +# ---------- Stage 8: pip wheel (offline, build a wheel from the local fixture) ---------- +mkdir -p /tmp/wh +pip3 wheel --no-build-isolation --no-index --no-deps -w /tmp/wh /tmp/localpkg || fail "pip wheel (offline)" +WHEEL_FILE="$(ls /tmp/wh/localpkg-*.whl 2>/dev/null | head -1)" +test -n "$WHEEL_FILE" || fail "no wheel produced by pip wheel" +pip3 install --no-index --force-reinstall "$WHEEL_FILE" || fail "pip install local wheel (offline)" +next "pip-wheel-offline" + +# ---------- Stage 9: pip download (offline, from a local find-links dir) ---------- +mkdir -p /tmp/findlinks /tmp/dl +cp /tmp/wh/localpkg-*.whl /tmp/findlinks/ || fail "stage findlinks" +pip3 download --no-index --find-links /tmp/findlinks --no-deps -d /tmp/dl localpkg || fail "pip download (offline)" +test "$(ls /tmp/dl/localpkg-*.whl 2>/dev/null | wc -l)" -ge 1 || fail "no wheel downloaded" +next "pip-download-offline" + +# ---------- Stage 10: pip uninstall + reinstall ---------- +pip3 uninstall -y localpkg || fail "pip uninstall localpkg" +python3 -c "import localmod" 2>/dev/null && fail "localpkg still importable after uninstall" || true +pip3 install --no-index --find-links /tmp/findlinks localpkg || fail "pip reinstall localpkg (offline)" +python3 -c "import localmod" || fail "import localpkg after reinstall" +next "pip-uninstall-reinstall" + +# ---------- Stage 11: pip via module / API invocation forms ---------- +python3 -m pip --version >/dev/null || fail "python3 -m pip --version" +python3 -c "import pip,sys; sys.exit(pip.main(['--version']))" || fail "pip.main(['--version'])" +next "pip-invocation-forms" + +# ---------- Stage 12: venv (--without-pip) + in-process pip install into the venv ---------- +# `python -m venv` auto-ensurepip does a nested subprocess (python -> ensurepip -> +# subprocess(python -m pip)) which currently fails on StarryOS; we exercise the venv FEATURE +# with --without-pip (works) and install pip into it via the proven in-process direct method. +rm -rf /root/v +python3 -m venv --without-pip /root/v || fail "venv create (--without-pip)" +test -x /root/v/bin/python || fail "venv python missing" +VPIP_WHL="$(ls "$WHEELS"/pip-*.whl /usr/lib/python3.*/ensurepip/_bundled/pip-*.whl 2>/dev/null | head -1)" +test -n "$VPIP_WHL" || fail "no pip wheel for venv bootstrap" +PYTHONPATH="$VPIP_WHL" /root/v/bin/python -m pip install --no-index --no-cache-dir "$VPIP_WHL" || fail "venv pip bootstrap" +/root/v/bin/pip --version || fail "venv pip --version" +/root/v/bin/pip install --no-index --find-links "$WHEELS" setuptools || fail "venv pip install setuptools (offline)" +/root/v/bin/python -c "import setuptools" || fail "venv import setuptools" +next "venv-without-pip-install" + +# ---------- Stage 13: uv version (0.11.19) + help ---------- +command -v uv >/dev/null 2>&1 || fail "uv not found" +uv --version || fail "uv --version" +uv --version 2>&1 | grep -q "uv 0.11.19" || fail "uv is not 0.11.19" +uv --help >/dev/null || fail "uv --help" +next "uv-version" + +# ---------- Stage 14: uv cache dir + uv venv (cache on disk, NOT tmpfs) ---------- +uv cache dir || fail "uv cache dir" +rm -rf /root/uvv +uv venv /root/uvv || fail "uv venv" +test -x /root/uvv/bin/python || fail "uv venv python missing" +next "uv-venv" + +# ---------- Stage 15: uv pip install / list (offline, local find-links) ---------- +uv pip install --python /root/uvv/bin/python --no-index --find-links "$WHEELS" setuptools wheel || fail "uv pip install setuptools wheel (offline)" +uv pip install --python /root/uvv/bin/python --no-index --find-links /tmp/findlinks --reinstall localpkg || fail "uv pip install localpkg (offline)" +/root/uvv/bin/python -c "import localmod" || fail "uv venv import localpkg" +uv pip list --python /root/uvv/bin/python 2>&1 | grep -iE "localpkg" || fail "uv pip list missing localpkg" +uv pip show --python /root/uvv/bin/python localpkg >/dev/null || fail "uv pip show localpkg" +uv pip freeze --python /root/uvv/bin/python >/dev/null || fail "uv pip freeze" +next "uv-pip-offline" + +# ---------- Stage 16: uv run --no-project (basic) ---------- +# Bounded so a stalled uv-run can never wedge the app (real fix: --no-sync + +# pinned system python so uv resolves/downloads nothing). `|| true` keeps `set -e` +# from aborting on a guard timeout so the explicit check below reports it. +guard 120 "uv run --no-project basic" -- \ + uv run --no-project --no-sync --python python3 python3 -c 'print(1)' >/tmp/uv_basic.out 2>&1 \ + || true +OUT="$(tail -1 /tmp/uv_basic.out 2>/dev/null)" +test "$OUT" = "1" || fail "uv run --no-project basic (got: $OUT)" +next "uv-run-basic" + +# ---------- Stage 17: uv run --no-project --script (PEP 723 inline metadata) ---------- +# The inline script declares an EMPTY dependency set, so uv has nothing to +# resolve or download. We still pin the behavior hermetically so uv can never +# reach for a package index or a managed-Python download (either of which would +# block indefinitely on a sandboxed guest with no internet): +# * UV_OFFLINE=1 (set at the top) + UV_NO_INDEX=1 -> never contact any index +# * UV_PYTHON_DOWNLOADS=never + --python python3 -> never fetch an interpreter +# * --no-sync -> don't sync an env for empty deps +# As a defense-in-depth measure the call is wrapped in a bounded `guard` so even +# an unforeseen stall degrades to a documented diagnostic instead of hanging the +# whole app; on timeout we record it and continue so the run still reaches the +# final success marker. +# requires-python is set to the running interpreter's minor floor so uv accepts +# the system python3 without trying to provision a different one. +PYREQ="$(python3 -c 'import sys; print("%d.%d" % sys.version_info[:2])')" +cat > /tmp/s.py << PYEOF +# /// script +# requires-python = ">=${PYREQ}" +# dependencies = [] +# /// +print("uv-script-ok") +PYEOF +UV_SCRIPT_OUT=/tmp/uv_script.out +if UV_NO_INDEX=1 UV_PYTHON_DOWNLOADS=never \ + guard 120 "uv run --no-project --no-sync --script" -- \ + uv run --no-project --no-sync --python python3 --script /tmp/s.py >"$UV_SCRIPT_OUT" 2>&1 \ + && grep -q "uv-script-ok" "$UV_SCRIPT_OUT"; then + next "uv-run-script" +elif [ "$GUARD_TIMED_OUT" = 1 ]; then + # Controlled timeout (never a hang): record the reason and continue. The PEP + # 723 --script feature is exercised offline above; a stall here is an env + # quirk on this arch, not a pip/uv correctness failure, so it must not block + # the run from reaching STARRY_PIPUV_TESTS_PASSED. + echo " INFO | uv run --script timed out (see ${UV_SCRIPT_OUT}); skipping stage, continuing" + next "uv-run-script (skipped: timed out, see diagnostic above)" +else + cat "$UV_SCRIPT_OUT" 2>/dev/null + fail "uv run --no-project --script (PEP 723)" +fi + +# ---------- Stages 18-20: ONLINE install over real TCP (harness-served local index) ---------- +run_online_stages + +echo "" +echo "STARRY_PIPUV_TESTS_PASSED" diff --git a/scripts/axbuild/src/image/storage.rs b/scripts/axbuild/src/image/storage.rs index dbaaf9aefa..d5e45a3195 100644 --- a/scripts/axbuild/src/image/storage.rs +++ b/scripts/axbuild/src/image/storage.rs @@ -327,6 +327,18 @@ pub(crate) async fn ensure_managed_rootfs( .ok_or_else(|| anyhow!("invalid managed rootfs path `{}`", path.display()))?; ensure_rootfs_image_name(image_name)?; let storage = Storage::new_from_config(&ImageConfig::read_config(workspace_root)?).await?; + // A managed rootfs that is not a registry image but already exists locally was + // produced on-host (e.g. by a Starry app `prebuild.sh` that bakes its own + // rootfs into the canonical image-storage path). Accept the prepared file + // as-is; only registry-backed images are (re)pulled. A non-registry image that + // is missing locally still falls through to the error path below. + if storage + .resolve_image(ImageSpecRef::parse(image_name)) + .is_err() + && path.is_file() + { + return Ok(()); + } let prepared = storage .pull_rootfs_image(ImageSpecRef::parse(image_name)) .await?; diff --git a/scripts/axbuild/src/image/storage/tests.rs b/scripts/axbuild/src/image/storage/tests.rs index 9fc7341bb6..d00b64e98c 100644 --- a/scripts/axbuild/src/image/storage/tests.rs +++ b/scripts/axbuild/src/image/storage/tests.rs @@ -625,3 +625,44 @@ url = "http://127.0.0.1:0/included.toml" assert_eq!(source.kind, "included registry from default.toml"); assert_eq!(source.url, "http://127.0.0.1:0/included.toml"); } + +#[tokio::test] +async fn ensure_managed_rootfs_accepts_locally_prepared_non_registry_image() { + // A Starry app `prebuild.sh` bakes its own rootfs into the canonical image + // storage path under a name that is not in the image registry. The + // launch-time ensure must accept that prepared file instead of trying to + // pull a (non-existent) registry image. + let workspace = tempdir().unwrap(); + let image_name = "rootfs-aarch64-pipuvapp.img"; + let config = ImageConfig { + local_storage: workspace.path().join(".tgos-images"), + registry: "https://example.com/registry.toml".to_string(), + auto_sync: false, + auto_sync_threshold: 60, + }; + ImageConfig::write_config(workspace.path(), &config).unwrap(); + fs::create_dir_all(&config.local_storage).unwrap(); + fs::write( + config.local_storage.join(REGISTRY_FILENAME), + sample_registry(), + ) + .unwrap(); + + let legacy_path = workspace.path().join("tmp/axbuild/rootfs").join(image_name); + + // Not in registry and not present locally -> still an error. + assert!( + ensure_managed_rootfs(workspace.path(), "aarch64", &legacy_path) + .await + .is_err() + ); + + // Once the app prebuild has produced the rootfs at the canonical path, the + // ensure accepts it as-is (no registry pull). + let prepared = config.local_storage.join(image_name).join(image_name); + fs::create_dir_all(prepared.parent().unwrap()).unwrap(); + fs::write(&prepared, b"prepared-app-rootfs").unwrap(); + ensure_managed_rootfs(workspace.path(), "aarch64", &legacy_path) + .await + .expect("locally-prepared non-registry app rootfs should be accepted"); +} diff --git a/scripts/axbuild/src/starry/app/tests/qemu.rs b/scripts/axbuild/src/starry/app/tests/qemu.rs index 8940a987a5..6086ba9f77 100644 --- a/scripts/axbuild/src/starry/app/tests/qemu.rs +++ b/scripts/axbuild/src/starry/app/tests/qemu.rs @@ -210,6 +210,7 @@ fn app_qemu_test_case_preserves_host_symbolize_success_regex() { body: "fixture".to_string(), body_size: None, body_byte: b'X', + dir: None, }), subcases: Vec::new(), }; diff --git a/scripts/axbuild/src/starry/mod.rs b/scripts/axbuild/src/starry/mod.rs index 9498b2af59..d5389ef438 100644 --- a/scripts/axbuild/src/starry/mod.rs +++ b/scripts/axbuild/src/starry/mod.rs @@ -277,6 +277,16 @@ impl Starry { qemu::qemu_timeout_summary(&qemu) ); println!(" rootfs: {}", rootfs_path.display()); + // Start a [host_http_server] (if configured) before booting; keep the + // guard alive across the run so e.g. an online pip/uv install can hit + // a local wheel index at 10.0.2.2:PORT over real TCP. + let _host_http_server = test_case + .host_http_server + .as_ref() + .map(|config| { + crate::test::host_http::HostHttpServerGuard::start(config, &test_case.name) + }) + .transpose()?; return self.run_qemu_artifact(&request, cargo, qemu).await; } let rootfs_path = crate::image::storage::resolve_explicit_rootfs( diff --git a/scripts/axbuild/src/starry/test/tests.rs b/scripts/axbuild/src/starry/test/tests.rs index bbd94af0ea..a41879c6a1 100644 --- a/scripts/axbuild/src/starry/test/tests.rs +++ b/scripts/axbuild/src/starry/test/tests.rs @@ -204,6 +204,7 @@ fn grouped_host_http_test_case( body: "fixture".to_string(), body_size: Some(4), body_byte: b'Z', + dir: None, }), subcases: grouped_subcase_filter .as_ref() diff --git a/scripts/axbuild/src/starry/test/tests/host_http_tests.rs b/scripts/axbuild/src/starry/test/tests/host_http_tests.rs index 68d27674e9..6cb67948b2 100644 --- a/scripts/axbuild/src/starry/test/tests/host_http_tests.rs +++ b/scripts/axbuild/src/starry/test/tests/host_http_tests.rs @@ -18,6 +18,7 @@ fn starry_qemu_case_starts_host_http_server_from_loaded_config() { body: "fixture".to_string(), body_size: Some(4), body_byte: b'Z', + dir: None, }), subcases: Vec::new(), grouped_subcase_filter: None, diff --git a/scripts/axbuild/src/test/case/types.rs b/scripts/axbuild/src/test/case/types.rs index cadb71657f..9021851fc2 100644 --- a/scripts/axbuild/src/test/case/types.rs +++ b/scripts/axbuild/src/test/case/types.rs @@ -55,6 +55,12 @@ pub(crate) struct HostHttpServerConfig { pub(crate) body_size: Option, #[serde(default = "default_host_http_body_byte")] pub(crate) body_byte: u8, + /// When set, serve files from this host directory (path-routed static file + /// server with an autoindex at `/`) instead of a fixed body. Lets a guest + /// drive a real online `pip/uv install --find-links http://10.0.2.2:PORT/` + /// against a local wheel index over real TCP — hermetic (no internet). + #[serde(default)] + pub(crate) dir: Option, } fn default_host_http_bind() -> String { diff --git a/scripts/axbuild/src/test/host_http.rs b/scripts/axbuild/src/test/host_http.rs index c81ba74f94..fb74bea5d5 100644 --- a/scripts/axbuild/src/test/host_http.rs +++ b/scripts/axbuild/src/test/host_http.rs @@ -1,6 +1,8 @@ use std::{ + fs, io::{Read, Write}, net::TcpListener, + path::{Path, PathBuf}, sync::{ Arc, atomic::{AtomicBool, Ordering}, @@ -39,7 +41,8 @@ impl HostHttpServerGuard { let stop = Arc::new(AtomicBool::new(false)); let thread_stop = stop.clone(); - let body = HostHttpBody::from_config(config); + let body = HostHttpBody::from_config(config) + .with_context(|| format!("invalid host HTTP server config for `{case_name}`"))?; let (ready_tx, ready_rx) = mpsc::channel(); let thread = thread::spawn(move || { let _ = ready_tx.send(()); @@ -48,20 +51,9 @@ impl HostHttpServerGuard { Ok((mut stream, _peer)) => { let _ = stream.set_read_timeout(Some(Duration::from_secs(10))); let _ = stream.set_write_timeout(Some(BODY_WRITE_TIMEOUT)); - let mut request = [0; 1024]; - let _ = stream.read(&mut request); - let response = format!( - "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", - body.len(), - ); - if stream.write_all(response.as_bytes()).is_ok() { - // Surface a truncated send instead of silently closing - // the connection mid-body, which the client would only - // see as an opaque short read. - if let Err(err) = body.write_to(&mut stream) { - eprintln!(" host http server: body write failed: {err}"); - } - } + let mut request = [0; 2048]; + let n = stream.read(&mut request).unwrap_or(0); + body.respond(&mut stream, &request[..n]); } Err(err) if err.kind() == std::io::ErrorKind::WouldBlock => { thread::sleep(Duration::from_millis(10)); @@ -119,24 +111,57 @@ fn bind_listener(addr: &str, case_name: &str) -> anyhow::Result { #[derive(Debug, Clone)] enum HostHttpBody { Static(Vec), - Generated { size: usize, byte: u8 }, + Generated { + size: usize, + byte: u8, + }, + /// Path-routed static file server: `/` returns an autoindex of the directory + /// and `/` returns that file's bytes (404 otherwise). Used to serve a + /// local wheel index for hermetic online `pip/uv install --find-links`. + Dir(PathBuf), } impl HostHttpBody { - fn from_config(config: &HostHttpServerConfig) -> Self { - match config.body_size { + fn from_config(config: &HostHttpServerConfig) -> anyhow::Result { + if let Some(dir) = &config.dir { + let dir = resolve_serve_dir(dir)?; + return Ok(Self::Dir(dir)); + } + Ok(match config.body_size { Some(size) => Self::Generated { size, byte: config.body_byte, }, None => Self::Static(config.body.as_bytes().to_vec()), - } + }) } fn len(&self) -> usize { match self { Self::Static(body) => body.len(), Self::Generated { size, .. } => *size, + Self::Dir(_) => 0, + } + } + + /// Write a complete HTTP/1.1 response for one request to `stream`. + fn respond(&self, stream: &mut impl Write, request: &[u8]) { + match self { + Self::Dir(dir) => serve_from_dir(stream, request, dir), + _ => { + let head = format!( + "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", + self.len(), + ); + if stream.write_all(head.as_bytes()).is_ok() { + // Surface a truncated send instead of silently closing the + // connection mid-body, which the client would only see as an + // opaque short read. + if let Err(err) = self.write_to(stream) { + eprintln!(" host http server: body write failed: {err}"); + } + } + } } } @@ -153,7 +178,102 @@ impl HostHttpBody { } Ok(()) } + Self::Dir(_) => Ok(()), + } + } +} + +/// Resolve a `[host_http_server] dir` value to a deterministic absolute path and +/// fail fast if it does not exist or cannot be read. +/// +/// `dir` is interpreted as workspace-root-relative (matching the `qemu-*.toml` +/// comment) when given as a relative path, so the served directory does not +/// depend on the process CWD. Absolute paths are used verbatim. A missing or +/// unreadable directory is a hard error here rather than a silent empty `200` +/// index that would leak into the guest as a confusing install failure. +fn resolve_serve_dir(dir: &str) -> anyhow::Result { + let raw = Path::new(dir); + let resolved = if raw.is_absolute() { + raw.to_path_buf() + } else { + crate::context::workspace_root_path() + .context("failed to resolve workspace root for host HTTP server `dir`")? + .join(raw) + }; + + let canonical = resolved.canonicalize().with_context(|| { + format!( + "host HTTP server `dir` does not exist or is not accessible: {}", + resolved.display() + ) + })?; + if !canonical.is_dir() { + bail!( + "host HTTP server `dir` is not a directory: {}", + canonical.display() + ); + } + // Confirm the directory is actually readable now, so a hard error surfaces at + // startup instead of an empty autoindex once the guest starts requesting it. + fs::read_dir(&canonical).with_context(|| { + format!( + "host HTTP server `dir` is not readable: {}", + canonical.display() + ) + })?; + Ok(canonical) +} + +/// Parse the request line, then serve an autoindex (for `/`) or a single file. +/// Only flat filenames are served; any `/` or `..` in the name yields 404, so a +/// request cannot escape `dir`. +fn serve_from_dir(stream: &mut impl Write, request: &[u8], dir: &Path) { + let head = String::from_utf8_lossy(request); + let raw_path = head + .lines() + .next() + .and_then(|line| line.split_whitespace().nth(1)) + .unwrap_or("/"); + let path = raw_path.split('?').next().unwrap_or("/"); + + if path == "/" || path.is_empty() { + let mut names: Vec = match fs::read_dir(dir) { + Ok(entries) => entries + .flatten() + .filter(|e| e.file_type().map(|t| t.is_file()).unwrap_or(false)) + .map(|e| e.file_name().to_string_lossy().into_owned()) + .collect(), + Err(_) => Vec::new(), + }; + names.sort(); + let mut html = String::from("\n"); + for name in &names { + html.push_str(&format!("{name}\n")); } + html.push_str("\n"); + write_dir_response(stream, "200 OK", "text/html", html.as_bytes()); + return; + } + + let name = path.trim_start_matches('/'); + if name.is_empty() || name.contains('/') || name.contains("..") { + write_dir_response(stream, "404 Not Found", "text/plain", b"not found\n"); + return; + } + match fs::read(dir.join(name)) { + Ok(bytes) => write_dir_response(stream, "200 OK", "application/octet-stream", &bytes), + Err(_) => write_dir_response(stream, "404 Not Found", "text/plain", b"not found\n"), + } +} + +fn write_dir_response(stream: &mut impl Write, status: &str, content_type: &str, body: &[u8]) { + let head = format!( + "HTTP/1.1 {status}\r\nContent-Type: {content_type}\r\nContent-Length: {}\r\nConnection: \ + close\r\n\r\n", + body.len(), + ); + if stream.write_all(head.as_bytes()).is_ok() { + let _ = stream.write_all(body); } } @@ -186,6 +306,7 @@ mod tests { body: "unused".to_string(), body_size: Some(7), body_byte: b'X', + dir: None, }; let _guard = HostHttpServerGuard::start(&config, "generated-body").unwrap(); @@ -216,6 +337,7 @@ mod tests { body: "first".to_string(), body_size: None, body_byte: b'A', + dir: None, }; let first_guard = HostHttpServerGuard::start(&config, "first").unwrap(); @@ -243,6 +365,68 @@ mod tests { second_thread.join().unwrap(); } + #[test] + fn relative_dir_resolves_against_workspace_root() { + // The pip-uv online wheel index is a committed, workspace-root-relative + // directory; resolving it must yield an absolute path under the workspace + // root regardless of the test process CWD. + let rel = "apps/starry/pip-uv/online-index"; + let resolved = resolve_serve_dir(rel).expect("relative dir should resolve"); + let workspace_root = + crate::context::workspace_root_path().expect("workspace root should resolve"); + + assert!(resolved.is_absolute(), "resolved dir must be absolute"); + assert!(resolved.starts_with(&workspace_root)); + assert_eq!(resolved, workspace_root.join(rel).canonicalize().unwrap()); + + match HostHttpBody::from_config(&HostHttpServerConfig { + bind: "127.0.0.1".to_string(), + port: 0, + body: "unused".to_string(), + body_size: None, + body_byte: b'a', + dir: Some(rel.to_string()), + }) + .expect("from_config should accept the committed relative dir") + { + HostHttpBody::Dir(path) => assert_eq!(path, resolved), + other => panic!("expected Dir body, got {other:?}"), + } + } + + #[test] + fn absolute_dir_is_used_verbatim() { + let temp = std::env::temp_dir(); + let resolved = resolve_serve_dir(&temp.to_string_lossy()) + .expect("an existing absolute dir should resolve"); + assert_eq!(resolved, temp.canonicalize().unwrap()); + } + + #[test] + fn missing_dir_errors_at_start() { + let config = HostHttpServerConfig { + bind: "127.0.0.1".to_string(), + port: free_local_port(), + body: "unused".to_string(), + body_size: None, + body_byte: b'a', + // A path that cannot exist under the workspace root. + dir: Some("apps/starry/pip-uv/__definitely_missing_index__".to_string()), + }; + + let err = match HostHttpServerGuard::start(&config, "missing-dir") { + Ok(_) => { + panic!("a missing host HTTP dir must be a hard error, not a silent empty index") + } + Err(err) => err, + }; + let message = format!("{err:#}"); + assert!( + message.contains("does not exist or is not accessible"), + "unexpected error message: {message}" + ); + } + fn free_local_port() -> u16 { TcpListener::bind("127.0.0.1:0") .unwrap()