Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions apps/starry/pip/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# pip 测试.

在 StarryOS (Alpine rootfs) 上验证 Python pip 的常用操作。

## 测试内容

25 个阶段,覆盖日常 pip 使用场景:

- venv 创建/激活/销毁
- pip install / uninstall / upgrade / --dry-run
- pip list / show / freeze / check
- pip install -r requirements.txt
- pip install from local directory / local wheel
- pip wheel / pip download
- pip cache dir / info / purge
- pip install -e (editable)
- pip list --format=json / --outdated
- pip config / pip debug

## 运行测试
Comment thread
ZR233 marked this conversation as resolved.

```bash
cargo xtask starry app run -t pip
```

## 在 Linux 主机上快速验证测试脚本

```bash
sh apps/starry/pip/test_pip.sh
```

## 工作原理

1. `prebuild.sh` 被 xtask 框架调用,通过 `qemu-user-static` 在 staging rootfs 中执行 Alpine 原生 `apk` 安装 `python3 py3-pip`(使用 `--no-scripts` 跳过 busybox trigger)
2. 通过 `readelf` 解析运行时依赖并复制到 overlay
3. `test_pip.sh` 被复制到 overlay 的 `/usr/bin/`
4. overlay 被注入到 rootfs 镜像中
5. QEMU 启动后执行 `/usr/bin/test_pip.sh`

## 判定标准

- **通过**: 输出匹配 `STARRY_PIP_TESTS_PASSED`
- **失败**: 输出匹配 `STARRY_PIP_STAGE_.*_FAILED`,或出现 panic / page fault / segmentation fault
- **超时**: 600 秒
11 changes: 11 additions & 0 deletions apps/starry/pip/build-x86_64-unknown-none.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
target = "x86_64-unknown-none"
env = { AX_IP = "10.0.2.15", AX_GW = "10.0.2.2" }
log = "Warn"
features = [
"ax-hal/x86-pc",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

所有其他 Starry app(git、openssh、redis、codex-cli、openrc)的 build config 均使用 features = ["qemu"],包括同样使用 virtio-blk/net 的 redis。额外列出 ax-hal/x86-pcax-driver/plat-staticax-driver/virtio-blkax-driver/virtio-net 与其他 app 不一致。

如果 "qemu" feature 不足以启用 virtio 驱动,这应该是框架层面的问题。如果确实需要这些 feature,请在 PR body 中说明原因。

建议改为:

features = ["qemu"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

建议简化为 features = ["qemu"] 以与其他 app(redis、git 等)保持一致。当前写法功能上正确,不阻塞合入。

"qemu",
"ax-driver/plat-static",
"ax-driver/virtio-blk",
"ax-driver/virtio-net",
]
plat_dyn = false
178 changes: 178 additions & 0 deletions apps/starry/pip/prebuild.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,178 @@
#!/usr/bin/env bash
set -euo pipefail

app_dir="${STARRY_APP_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)}"
arch="${STARRY_ARCH:-x86_64}"
base_rootfs="${STARRY_BASE_ROOTFS:-}"
staging_root="${STARRY_STAGING_ROOT:-}"
overlay_dir="${STARRY_OVERLAY_DIR:-}"
apk_cache="${STARRY_WORKSPACE:-$(cd "$app_dir/../../.." && pwd)}/target/pip-apk-cache"

require_env() {
local name="$1"
local value="$2"
if [[ -z "$value" ]]; then
echo "error: $name is required" >&2
exit 1
fi
}

ensure_host_packages() {
local missing=()

command -v debugfs >/dev/null 2>&1 || missing+=(e2fsprogs)
command -v install >/dev/null 2>&1 || missing+=(coreutils)
command -v readelf >/dev/null 2>&1 || missing+=(binutils)

case "$arch" in
aarch64) command -v qemu-aarch64-static >/dev/null 2>&1 || missing+=(qemu-user-static) ;;
riscv64) command -v qemu-riscv64-static >/dev/null 2>&1 || missing+=(qemu-user-static) ;;
x86_64) command -v qemu-x86_64-static >/dev/null 2>&1 || missing+=(qemu-user-static) ;;
loongarch64) command -v qemu-loongarch64-static >/dev/null 2>&1 || missing+=(qemu-user-static) ;;
esac

if [[ ${#missing[@]} -eq 0 ]]; then
return
fi

if ! command -v apt-get >/dev/null 2>&1; then
echo "error: missing required host packages and apt-get is unavailable: ${missing[*]}" >&2
exit 1
fi

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

command -v apk 在 CI 容器(ghcr.io/rcore-os/tgoskits-container:latest,Ubuntu 24.04)中返回非零,因此此处 missing+=(apk-tools)

但下方的 apt-get install -y --no-install-recommends apk-tools 在 Ubuntu 24.04 中无法定位该包(exit code 100),set -euo pipefail 会使脚本立即失败。

建议改用 qemu-x86_64-static -L $staging_root $staging_root/usr/bin/apk --no-scripts ... 方式安装,不依赖宿主机 apk。容器中已有 qemu-x86_64-static


echo "installing missing host packages: ${missing[*]}"
apt-get update
apt-get install -y --no-install-recommends "${missing[@]}"
}

extract_base_rootfs() {
debugfs -R "rdump / $staging_root" "$base_rootfs"
}

install_pip_packages() {
local qemu_runner
case "$arch" in
aarch64) qemu_runner="qemu-aarch64-static" ;;
riscv64) qemu_runner="qemu-riscv64-static" ;;
x86_64) qemu_runner="qemu-x86_64-static" ;;
loongarch64) qemu_runner="qemu-loongarch64-static" ;;
*) echo "error: unsupported arch: $arch" >&2; exit 1 ;;
esac

if ! command -v "$qemu_runner" >/dev/null 2>&1; then
echo "error: $qemu_runner not found" >&2
exit 1
fi

# Copy host DNS config so apk can resolve hostnames inside qemu-user.
if [[ -f /etc/resolv.conf ]]; then
cp /etc/resolv.conf "$staging_root/etc/resolv.conf"
fi

mkdir -p "$apk_cache"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

宿主机没有 apk,此处会调用失败(exit code 1)。ensure_host_packages() 也无法补救(见上方 line 34 注释)。

正确做法:

"$qemu_runner" -L "$staging_root" \
    "$staging_root/usr/bin/apk" \
    --root / \
    --cache-dir "$apk_cache" \
    --update-cache --no-progress --no-scripts \
    add python3 py3-pip

--no-scripts 是关键,可防止 busybox trigger 运行 unshare 导致 Invalid argument 报错。

echo "[pip prebuild] installing python3 and py3-pip via qemu-user apk..."
"$qemu_runner" -L "$staging_root" \
"$staging_root/sbin/apk" \
--root / \
--cache-dir "$apk_cache" \
--update-cache \
--no-progress \
--no-scripts \
add python3 py3-pip
}

copy_file_to_overlay() {
local guest_path="$1"
local mode="$2"
local source="$staging_root${guest_path}"
local target="$overlay_dir${guest_path}"

if [[ ! -e "$source" ]]; then
echo "error: missing guest file after package install: $guest_path" >&2
exit 1
fi

if [[ -L "$source" ]]; then
source="$(readlink -f "$source")"
fi

install -Dm"$mode" "$source" "$target"
}

find_library_path() {
local library="$1"
local dir

for dir in lib usr/lib usr/local/lib; do
if [[ -e "$staging_root/$dir/$library" ]]; then
printf '/%s/%s\n' "$dir" "$library"
return 0
fi
done

return 1
}

copy_runtime_dependencies() {
local pending=("$@")
local seen=" "
local guest_path library

while [[ ${#pending[@]} -gt 0 ]]; do
guest_path="${pending[0]}"
pending=("${pending[@]:1}")

if [[ "$seen" == *" $guest_path "* ]]; then
continue
fi
seen+="$guest_path "

while IFS= read -r library; do
local library_path
if ! library_path="$(find_library_path "$library")"; then
continue
fi
copy_file_to_overlay "$library_path" 0644
pending+=("$library_path")
done < <(
readelf -d "$staging_root$guest_path" 2>/dev/null |
sed -n 's/.*Shared library: \[\(.*\)\].*/\1/p'
)
done
}

populate_overlay() {
copy_file_to_overlay /usr/bin/python3 0755
copy_file_to_overlay /usr/bin/pip3 0755

if [[ -e "$staging_root/usr/bin/pip" ]]; then
copy_file_to_overlay /usr/bin/pip 0755
fi

copy_runtime_dependencies /usr/bin/python3

# Copy Python standard library
local pyver
pyver="$(ls "$staging_root/usr/lib/python3"* -d 2>/dev/null | head -1 | xargs basename)"
if [[ -n "$pyver" && -d "$staging_root/usr/lib/$pyver" ]]; then
mkdir -p "$overlay_dir/usr/lib/$pyver"
cp -a "$staging_root/usr/lib/$pyver/." "$overlay_dir/usr/lib/$pyver/"
fi

# Copy pip site-packages
if [[ -d "$staging_root/usr/lib/python3" ]]; then
mkdir -p "$overlay_dir/usr/lib/python3"
cp -a "$staging_root/usr/lib/python3/." "$overlay_dir/usr/lib/python3/"
fi

install -Dm0755 "$app_dir/test_pip.sh" "$overlay_dir/usr/bin/test_pip.sh"
}

require_env STARRY_BASE_ROOTFS "$base_rootfs"
require_env STARRY_STAGING_ROOT "$staging_root"
require_env STARRY_OVERLAY_DIR "$overlay_dir"

ensure_host_packages
extract_base_rootfs
install_pip_packages
populate_overlay
29 changes: 29 additions & 0 deletions apps/starry/pip/qemu-x86_64.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
args = [
"-machine",
"q35",
"-nographic",
"-m",
"1G",
"-device",
"virtio-blk-pci,drive=disk0",
"-drive",
"id=disk0,if=none,format=raw,file=${workspace}/tmp/axbuild/rootfs/rootfs-x86_64-pip.img",
"-device",
"virtio-net-pci,netdev=net0",
"-netdev",
"user,id=net0",
"-snapshot",
]
uefi = false
to_bin = false
shell_prefix = "root@starry:"
Comment thread
ZR233 marked this conversation as resolved.
shell_init_cmd = "/usr/bin/test_pip.sh"
success_regex = ["(?m)^STARRY_PIP_TESTS_PASSED\\s*$"]
fail_regex = [
"(?i)\\bpanic(?:ked)?\\b",
"(?i)page fault",
"(?i)segmentation fault",
"No such process",
"STARRY_PIP_STAGE_.*_FAILED",
]
timeout = 600
Loading
Loading