From caa785724282adc665f11f18e134834e2bcd3188 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sat, 8 Aug 2026 07:34:34 +0800 Subject: [PATCH] ci: a three-state contract for verification scripts, and a closure check that runs Split out of the single large PR so its CI cost matches its risk: this touches no pkgs/ file at all, so the per-package install tests skip entirely. THE CONTRACT 0 proven / 1 broken / 2 inconclusive / 3 could-not-run-here, written into .agents/tools/README.md and obeyed by eleven scripts. A caller must map 3 to "not run" and must never fold it into a pass. It exists because skip() in selfcontained-check.sh used to exit 0, and 0 is what the caller reads as "S1-S4 pass" -- so a machine without bwrap printed a tick for a check that ran nothing. Applying the contract turned up two more of the same: verify-host-link's check 4 and verify-stack's app-RPATH cell both printed a pass with no patchelf present. BUILD INPUTS, NOT JUST OUTPUTS build-in-subos.sh verified the produced payload and never what the build linked against, so a build configured entirely against /usr passed "no host references". The sealed-bwrap approach was tried first and does not work here (the subos has no POSIX userland, so clean and dirty builds fail alike); configure-log parsing instead, filtered to resolved records. It reproduces both known instances -- the host libzstd in the LLVM tree behind our mesa, and a constructed case -- and found a third: the --deps .pc rewrite patched only prefix=, leaving libdir=/usr/lib in 40 files. DECLARED DEPS vs REAL DT_NEEDED dep-closure-check.sh, run per package from posix-test.sh. D1: a soname provided by an installed package must be declared DIRECTLY, because a transitive dep does not put its libdir in the closure. D2: if the payload uses our interpreter, a soname with no provider is fatal -- our ld.so has a cache path that exists on no machine, so there is no host fallback behind it. Strictness follows the payload's own state (interpreter, or an RPATH into xpkgs), so host-integrated packages are reported and not failed. A check that fires on correct recipes gets switched off. The tooling step matters as much as the script: linux-install-test is a separate job from linux-test, so lua5.4 was absent and the check exited 3 on every run -- "not evaluated", job green. Measured after fixing it: 20 passes, 6 not-evaluated, 0 failures across a 25-package changed set. Two uninstall tolerances, each keyed on its exact diagnostic rather than on the package type alone: a type="config" package registers no xvm version (11 of the 12 in this index call xvm.add zero times), and a bump PR cannot remove the xlings it is running. The latter is new since #543 -- overlaying the recipe into the index made it install as xim:xlings rather than local:xlings, which the running-binary guard matches. Left alone it is red on every future release bump. Also stops posix-test.sh from printing hundreds of "ignored null byte in input" warnings: it read ELF magic with head -c4 inside a command substitution while walking the whole store. --- .agents/tools/README.md | 84 ++++ .agents/tools/build-llvm-subpkg.sh | 12 +- .agents/tools/build-llvm-tools.sh | 9 + .agents/tools/build-musl.sh | 6 +- .agents/tools/graphics/build-glibc.sh | 5 +- .agents/tools/graphics/build-in-subos.sh | 361 +++++++++++++++++- .agents/tools/graphics/build-libllvm.sh | 12 +- .agents/tools/graphics/collect-matrix.md | 108 ++++++ .agents/tools/graphics/publish.sh | 21 +- .agents/tools/graphics/selfcontained-check.sh | 8 +- .agents/tools/graphics/tiers.sh | 8 +- .agents/tools/graphics/verify-host-link.sh | 26 +- .agents/tools/graphics/verify-stack.sh | 77 +++- .agents/tools/verify-toolchain.sh | 16 +- .github/scripts/dep-closure-check.sh | 262 +++++++++++++ .github/scripts/posix-test.sh | 96 ++++- .github/workflows/ci-test.yml | 19 +- 17 files changed, 1088 insertions(+), 42 deletions(-) create mode 100644 .agents/tools/README.md create mode 100644 .agents/tools/graphics/collect-matrix.md create mode 100755 .github/scripts/dep-closure-check.sh diff --git a/.agents/tools/README.md b/.agents/tools/README.md new file mode 100644 index 00000000..1a4e8511 --- /dev/null +++ b/.agents/tools/README.md @@ -0,0 +1,84 @@ +# `.agents/tools/` — the exit-code contract + +Every script here reports through its exit code. There are four states, and a +caller has to handle all four: + +``` +0 = proven the assertion was evaluated and holds +1 = broken the assertion was evaluated and fails +2 = inconclusive the check ran but cannot distinguish its cases +3 = could-not-run this machine cannot exercise the check at all +``` + +**A caller MUST map 3 to "not run". It MUST NOT fold 3 into a pass**, and it +should not fold it into a failure either — a machine without an AMD GPU has not +found a bug in radeonsi. + +## Why 3 exists + +`skip()` in `graphics/selfcontained-check.sh` used to `exit 0`, and 0 is what +the caller reads as "S1-S4 pass". A machine without bwrap, or without a compiler +inside the subos, therefore printed + +``` + ✓ empty-host self-containment S1-S4 pass +``` + +for a check that ran nothing at all. Not a weaker pass, not a partial one: the +same green tick, the same words, from a script that exited before it built the +probe. + +That is the exact bug class this tooling exists to catch, sitting inside the +tooling. Not-run and succeeded produced identical output — which is the whole +reason `verify-stack.sh` counts a third outcome, and the reason +`selfcontained-check.sh` runs a control container before it is willing to blame +the closure. + +The reason it survived is worth keeping too: `verify-stack.sh` happens to probe +`bwrap` itself before calling the script, so the one skip anybody exercised was +shadowed by a caller-side guard. Every other skip path was live, and each of +them printed the tick. + +## What each state means in practice + +**2 and 3 are different questions.** 2 says the check ran and its result does +not separate the cases it was written to separate — `selfcontained-check.sh` +returns it when the *control* container fails, because then a sealed-container +failure proves nothing about self-containment and "S1: closure incomplete" would +name the wrong cause. 3 says nothing was measured, and says why: no bwrap, no +GPU, no compiler, no `/dev/dxg`, no payload to look at. + +A check that cannot tell which it is should return 2. A wrong cause is worse +than no cause, because someone acts on it. + +**1 is a claim about the subject, not about the environment.** A missing +`patchelf` is not a broken package. A missing subos is not an unsealed closure. +Build scripts follow the same split: 3 when the build never started (no subos, +no cmake, wrong architecture), 1 only when it started and broke — that is the +only status worth opening a log for. + +An argument error (unknown flag, missing required option) sits outside the four: +nothing was checked and no verdict exists. Those exit 2, which is the safe +direction — a caller that follows the contract will not read it as a pass. + +## Obligations on a caller + +- Match on the code. `if script; then ok; else bad; fi` is the caller half of + the same bug: it renders 2 and 3 as red. `verify-stack.sh` did exactly this to + `verify-host-link.sh`, so a machine with no host compiler got a failing NVIDIA + cell for a probe that was never built. +- Pass the child's code through rather than rewriting it. `|| return 1` in a + driver loop erases the distinction the child went to the trouble of making. +- Carry the reason, not just the state. `verify-stack.sh`'s `na()` requires one, + because "not applicable here" with no reason is indistinguishable from "nobody + implemented it". +- Never let an absent tool answer the question. `patchelf --print-rpath` on a + machine with no patchelf prints nothing, and "no host directories on the + RPATH" is what nothing looks like. Probe for the tool; route to not-run. + +## Reporting coverage + +`verify-stack.sh` is the union point: it runs the whole matrix on one machine +and marks every cell it could not exercise. No single machine covers the matrix, +so the not-run cells are a recruitment list rather than an embarrassment. See +`graphics/collect-matrix.md` for what to run and where to send it. diff --git a/.agents/tools/build-llvm-subpkg.sh b/.agents/tools/build-llvm-subpkg.sh index 23302cd8..dad3fc97 100755 --- a/.agents/tools/build-llvm-subpkg.sh +++ b/.agents/tools/build-llvm-subpkg.sh @@ -30,6 +30,9 @@ set -euo pipefail die() { echo "error: $*" >&2; exit 1; } log() { echo ">> $*" >&2; } +# 3 = this machine cannot do the work; 1 = it tried and the work is wrong. +# Contract: .agents/tools/README.md. +skip() { echo "SKIP: $*" >&2; exit 3; } IN="" PKG="" VERSION="" PLATFORM="" ARCH="" OUT="$PWD" FORMATS="" while [ $# -gt 0 ]; do @@ -208,8 +211,10 @@ do_libcxx() { cxxdir=$(dirname "$(find "$DEST/lib" -maxdepth 2 -name 'libc++.so.1' 2>/dev/null | head -1)") [ -n "$cxxdir" ] && [ -d "$cxxdir" ] \ || die "libatomic: cannot locate libc++ triple lib dir under $DEST/lib" + # 3, not 1: the carve is fine, this host just has no GCC to source + # libatomic from. Producing the package here is impossible, not wrong. command -v gcc >/dev/null 2>&1 \ - || die "libatomic: gcc not on build host (needed to source libatomic for self-containment)" + || skip "libatomic: gcc not on build host (needed to source libatomic for self-containment)" atomic_real=$(readlink -f "$(gcc -print-file-name=libatomic.so.1)" 2>/dev/null) [ -n "$atomic_real" ] && [ -e "$atomic_real" ] \ || die "libatomic: 'gcc -print-file-name=libatomic.so.1' did not resolve to a real file" @@ -239,6 +244,11 @@ esac # --- macOS self-containment check (Mach-O LC_LOAD_DYLIB) ------------------- if [ "$PLATFORM" = "macosx" ] && [ -d "$DEST/bin" ]; then + # Without this probe a missing python3 exits 127 through the `|| die` below + # and the carve reports "self-containment check failed" -- a specific and + # entirely wrong claim about the bundle. The Mach-O reader never ran. + command -v python3 >/dev/null 2>&1 \ + || skip "no python3 — the macOS self-containment check was NOT performed on this bundle" log "verifying macOS binaries are self-contained (system-only dylibs) ..." # only real Mach-O files (skip symlinks) mapfile -t MACHO < <(find "$DEST/bin" -type f) diff --git a/.agents/tools/build-llvm-tools.sh b/.agents/tools/build-llvm-tools.sh index 1f4fe901..e6db1349 100755 --- a/.agents/tools/build-llvm-tools.sh +++ b/.agents/tools/build-llvm-tools.sh @@ -30,6 +30,9 @@ set -euo pipefail die() { echo "error: $*" >&2; exit 1; } log() { echo ">> $*" >&2; } +# 3 = this machine cannot do the work; 1 = it tried and the work is wrong. +# Contract: .agents/tools/README.md. +skip() { echo "SKIP: $*" >&2; exit 3; } IN="" VERSION="" PLATFORM="" ARCH="" OUT="$PWD" FMT="tar.xz" while [ $# -gt 0 ]; do @@ -108,6 +111,12 @@ log " + lib/clang/${RESVER}/include ($(du -sh "$WORK/$BUNDLE/lib/clang/${RESVE # --- macOS self-containment check (Mach-O LC_LOAD_DYLIB) ------------------- if [ "$PLATFORM" = "macosx" ]; then + # Without this probe a missing python3 exits 127 through the `|| die`, and + # the carve reports "self-containment check failed" -- a specific, + # actionable and entirely wrong claim about the bundle. The Mach-O reader + # never ran. + command -v python3 >/dev/null 2>&1 \ + || skip "no python3 — the macOS self-containment check was NOT performed on this bundle" log "verifying macOS binaries are self-contained (system-only dylibs) ..." python3 - "$WORK/$BUNDLE/bin"/* <<'PY' || die "self-containment check failed" import struct, sys diff --git a/.agents/tools/build-musl.sh b/.agents/tools/build-musl.sh index 618a6ae7..4b5010bc 100755 --- a/.agents/tools/build-musl.sh +++ b/.agents/tools/build-musl.sh @@ -36,8 +36,12 @@ DIST="$WORK/dist" log() { echo "[build:$NAME] $*"; } fail() { echo "[build:$NAME] FAIL: $*" >&2; exit 1; } +# 3, not 1. Exit-code contract: .agents/tools/README.md. "this machine is the +# wrong architecture" is not a failed build, and a matrix driver that reads 1 as +# a build failure would open a bug against musl for a working script. +skip() { echo "[build:$NAME] SKIP: $*" >&2; exit 3; } -[[ "$ARCH" == "x86_64" ]] || fail "this script only builds the x86_64 payload (host is $ARCH)" +[[ "$ARCH" == "x86_64" ]] || skip "this script only builds the x86_64 payload (host is $ARCH)" rm -rf "$STAGE"; mkdir -p "$SRC" "$STAGE" "$DIST" diff --git a/.agents/tools/graphics/build-glibc.sh b/.agents/tools/graphics/build-glibc.sh index 4f78e07e..54d1ce37 100644 --- a/.agents/tools/graphics/build-glibc.sh +++ b/.agents/tools/graphics/build-glibc.sh @@ -33,8 +33,11 @@ DIST="$WORK/dist" log() { echo "[gfx-build:$NAME] $*"; } fail() { echo "[gfx-build:$NAME] FAIL: $*" >&2; exit 1; } +# 3, not 1: nothing was built because there was nowhere to build it. See the +# exit-code contract in .agents/tools/README.md. +skip() { echo "[gfx-build:$NAME] SKIP: $*" >&2; exit 3; } -[[ -d "$SUBOS" ]] || fail "subos '$SUBOS_NAME' not found" +[[ -d "$SUBOS" ]] || skip "subos '$SUBOS_NAME' not found — xlings subos new $SUBOS_NAME" rm -rf "$STAGE"; mkdir -p "$SRC" "$STAGE" "$DIST" # Same shape as the published 2.39, so a home holding both resolves them the diff --git a/.agents/tools/graphics/build-in-subos.sh b/.agents/tools/graphics/build-in-subos.sh index 82b588d7..099a75d2 100755 --- a/.agents/tools/graphics/build-in-subos.sh +++ b/.agents/tools/graphics/build-in-subos.sh @@ -13,22 +13,45 @@ # So every build runs with the subos supplying the compiler, the sysroot and # the libraries, and the result is checked before it is allowed to ship: # -# * no DT_NEEDED that resolves outside the subos or the package itself # * no RPATH/RUNPATH naming a host path # * no absolute host path baked into a .pc, .la or config script # +# and, since 2026-08-08, the other half of the same question: +# +# * nothing the build was CONFIGURED against came from the host either +# +# That last one is not a refinement of the first two, it is the gap they left. +# See "the check on the INPUTS" below for the two shipped packages that walked +# straight through the payload check. +# +# KNOWN GAP. This list used to open with "no DT_NEEDED that resolves outside +# the subos or the package itself". No such check has ever existed in this file +# — grep it. It is the one that would catch an autotools package that picked up +# a host library through a bare `-lfoo`: no absolute path for the input check +# to see, and a bare SONAME for the payload check to shrug at. Recorded here +# rather than quietly deleted, because it is the complement of the input check +# and not a duplicate of it. +# # That check is the reason this script exists rather than a README saying # "build it in the subos". A leaked host path does not fail the build; it fails # months later on someone else's machine. # +# Exit codes (the contract in .agents/tools/README.md): +# 0 proven — built, and neither its inputs nor its payload reach the host +# 1 broken — a host reference, in the inputs or in the payload +# 2 inconclusive — could not read what the build linked against +# 3 could not be exercised here — a precondition this machine does not meet +# # Usage: # build-in-subos.sh --name libXau --version 1.0.11 \ # --url https://.../libXau-1.0.11.tar.xz \ # [--system autotools|meson|cmake] [--deps 'xorgproto libX11'] \ # [-- ] +# +# build-in-subos.sh --check-inputs # just the input check set -uo pipefail -NAME= VERSION= URL= SYSTEM=auto DEPS= EXTRA=() +NAME= VERSION= URL= SYSTEM=auto DEPS= EXTRA=() CHECK_INPUTS= while [[ $# -gt 0 ]]; do case "$1" in --name) NAME="$2"; shift 2 ;; @@ -36,12 +59,20 @@ while [[ $# -gt 0 ]]; do --url) URL="$2"; shift 2 ;; --system) SYSTEM="$2"; shift 2 ;; --deps) DEPS="$2"; shift 2 ;; + # Run the input check alone, against a build tree that already exists. + # + # A check that can only be exercised by rerunning a two-hour mesa build + # is a check nobody reruns, and both builds that motivated it are + # already sitting in $XLINGS_GFX_WORK/src. Being able to point it at + # one of those is what makes its verdict evidence rather than a claim. + --check-inputs) CHECK_INPUTS="$2"; NAME="${NAME:-inputs}"; shift 2 ;; --) shift; EXTRA=("$@"); break ;; *) echo "unknown argument: $1" >&2; exit 2 ;; esac done -[[ -n "$NAME" && -n "$VERSION" && -n "$URL" ]] || { +[[ -n "$CHECK_INPUTS" || ( -n "$NAME" && -n "$VERSION" && -n "$URL" ) ]] || { echo "usage: $0 --name N --version V --url U [--system auto|autotools|meson|cmake] [--deps '...'] [-- args]" >&2 + echo " $0 --check-inputs " >&2 exit 2 } @@ -54,8 +85,278 @@ SRC="$WORK/src" log() { echo "[gfx-build:$NAME] $*"; } fail() { echo "[gfx-build:$NAME] FAIL: $*" >&2; exit 1; } +# 2 and 3 are not decoration. `fail` says the package is broken; these two say +# the check did not get to run, which is a different thing to act on, and +# neither of them is 0. +inconclusive() { echo "[gfx-build:$NAME] INCONCLUSIVE: $*" >&2; exit 2; } +skip() { echo "[gfx-build:$NAME] SKIP: $*" >&2; exit 3; } + +# ══ the check on the INPUTS ═════════════════════════════════════════════ +# +# Everything else in this script inspects the PAYLOAD. That is necessary and it +# is not sufficient, and two shipped packages are the proof: +# +# * mesa 25.0.7.1 cannot have been built against the index's `llvm` package. +# That package ships bin/llvm-config and none of the ~40 component .a files +# and no LLVM API headers, so neither of meson's two detection methods can +# succeed against it. The published mesa was built against the HOST's LLVM. +# The payload check reported "no host references" — correctly, on its own +# terms: a static .a leaves no DT_NEEDED at all, and a shared libLLVM +# leaves a bare SONAME with no path in it. +# +# * building LLVM here on 2026-08-07, cmake resolved zstd to the host's +# /usr/lib/x86_64-linux-gnu/libzstd.so. There is no zstd package in this +# home at all. Nothing in the build said so; a runtime failure on another +# machine did. +# +# Both builds were configured against something that will not exist on the +# user's machine, and the files they produced said nothing about it. So this +# asks the other half of the question: what went IN. +# +# ── why this reads logs instead of sealing the container ──────────────── +# +# The check you would rather have is the one selfcontained-check.sh uses for +# runtime: run the build inside bwrap with no /usr bound, so the host cannot be +# consumed because it is not there. Measured on this machine 2026-08-08, that +# cannot work for a BUILD, for two independent reasons: +# +# * the subos carries no POSIX userland. sh, sed, grep, awk, m4, install, ln, +# mkdir, cp, env, make, pkg-config — every one of those resolves to /usr on +# a gfxbuild subos with the full graphics stack installed. `./configure` is +# a /bin/sh script and `/bin/meson` begins `#!/bin/sh`; with /usr +# gone, neither can start. +# +# * the subos's own toolchain entries are xlings shims that dispatch through +# xvm, and dispatching needs the host. Measured, with $XLINGS_HOME bound +# and `/bin/gcc --version` as the whole command: +# +# /usr + /bin + /lib64 present → rc 0, prints the version +# any subset of those → rc 127, nothing on stdout or stderr +# +# The real gcc under data/xpkgs/ runs sealed perfectly well. It is the shim +# layer that does not, so a sealed build cannot reach `gcc --version`. +# +# A sealed build would therefore not measure host leakage, it would measure +# whether bwrap started — and it would fail identically for a clean package and +# a dirty one, which is the exact failure mode this whole file exists to avoid. +# Reading what the build wrote down is weaker: it can only see inputs the build +# recorded. But what it does see is real. +# +# ── probed-and-rejected vs resolved-and-used ──────────────────────────── +# +# The reason this is a parser and not a grep for "/usr". mesa's meson-log.txt +# is 4711 lines and 841 of them contain "/usr"; flag those and the check is +# commented out inside a week. Three separate things produce that noise, and +# each needs its own rule: +# +# 1. the subos's own paths END in /usr — `/usr/lib/pkgconfig`. So a +# path only counts when it starts at the filesystem root. That alone takes +# mesa from 841 to 328. +# +# 2. of those 328: /usr/bin/pkg-config (321), /usr/bin/bison, /usr/bin/flex, +# /usr/bin/ln — host PROGRAMS the build ran. A clean autotools config.log +# is the same story: xz's resolved-variables section holds 12 /usr paths +# and all 12 are grep, sed, msgfmt, install, mkdir. Programs generate +# source and do not end up in the result. +# +# 3. and /usr/lib/gbm, /usr/lib/dri, /usr/lib/libvulkan_radeon.so — mesa's +# own INSTALL destinations, because --prefix is /usr. Under this script +# every install path looks exactly like a host path. +# +# So the rule is not "a /usr path appears" but "a path was handed to the +# compiler as a search path, or to the linker as a file": -I, -isystem, +# -iquote, -idirafter, -L, and absolute *.so/*.a arguments. Neither a program +# nor an install destination ever takes that shape. Library FILES get one +# extra test — the file must exist — because that is what separates +# /usr/lib/libvulkan_radeon.so, where mesa will install, from +# /usr/lib/x86_64-linux-gnu/libzstd.so, which cmake actually linked. +# +# The records themselves are filtered before any of that: meson logs every +# subprocess with its exit status, and a `-> 1` block is a dependency it +# probed and was told no; cmake writes *-NOTFOUND for the same thing, and +# LLVM's cache here has ZLIB_LIBRARY_DEBUG:FILEPATH=ZLIB_LIBRARY_DEBUG-NOTFOUND +# sitting two lines from a real one. +# +# Be honest about the meson half of that, though: measured against mesa's +# log, dropping the `-> 1` blocks changes the verdict not at all. 4711 lines +# become 541; the 105 lines of rejected blocks contain no -I or -L path that +# is not also in an accepted one, and the only root-/usr path in them is +# /usr/bin/pkg-config, which the shape rule already ignores. The filter is +# kept because a rejected `llvm-config --libs` or `pkg-config --cflags` can +# still print a -L before failing, and because reading a rejected probe as an +# input is the kind of wrong answer that gets a check deleted. It is not +# carrying the noise reduction — rules 1 to 3 above are. +# +# CMakeFiles/CMakeError.log is deliberately NOT read. It is by definition the +# record of probes that failed, so every path in it is one the build did not +# use; parsing it would manufacture the noise the rest of this is avoiding. +# +# ── an allowlist of ours, not a denylist of theirs ────────────────────── +# +# "/usr, /lib, /lib64" is one distro wide. The mesa tree in this work dir right +# now resolves LLVM to +# -I/tmp/.../scratchpad/llvmsrc/out/include +# -L/tmp/.../scratchpad/llvmsrc/out/lib +# a scratch build tree that is neither /usr nor XLINGS_HOME and that ships +# every bit as badly — that is the mesa/LLVM instance above, caught in the act. +# /opt/rocm, /nix/store and ~/.local are the same shape. So the test is the +# other way round: a resolved input is inside XLINGS_HOME, or inside this +# script's own work tree, or it is a leak. + +# -I/-L/-isystem/-iquote/-idirafter search paths, absolute only. +_gfx_search_paths() { # + { grep -oE -- '-(I|L)/[-A-Za-z0-9_.+@%~/]*' "$1" + grep -oE -- '-i(system|quote|dirafter)[ =]*/[-A-Za-z0-9_.+@%~/]*' "$1" + } 2>/dev/null | sed -E 's/^-(I|L|i(system|quote|dirafter)[ =]*)//' | sort -u +} +# Absolute library files named on a command line. The leading class keeps this +# from matching the tail of a longer path (a subos path ends in .so too). +_gfx_lib_files() { # + grep -oE -- '(^|[^-A-Za-z0-9_.+@%~/])/[-A-Za-z0-9_.+@%~/]*\.(so|a)(\.[0-9]+)*' "$1" \ + 2>/dev/null | sed -E 's#^[^/]*##' | sort -u +} + +check_build_inputs() { # → 0 clean, 1 host input, 2 no record read + local bd="$1" tmpd real f p line n=0 leaks=0 + tmpd="$(mktemp -d "${TMPDIR:-/tmp}/gfx-inputs.XXXXXX")" || { + echo "[gfx-build:$NAME] INCONCLUSIVE: cannot create a temp dir" >&2; return 2; } + + # Ours: XLINGS_HOME covers the subos and data/xpkgs; WORK covers src/, + # deplib/ (the rpath-patched dependency copies) and pc/ (the rewritten .pc + # files) — all three are this script's own and none of them ship. + local -a allow=("$XHOME" "$WORK") + for real in "$XHOME" "$WORK"; do + p="$(readlink -f "$real" 2>/dev/null)" + [[ -n "$p" && "$p" != "$real" ]] && allow+=("$p") + done + + # ── the records, one filtered stream per source ── + local -a used=() # what we actually read, for the pass line + local -a origin=() # parallel: which file each stream came from + local -a mode=() # parallel: flags | values + + # meson. Every subprocess is logged as + # Called: `` -> / stdout: … / ----------- + # and only rc 0 is a resolution. The argv is kept as well as the output: + # llvm-config's own -I/-L come back on stdout, but a compiler check that + # SUCCEEDED with a host include path put it in the argv. + for f in "$bd/_b/meson-logs/meson-log.txt" "$bd/meson-logs/meson-log.txt"; do + [[ -f "$f" ]] || continue + awk '/^Called: `.*` -> [0-9]+$/ { keep = /-> 0$/ } + keep { print } + /^-+$/ { keep = 0 }' "$f" > "$tmpd/meson-log" + used+=("$tmpd/meson-log"); origin+=("$f"); mode+=(flags); break + done + + # meson and cmake both emit build.ninja, and it is the stronger record: + # meson-log.txt says what was detected, build.ninja is the command line + # that will actually run. + for f in "$bd/_b/build.ninja" "$bd/build.ninja"; do + [[ -f "$f" ]] || continue + used+=("$f"); origin+=("$f"); mode+=(flags); break + done + + # cmake's cache, for the entries that hold a resolved path. -NOTFOUND is + # cmake's way of saying it looked and did not find, and CMAKE_INSTALL_* is + # a destination — neither matches the name pattern. + for f in "$bd/_b/CMakeCache.txt" "$bd/CMakeCache.txt"; do + [[ -f "$f" ]] || continue + grep -E '^[A-Za-z_][A-Za-z0-9_]*(_LIBRARY|_LIBRARIES|_LIBRARY_DIR|_LIBRARY_DIRS|_LIBRARY_PATH|_INCLUDE_DIR|_INCLUDE_DIRS|_INCLUDE_PATH)(_[A-Z0-9]+)?:[A-Z]+=' "$f" \ + | grep -vi 'NOTFOUND' | sed -E 's/^[^=]*=//' | tr ';' '\n' > "$tmpd/cmake-cache" + grep -E '^[A-Za-z_][A-Za-z0-9_]*FLAGS[A-Za-z0-9_]*:[A-Z]+=' "$f" > "$tmpd/cmake-flags" + used+=("$tmpd/cmake-cache" "$tmpd/cmake-flags") + origin+=("$f" "$f"); mode+=(values flags); break + done + + # autotools. The transcript above the cache is every probe, successful or + # not; the two sections at the end are what configure SETTLED on. Only + # variables that carry compiler or linker arguments — which is why + # `oldincludedir='/usr/include'`, an install default no build ever uses, + # does not come through: it is lowercase and it is not a flags variable. + if [[ -f "$bd/config.log" ]]; then + awk '/^## Cache variables/{s=1} /^## confdefs.h/{s=0} s' "$bd/config.log" \ + | grep -E '^[A-Za-z_][A-Za-z0-9_]*(CFLAGS|CPPFLAGS|CXXFLAGS|LDFLAGS|LIBS|INCLUDES)[A-Za-z0-9_]*=' \ + > "$tmpd/config-log" + used+=("$tmpd/config-log"); origin+=("$bd/config.log"); mode+=(flags) + fi + + if [[ ${#used[@]} -eq 0 ]]; then + echo "[gfx-build:$NAME] INCONCLUSIVE: no configure record under $bd" >&2 + echo " looked for: _b/meson-logs/meson-log.txt, _b/build.ninja," >&2 + echo " _b/CMakeCache.txt, config.log" >&2 + echo " the build's inputs are UNCHECKED. That is not a pass." >&2 + rm -rf "$tmpd"; return 2 + fi + + log "checking what the build linked AGAINST" + local i + for ((i = 0; i < ${#used[@]}; i++)); do + local stream="${used[$i]}" src="${origin[$i]}" + local -a paths=() + if [[ "${mode[$i]}" == values ]]; then + # Already one bare path per line. + mapfile -t paths < <(grep -E '^/' "$stream" | sort -u) + else + mapfile -t paths < <(_gfx_search_paths "$stream") + # A library file only counts if it is on the disk: with --prefix=/usr + # this build's own install destinations are spelled the same way as + # a host library, and they do not exist yet. + while IFS= read -r p; do + [[ -e "$p" ]] && paths+=("$p") + done < <(_gfx_lib_files "$stream") + fi + for p in "${paths[@]}"; do + [[ -n "$p" ]] || continue + n=$((n + 1)) + local ok=1 a + for a in "${allow[@]}"; do + [[ -n "$a" && ( "$p" == "$a" || "$p" == "$a"/* ) ]] && { ok=0; break; } + done + [[ $ok -eq 0 ]] && continue + line="$(grep -nF -m1 -- "$p" "$src" 2>/dev/null | cut -d: -f1)" + echo " $p" + echo " ← ${src#"$bd"/}${line:+:$line}" + leaks=$((leaks + 1)) + done + done + rm -rf "$tmpd" + + # Say what was read and how much was checked, on the way past. A pass that + # prints only "ok" is indistinguishable from a pass that examined nothing, + # and this file has already shipped that bug once. + local names="" + for ((i = 0; i < ${#origin[@]}; i++)); do names="$names ${origin[$i]#"$bd"/}"; done + log " read:$(echo "$names" | tr ' ' '\n' | sort -u | tr '\n' ' ')" + + if [[ $leaks -gt 0 ]]; then + echo "[gfx-build:$NAME] FAIL: $leaks of $n resolved input path(s) are outside" >&2 + echo " XLINGS_HOME ($XHOME) and this script's work tree ($WORK)." >&2 + echo " The build consumed something that will not exist on the user's" >&2 + echo " machine. The payload can still come out clean — that is how" >&2 + echo " mesa 25.0.7.1 shipped against the host's LLVM." >&2 + return 1 + fi + log " all $n resolved input path(s) are ours" + return 0 +} + +# --check-inputs: just that, against a tree already on disk. Before the subos +# and fetch preconditions below, because none of them apply. +if [[ -n "$CHECK_INPUTS" ]]; then + [[ -d "$CHECK_INPUTS" ]] || skip "no such build directory: $CHECK_INPUTS" + check_build_inputs "$CHECK_INPUTS" + exit $? +fi -[[ -d "$SUBOS" ]] || fail "subos '$SUBOS_NAME' not found — xlings subos new $SUBOS_NAME" +# A missing subos is not a broken package — it is this machine not being set up +# to answer the question, which is exit 3. +[[ -d "$SUBOS" ]] || skip "subos '$SUBOS_NAME' not found — xlings subos new $SUBOS_NAME" +# And patchelf, before anything relies on it. Every use of it below is +# `patchelf … 2>/dev/null || true`, which is right for a file it cannot rewrite +# and disastrous for a machine that does not have it: the payload check reads +# `patchelf --print-rpath` into an empty string and an empty RPATH is precisely +# what "clean" looks like. Absent tool, silent pass — probe for it instead. +command -v patchelf >/dev/null || skip "no patchelf (xlings install patchelf)" # STAGE is wiped, not just created: a previous run with a different --libdir # leaves its own tree here, and `make install DESTDIR=` only adds. The stale # copy then rides into the payload and ships two layouts of the same library. @@ -122,7 +423,7 @@ export CPPFLAGS="-I$SUBOS/usr/include" export LDFLAGS="-L$SUBOS/lib -L$SUBOS/usr/lib -Wl,-rpath-link,$SUBOS/usr/lib -Wl,-rpath-link,$SUBOS/lib -Wl,-rpath,\$ORIGIN" export CC="$SUBOS/bin/gcc" export CXX="$SUBOS/bin/g++" -[[ -x "$CC" ]] || fail "no gcc in the subos — xlings install gcc" +[[ -x "$CC" ]] || skip "no gcc in the subos — xlings install gcc" # So the binaries the BUILD runs can actually run. # @@ -198,7 +499,7 @@ BUILD_ENV=() for dep in $DEPS; do depdir="$(ls -d "$XHOME"/data/xpkgs/*-"$dep"/*/ 2>/dev/null | head -1)" if [[ -z "$depdir" ]]; then - fail "--deps $dep: not installed in $XHOME (xlings install $dep)" + skip "--deps $dep: not installed in $XHOME (xlings install $dep)" fi log " dep $dep -> ${depdir#"$XHOME"/data/xpkgs/}" # BOTH pkgconfig locations, and the second one is not a nicety. @@ -227,11 +528,41 @@ for dep in $DEPS; do # # The payload is not touched: it is shared between subos and read-only # as far as a build is concerned. + # + # `prefix=` alone is not enough, and that took the input check below to + # notice. A payload built by THIS script was configured with + # --prefix=/usr --libdir=/usr/lib, so its .pc reads + # + # prefix= + # libdir=/usr/lib ← absolute, does not use ${prefix} + # includedir=${prefix}/include + # + # and pkg-config duly answers `-L/usr/lib -lelf`. mesa's build was + # handed a host library search path for elfutils, libxcb, libX11, + # libXext, libXi, libXrender, libXcursor, libxshmfence and the rest — + # 40 of the .pc copies in the work tree carried it. It linked the + # right libraries anyway, but only because Debian puts them in + # /usr/lib/x86_64-linux-gnu; on Arch, where /usr/lib IS the library + # directory, that link picks up the host's copy and nothing says so. + # The payload check cannot see it either: it wants `/usr/lib/` with a + # trailing slash, so `libdir=/usr/lib` reads as clean. + # + # And running pkg-config by hand does not show it. pkg-config drops a + # -L for a directory it considers a system one, so `pkg-config --libs + # libelf` prints `-lelf` and looks fine — while meson, which sets + # PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 on purpose, gets `-L/usr/lib -lelf`. + # Measured both ways against elfutils' .pc on 2026-08-08. + # + # So every absolute /usr in the copy is rewritten, not just prefix. pcdir="$WORK/pc/$dep" mkdir -p "$pcdir" for pc in "$pcsrc"/*.pc; do [[ -e "$pc" ]] || continue - sed "s#^prefix=.*#prefix=${depdir%/}#" "$pc" > "$pcdir/$(basename "$pc")" + sed -e "s#^prefix=.*#prefix=${depdir%/}#" \ + -e "s#=/usr\$#=${depdir%/}#" \ + -e "s#=/usr/#=${depdir%/}/#g" \ + -e "s#\(-[IL]\)/usr/#\1${depdir%/}/#g" \ + "$pc" > "$pcdir/$(basename "$pc")" done case ":$PKG_CONFIG_LIBDIR:" in *":$pcdir:"*) ;; @@ -321,6 +652,18 @@ case "$SYSTEM" in *) fail "unknown build system '$SYSTEM'" ;; esac +# Inputs before payload, and before anything is packaged. +# +# Ordering, not taste: a host input is the cause and a host path in the payload +# is one of its symptoms, so reporting the cause first is what stops someone +# patching an RPATH and calling it fixed. And failing here means no tarball is +# written — the mesa that shipped against the host's LLVM had already been +# packaged and staged by the time anyone looked. +# +# Its exit code is this script's exit code: 1 for a host input, 2 for "could +# not read what the build linked against". Neither becomes 0. +check_build_inputs "$BUILDDIR" || exit $? + # ── flatten DESTDIR/usr into the payload root ─────────────────────────── PAYLOAD="$WORK/payload/$NAME-$VERSION" rm -rf "$PAYLOAD"; mkdir -p "$PAYLOAD" @@ -356,7 +699,9 @@ while IFS= read -r -d '' f; do esac done < <(find "$PAYLOAD" -type f ! -type l -print0) -# ── the check that makes this worth scripting ─────────────────────────── +# ── the check on the OUTPUT ───────────────────────────────────────────── +# The inputs were checked above. This is the payload: what a host path in here +# breaks is the NEXT machine, not this one. leaks=0 report_leak() { echo " $*"; leaks=$((leaks+1)); } diff --git a/.agents/tools/graphics/build-libllvm.sh b/.agents/tools/graphics/build-libllvm.sh index b318e93e..37e1c6bb 100755 --- a/.agents/tools/graphics/build-libllvm.sh +++ b/.agents/tools/graphics/build-libllvm.sh @@ -35,8 +35,12 @@ WORK="${XLINGS_GFX_WORK:-${TMPDIR:-/tmp}/xlings-gfx}" log() { echo "[libllvm] $*"; } fail() { echo "[libllvm] FAIL: $*" >&2; exit 1; } +# 3, not 1: a missing subos, cmake, ninja or compiler means the build never +# started. 1 is reserved for "it started and broke", which is the only signal +# worth reading the logs for. Contract: .agents/tools/README.md. +skip() { echo "[libllvm] SKIP: $*" >&2; exit 3; } -[[ -d "$SUBOS" ]] || fail "subos '$SUBOS_NAME' not found" +[[ -d "$SUBOS" ]] || skip "subos '$SUBOS_NAME' not found — xlings subos new $SUBOS_NAME" mkdir -p "$WORK/src" "$WORK/dist" SRC="$WORK/src/llvm-$VERSION" @@ -56,8 +60,8 @@ fi BUILD="$WORK/src/llvm-$VERSION-build" rm -rf "$BUILD"; mkdir -p "$BUILD" -CMAKE="$SUBOS/bin/cmake"; [[ -x "$CMAKE" ]] || CMAKE="$(command -v cmake)" || fail "no cmake" -NINJA="$SUBOS/bin/ninja"; [[ -x "$NINJA" ]] || NINJA="$(command -v ninja)" || fail "no ninja" +CMAKE="$SUBOS/bin/cmake"; [[ -x "$CMAKE" ]] || CMAKE="$(command -v cmake)" || skip "no cmake" +NINJA="$SUBOS/bin/ninja"; [[ -x "$NINJA" ]] || NINJA="$(command -v ninja)" || skip "no ninja" # gcc 15.1.0, explicitly — not the subos default and not clang. # @@ -80,7 +84,7 @@ NINJA="$SUBOS/bin/ninja"; [[ -x "$NINJA" ]] || NINJA="$(command -v ninja)" || # Select the version with `xlings use gcc 15.1.0` first. BUILD_CC="$SUBOS/bin/gcc" BUILD_CXX="$SUBOS/bin/g++" -[[ -x "$BUILD_CC" ]] || fail "no gcc shim in the subos" +[[ -x "$BUILD_CC" ]] || skip "no gcc shim in the subos (xlings install gcc, then xlings use gcc 15.1.0)" log "compiler: $("$BUILD_CC" --version | head -1)" log "configuring (X86;AMDGPU, shared libLLVM)" diff --git a/.agents/tools/graphics/collect-matrix.md b/.agents/tools/graphics/collect-matrix.md new file mode 100644 index 00000000..7088a12f --- /dev/null +++ b/.agents/tools/graphics/collect-matrix.md @@ -0,0 +1,108 @@ +# Running the graphics matrix on hardware we do not have + +The stack ships drivers nobody has ever run. `radeonsi` and `nouveau` are in the +mesa payload — built, packaged, published — and every verification of this +ecosystem to date happened on one machine with one GPU, an RTX 4080. Those two +cells have only ever printed `not here: no amd GPU in /sys/class/drm` and +`not here: proprietary nvidia.ko is bound to this GPU`. + +That is not a gap in the payload. It is a gap in the evidence, and it can only +be closed by someone who owns the card. + +## The command + +`verify-stack.sh` builds its probe from `glprobe.c` in the same directory, so +run it from a clone rather than by piping a single file: + +```sh +git clone https://github.com/openxlings/xim-pkgindex +bash xim-pkgindex/.agents/tools/graphics/verify-stack.sh --json --keep | tee gfx-matrix.txt +``` + +You need `xlings` on `PATH` (or `XLINGS_BIN=/path/to/xlings`). Nothing else: the +script creates its own subos, `gfxverify`, and installs the `graphics` package +into it. + +It writes to your real `XLINGS_HOME`. `--home DIR` points it at a scratch one +instead — but a freshly created home starts on the **GLOBAL** source, so from +the mainland run `XLINGS_HOME=DIR xlings config --mirror CN` first or the +install will look like it hung. `--subos NAME` renames the subos. + +`--keep` only suppresses the "remove it with…" reminder; the subos is kept +either way. It is in the command above so the JSON stays the last line — +`xlings subos remove gfxverify` cleans up when you are done. + +## What comes back + +Seven sections of `✓` / `✗` / `·` lines, a summary, and — with `--json` — one +line of JSON, starting `{"host":`: + +```json +{"host":{"vendors":"amd ","nvidia":"","dxg":false}, + "results":[{"status":"PASS","cell":"software rendering (llvmpipe)","note":"llvmpipe (LLVM 20.1.7, 256 bits)"}, + {"status":"SKIP","cell":"WSL2 d3d12","note":"/dev/dxg absent (not WSL2)"}]} +``` + +`host` is what the machine is, probed from `/sys` rather than guessed from +`/etc/os-release`. Each result is one cell of the matrix: + +| status | meaning | +|---|---| +| `PASS` | the cell was exercised here and holds | +| `FAIL` | the cell was exercised here and does not hold | +| `SKIP` | **not exercised on this machine**, and `note` says why | + +`SKIP` is the point of the format. It is not a pass with an asterisk and not a +soft failure — it is the third outcome, and the reason it is carried all the way +into the JSON is that "we do not have that hardware" and "it works" must never +render the same way. (The full rule, and the bug that produced it, are in +`.agents/tools/README.md`.) + +Exit codes: `0` everything that ran passed, `1` something real failed, `3` +nothing was proven at all. **Send the output whichever it is** — a `1` from a +GPU we have never tested is the single most useful thing this doc can produce. + +## The cells that matter + +| cell | why it is worth your run | +|---|---| +| `radeonsi (hardware amd)` | `radeonsi_dri.so` is in the payload and has never rendered a pixel outside a build machine. **The highest-value cell in the table.** | +| `nouveau (hardware nvidia)` | Same: shipped, never run. Needs a GPU the *open* driver owns — the cell skips itself when the proprietary `nvidia.ko` is bound, so a machine with the NVIDIA driver installed cannot answer this one. | +| `WSL2 d3d12` | `/dev/dxg` support is new and has run on one machine. See below for what to expect — the cell itself will be red. | +| `Vulkan loader + our ICD` | RADV ships for AMD; on an AMD box this and `radeonsi` come as a pair. | +| `GUI application starts` | Only fires if godot is installed (`xlings install godot`). A real application dlopens libraries a surfaceless probe never touches, so this catches gaps no probe can. | +| `Wayland` | Always skips: no probe exists yet. Listed so it stays visible. | + +## Two cells are red on purpose + +If you are on Intel or WSL2, expect a `✗` and do not go looking for a mistake on +your side: + +- `iris (hardware intel)` — **the driver is not in the payload.** mesa is built + `-Dgallium-drivers=llvmpipe,softpipe,radeonsi,nouveau,zink`; iris needs + `libclc`, which is a missing package rather than a missing capability. +- `WSL2 d3d12` — same shape, and only half of it. The host-side sentinel + (`wsl-gl-host-link`, which links Windows' `libd3d12core.so` / `libdxcore.so` + out of `/usr/lib/wsl/lib`) does ship; the mesa driver that would consume them + does not, because it needs `DirectX-Headers`. GL on WSL2 therefore lands on + llvmpipe. + +Both cells end in `… is NOT in the payload`, and that is the correct answer. +Run it anyway: everything else in the matrix — the subos, +the install, discovery, llvmpipe, self-containment — is untested on your +platform too, and a WSL2 run is the only way to find out whether the `/dev/dxg` +detection works outside the one machine it was written on. + +## Where to send it + +Open an issue on titled + +``` +graphics matrix: e.g. graphics matrix: AMD radeonsi +``` + +and attach `gfx-matrix.txt`, plus the distro and GPU model. The JSON line alone +is enough if you would rather not paste the whole run — but if it does not +parse, send the plain-text summary: the `note` fields are not escaped, so a +driver error message containing a quote breaks the JSON, and the text summary +carries the same information. diff --git a/.agents/tools/graphics/publish.sh b/.agents/tools/graphics/publish.sh index 70a2c6a9..78b50c49 100755 --- a/.agents/tools/graphics/publish.sh +++ b/.agents/tools/graphics/publish.sh @@ -25,7 +25,8 @@ warn() { echo "[publish] WARN: $*" >&2; } fail() { echo "[publish] FAIL: $*" >&2; exit 1; } command -v gh >/dev/null || fail "gh not found" -command -v gtc >/dev/null || warn "gtc not found — CN mirror will be skipped" +HAVE_GTC=1 +command -v gtc >/dev/null || { HAVE_GTC=0; warn "gtc not found — CN mirror will be skipped"; } shopt -s nullglob FILES=("$DIST"/*.tar.gz) @@ -35,6 +36,7 @@ README_DIR="${XLINGS_GFX_README_DIR:-/tmp/xlings-res-readme}" MANIFEST="$DIST/RECIPE-DATA.txt" : > "$MANIFEST" +BADGLOBAL=0 for f in "${FILES[@]}"; do base="$(basename "$f")" # name-version-linux-x86_64.tar.gz @@ -119,9 +121,24 @@ for f in "${FILES[@]}"; do && cmp -s "$TMP/c.bin" "$f"; then ok_cn=yes; fi log " GLOBAL=$ok_global CN=$ok_cn sha256=${sha:0:16}…" - [[ "$ok_global" == yes ]] || warn "$name: GLOBAL asset does not match the local file" + [[ "$ok_global" == yes ]] || { warn "$name: GLOBAL asset does not match the local file"; BADGLOBAL=$((BADGLOBAL+1)); } printf '%s|%s|%s|%s|%s|%s\n' "$name" "$version" "$sha" "$GLOBAL" "$CN" "$ok_cn" >> "$MANIFEST" done log "recipe data → $MANIFEST" + +# Every failure above was a `warn` and a `continue`, and the script then ended +# on a success line. So a run where GitHub refused all twenty-two releases and +# one where all twenty-two verified byte-for-byte both exited 0 — and the exit +# code is the only part a caller reads. +# +# 1 if a GLOBAL asset does not match what we uploaded; 3 if the CN half was +# never attempted (a publish that reached one region of two is not a publish). +if [[ "$BADGLOBAL" -gt 0 ]]; then + fail "$BADGLOBAL package(s): the GLOBAL asset does not match the local file" +fi +if [[ $HAVE_GTC -eq 0 ]]; then + echo "[publish] NOT DONE: no gtc, so the CN mirror was not published or verified" >&2 + exit 3 +fi diff --git a/.agents/tools/graphics/selfcontained-check.sh b/.agents/tools/graphics/selfcontained-check.sh index 0439cd11..ae0eb8f6 100755 --- a/.agents/tools/graphics/selfcontained-check.sh +++ b/.agents/tools/graphics/selfcontained-check.sh @@ -48,6 +48,7 @@ fail() { echo "[gfx-check] FAIL: $*" >&2; exit 1; } # itself before calling here — every other skip was live. # # 0 = proven, 1 = broken, 2 = inconclusive, 3 = could not be exercised here. +# The contract, and every caller's obligation under it: .agents/tools/README.md. skip() { echo "[gfx-check] SKIP: $*"; exit 3; } command -v bwrap >/dev/null || skip "bwrap not available (xlings install bwrap)" @@ -56,7 +57,12 @@ XLINGS_BIN="${XLINGS_BIN:-$(command -v xlings)}" XHOME="${XLINGS_HOME:-$HOME/.xlings}" SUBOS_DIR="$XHOME/subos/$SUBOS_NAME" -[[ -d "$SUBOS_DIR" ]] || fail "subos '$SUBOS_NAME' does not exist — create it and install \`mesa\` into it first" +# skip, not fail. verify-stack.sh creates the subos before calling here, so this +# only fires when the script is run on its own -- and "you have not built the +# thing yet" is not evidence that the thing is not self-contained. As `fail` it +# exited 1, which verify-stack's default branch renders as +# ✗ empty-host self-containment, a red cell for a check with no subject. +[[ -d "$SUBOS_DIR" ]] || skip "subos '$SUBOS_NAME' does not exist — create it and install \`mesa\` into it first" # ── the probe ─────────────────────────────────────────────────────────── # Built against the SUBOS's headers and libraries, not the host's. Building it diff --git a/.agents/tools/graphics/tiers.sh b/.agents/tools/graphics/tiers.sh index 7627d483..f9ac00c1 100755 --- a/.agents/tools/graphics/tiers.sh +++ b/.agents/tools/graphics/tiers.sh @@ -109,11 +109,15 @@ run_tier() { # local args=(--name "$name" --version "$version" --url "$url" --system "$system") [[ -n "${deps:-}" ]] && args+=(--deps "$deps") + # `|| return $?`, not `|| return 1`: this is a pass-through, and + # rewriting the child's status to 1 erases the difference between "the + # build broke" and "this machine could not start it" (exit 3, see + # .agents/tools/README.md). The tier still stops either way. # shellcheck disable=SC2086 if [[ -n "$extra" ]]; then - bash "$BUILD" "${args[@]}" -- $extra || return 1 + bash "$BUILD" "${args[@]}" -- $extra || return $? else - bash "$BUILD" "${args[@]}" || return 1 + bash "$BUILD" "${args[@]}" || return $? fi done } diff --git a/.agents/tools/graphics/verify-host-link.sh b/.agents/tools/graphics/verify-host-link.sh index 0864decf..94330923 100755 --- a/.agents/tools/graphics/verify-host-link.sh +++ b/.agents/tools/graphics/verify-host-link.sh @@ -18,6 +18,9 @@ # Usage: verify-host-link.sh [subos] # Requires: the home already has nvidia-gl-host-link installed, an X server on # $DISPLAY, and an NVIDIA driver on the host. +# +# Exit codes follow .agents/tools/README.md: 0 proven, 1 broken, 2 inconclusive, +# 3 could-not-run. set -euo pipefail HOME_DIR="${1:?usage: verify-host-link.sh [subos]}" @@ -50,7 +53,17 @@ if [[ -z "$CC" ]]; then [[ -x "$c" ]] && { CC="$c"; break; } done fi -[[ -n "$CC" ]] || { echo "no host compiler (/usr/bin/gcc); set CC=" >&2; exit 2; } +# 3, not 2: no compiler means no probe was ever built, which is a property of +# this machine and not an ambiguous result. verify-stack.sh called this in an +# `if`, so the old 2 landed in the else branch and painted the NVIDIA cell red. +[[ -n "$CC" ]] || { echo "no host compiler (/usr/bin/gcc); set CC=" >&2; exit 3; } + +# patchelf reads the artifact in checks 0 and 4, and its absence is not visible +# in either of their results: `patchelf --print-needed | grep -q '^/'` on a +# missing patchelf greps empty input, answers no, and check 4 then prints +# ✓ host vendor libraries are untouched having examined nothing. Probe once and +# route both to `skip`. +HAVE_PATCHELF=1; command -v patchelf >/dev/null 2>&1 || HAVE_PATCHELF=0 # Headers come from the payloads, not the sysroot: the subos's own # `usr/include` is glibc's, and libglvnd's EGL/GL headers are not linked into @@ -71,6 +84,8 @@ else skip "$f absent on this host — that entry point is unproven" elif [[ -L "$NVLIB/$f" ]]; then bad "$f is a symlink into $(readlink "$NVLIB/$f") — its deps resolve from the host" + elif [[ $HAVE_PATCHELF -eq 0 ]]; then + skip "$f is a real file, but with no patchelf its soname/DT_NEEDED cannot be read — the interposer shape is unproven" else # An interposer is not just "a real file": assert the shape patchelf # was asked to produce, on the artifact. @@ -171,6 +186,8 @@ if [[ -z "$NVHOST" ]]; then # NOT a silent skip. A run that could not find the host's vendor cannot make # any claim about it, and saying nothing reads identically to "checked, fine". skip "no 64-bit host NVIDIA vendor found — check 4 NOT PERFORMED" +elif [[ $HAVE_PATCHELF -eq 0 ]]; then + skip "no patchelf — check 4 NOT PERFORMED (an unreadable DT_NEEDED reads exactly like a clean one)" else echo " · host vendor dir: $NVHOST" hostbad=0 @@ -195,7 +212,12 @@ echo if [[ $skipped -gt 0 ]]; then echo "NOT PERFORMED: $skipped check(s) — see the ! lines above" fi -if [[ $fail -eq 0 ]]; then +if [[ $fail -eq 0 && $pass -eq 0 ]]; then + # Nothing failed because nothing ran. "PASS: 0 checks" is the whole bug class + # in one line, so this is a 3 and the caller must render it as not-run. + echo "NOT RUN: 0 checks executed, $skipped not performed" + exit 3 +elif [[ $fail -eq 0 ]]; then echo "PASS: $pass checks$([[ $skipped -gt 0 ]] && echo ", $skipped not performed")" exit 0 else diff --git a/.agents/tools/graphics/verify-stack.sh b/.agents/tools/graphics/verify-stack.sh index 6ddb6e85..373c4fcf 100755 --- a/.agents/tools/graphics/verify-stack.sh +++ b/.agents/tools/graphics/verify-stack.sh @@ -3,6 +3,9 @@ # # verify-stack.sh [--subos NAME] [--home DIR] [--keep] [--json] # +# exit 0 everything that ran passed · 1 something failed · 3 nothing ran +# sending a run back from hardware we do not have: graphics/collect-matrix.md +# # WHY THIS EXISTS # # Verification of this stack was three scripts that each covered one slice: @@ -38,13 +41,17 @@ while [[ $# -gt 0 ]]; do --home) XHOME_ARG="$2"; shift 2 ;; --keep) KEEP=1; shift ;; --json) JSON=1; shift ;; - -h|--help) sed -n '2,8p' "$0"; exit 0 ;; + -h|--help) sed -n '2,7p' "$0"; exit 0 ;; *) echo "unknown argument: $1" >&2; exit 2 ;; esac done XB="${XLINGS_BIN:-$(command -v xlings 2>/dev/null)}" -[[ -x "$XB" ]] || { echo "no xlings on PATH; set XLINGS_BIN" >&2; exit 2; } +# 3, not 2: without xlings not one cell below can be attempted. See the exit-code +# contract in .agents/tools/README.md -- a caller must count this as "not run", +# never as a pass. (An unknown flag above stays 2: the run produced no verdict +# either way, and both are outside the pass/fail axis.) +[[ -x "$XB" ]] || { echo "no xlings on PATH; set XLINGS_BIN" >&2; exit 3; } export XLINGS_HOME="${XHOME_ARG:-${XLINGS_HOME:-$HOME/.xlings}}" HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" S="$XLINGS_HOME/subos/$SUBOS" @@ -142,7 +149,16 @@ CC="${CC:-}"; [[ -n "$CC" ]] || for c in /usr/bin/gcc /usr/bin/cc /usr/bin/clang [[ -x "$c" ]] && { CC="$c"; break; } done PROBE="" +# Why the reason is a variable: every cell below skips on `-z "$PROBE"` and each +# one used to print "no compiler to build the probe". There are three ways to get +# here and only one of them is that. A machine with gcc where the probe fails to +# LINK (no libEGL in the subos, say) was told it had no compiler, which is a +# wrong cause -- and a wrong cause is worse than no cause, because someone acts +# on it. +PROBE_WHY="no host compiler to build the probe" +[[ -n "$CC" && ! -f "$HERE/glprobe.c" ]] && PROBE_WHY="glprobe.c is not next to this script" if [[ -n "$CC" && -f "$HERE/glprobe.c" ]]; then + PROBE_WHY="the probe did not build against this subos — see /tmp/gfxverify-probe.log" INC="$(find "$XLINGS_HOME/data/xpkgs" -maxdepth 4 -type d -path '*libglvnd*/include' 2>/dev/null | head -1)" if "$CC" -O0 -o /tmp/gfxverify-probe "$HERE/glprobe.c" ${INC:+-I"$INC"} \ -L"$S/lib" -lEGL -lGL -Wl,--dynamic-linker="$S/lib/ld-linux-x86-64.so.2" \ @@ -161,7 +177,7 @@ run_probe() { # $1..: extra env assignments } if [[ -z "$PROBE" ]]; then - na "software rendering (llvmpipe)" "no host compiler to build the probe" + na "software rendering (llvmpipe)" "$PROBE_WHY" else # LIBGL_ALWAYS_SOFTWARE, not just "pick the mesa vendor". # @@ -185,11 +201,20 @@ fi # provenance rather than the renderer string. if [[ $has_nvidia -eq 1 ]]; then if [[ -x "$HERE/verify-host-link.sh" ]]; then - if XLINGS_BIN="$XB" CC="$CC" bash "$HERE/verify-host-link.sh" "$XLINGS_HOME" "$SUBOS" >/tmp/gfxverify-nv.log 2>&1; then - ok "NVIDIA proprietary (interposed)" "$(grep -oE 'PASS: [0-9]+ checks' /tmp/gfxverify-nv.log | head -1)" - else - bad "NVIDIA proprietary (interposed)" "$(grep -oE 'FAIL:.*' /tmp/gfxverify-nv.log | head -1)" - fi + # `if cmd; then ok; else bad; fi` is the caller half of the same bug: it + # reads every non-zero as failure, so verify-host-link's 3 (no host + # compiler, so no probe was ever built) painted this cell red. Match on the + # code, and pass the whole PASS line through -- it carries ", N not + # performed", which the old `PASS: [0-9]+ checks` pattern cut off, leaving a + # green cell that hid four unproven entry points. + XLINGS_BIN="$XB" CC="$CC" bash "$HERE/verify-host-link.sh" "$XLINGS_HOME" "$SUBOS" >/tmp/gfxverify-nv.log 2>&1 + rc=$? + case $rc in + 0) ok "NVIDIA proprietary (interposed)" "$(grep -oE 'PASS:.*' /tmp/gfxverify-nv.log | head -1)" ;; + 2) na "NVIDIA proprietary (interposed)" "INCONCLUSIVE — see /tmp/gfxverify-nv.log" ;; + 3) na "NVIDIA proprietary (interposed)" "$(grep -oE '(NOT RUN|no host compiler).*' /tmp/gfxverify-nv.log | head -1)" ;; + *) bad "NVIDIA proprietary (interposed)" "$(grep -oE 'FAIL:.*' /tmp/gfxverify-nv.log | head -1)" ;; + esac else na "NVIDIA proprietary (interposed)" "verify-host-link.sh not present" fi @@ -210,7 +235,7 @@ for pair in "amd:radeonsi" "intel:iris" "nvidia:nouveau"; do # The open driver cannot bind a GPU the proprietary kernel module owns. na "$drv (hardware $v)" "proprietary nvidia.ko is bound to this GPU" elif [[ -z "$PROBE" ]]; then - na "$drv (hardware $v)" "no compiler to build the probe" + na "$drv (hardware $v)" "$PROBE_WHY" else out="$(run_probe __EGL_VENDOR_LIBRARY_FILENAMES="$VD/50_mesa.json" MESA_LOADER_DRIVER_OVERRIDE="$drv")" r="$(sed -n 's/^GL_RENDERER=//p' <<<"$out")" @@ -234,7 +259,7 @@ if [[ $has_dxg -eq 1 ]]; then if [[ ! -e "$S/usr/lib/dri/d3d12_dri.so" ]]; then bad "WSL2 d3d12" "/dev/dxg present but d3d12_dri.so is NOT in the payload" elif [[ -z "$PROBE" ]]; then - na "WSL2 d3d12" "no compiler to build the probe" + na "WSL2 d3d12" "$PROBE_WHY" else out="$(run_probe GALLIUM_DRIVER=d3d12 __EGL_VENDOR_LIBRARY_FILENAMES="$VD/50_mesa.json")" grep -q "^RESULT=ok" <<<"$out" \ @@ -308,10 +333,20 @@ else # The claim A7 rests on: with the stack present, no host directory is on the # application's RPATH. A renderer string cannot show this. - hostdirs=$(patchelf --print-rpath "$APP_EXE" 2>/dev/null | tr ':' '\n' | grep -cE '^/usr/lib|^/lib' || true) - [[ "${hostdirs:-0}" -eq 0 ]] \ - && ok "app RPATH free of host dirs" "0 host directories" \ - || bad "app RPATH free of host dirs" "$hostdirs host directories still on it" + # + # The patchelf probe is not decoration. Without it this pipeline read an empty + # RPATH from a command that does not exist, counted zero host directories in + # it, and printed ✓ app RPATH free of host dirs -- the strongest-sounding cell + # in the section, produced by running nothing. 2>/dev/null and `|| true` + # between them erased both the "command not found" and the non-zero status. + if ! command -v patchelf >/dev/null 2>&1; then + na "app RPATH free of host dirs" "no patchelf to read the RPATH with" + else + hostdirs=$(patchelf --print-rpath "$APP_EXE" 2>/dev/null | tr ':' '\n' | grep -cE '^/usr/lib|^/lib' || true) + [[ "${hostdirs:-0}" -eq 0 ]] \ + && ok "app RPATH free of host dirs" "0 host directories" \ + || bad "app RPATH free of host dirs" "$hostdirs host directories still on it" + fi fi # ── self-containment ──────────────────────────────────────────────────── @@ -356,4 +391,18 @@ fi [[ $KEEP -eq 1 ]] || echo " (subos '$SUBOS' kept; remove with: xlings subos remove $SUBOS)" # Skips do not fail the run. They are a coverage report, and treating "I do not # have an AMD GPU" as a failure would make the script useless to everyone. +# +# But `fail -eq 0` alone exits 0 for a machine where every cell was skipped, and +# pass 0 / fail 0 is not a pass. Both counters, in this order: a run where +# everything genuinely failed is broken (1), not could-not-run (3). +# +# As the sections stand this cannot fire — section 1 either records a pass or +# exits early — so it is a guard rather than a live path. It is here because the +# next `na` added above section 1, or a --no-install flag, would silently make +# "ran nothing" exit 0 again, which is the one outcome this script exists to +# rule out. +if [[ $pass -eq 0 && $fail -eq 0 ]]; then + echo " nothing was proven on this machine — reporting could-not-run, not success" + exit 3 +fi [[ $fail -eq 0 ]] && exit 0 || exit 1 diff --git a/.agents/tools/verify-toolchain.sh b/.agents/tools/verify-toolchain.sh index d4abd005..75b40c49 100755 --- a/.agents/tools/verify-toolchain.sh +++ b/.agents/tools/verify-toolchain.sh @@ -11,13 +11,23 @@ # # Usage: verify-toolchain.sh TARBALL [--loader LD] # Default loader: the xim glibc loader on this machine. +# +# Exit codes follow .agents/tools/README.md: 0 proven, 1 broken, 2 inconclusive, +# 3 could-not-run. Both SKIPs below used to be `exit 0`, so a machine with no +# usable loader, and a tarball with no compiler in it at all, reported the same +# status as a toolchain that compiled and ran a program. There is no way for a +# caller to tell those apart from 0. set -uo pipefail +skip() { echo "SKIP: $*"; exit 3; } + TARBALL="${1:-}"; shift || true LOADER="$HOME/.xlings/data/xpkgs/xim-x-glibc/2.39/lib64/ld-linux-x86-64.so.2" while [ $# -gt 0 ]; do case "$1" in --loader) LOADER="$2"; shift 2;; *) shift;; esac; done [ -f "$TARBALL" ] || { echo "error: tarball not found: $TARBALL" >&2; exit 2; } -command -v patchelf >/dev/null || { echo "error: patchelf required" >&2; exit 1; } +# 3, not 1: an absent patchelf says nothing about the tarball, and 1 here sends +# the reader to look for a corrupt artifact. +command -v patchelf >/dev/null || skip "patchelf required, not on PATH" T="$(mktemp -d)"; trap 'rm -rf "$T"' EXIT tar -xzf "$TARBALL" -C "$T" || { echo "FAIL: tarball does not extract" >&2; exit 1; } @@ -28,7 +38,7 @@ GLIBC_LIB="$(dirname "$LOADER")" CXX="$(find "$ROOT" -maxdepth 2 -path '*/bin/g++' | head -1)" [ -n "$CXX" ] || CXX="$(find "$ROOT" -maxdepth 2 -path '*/bin/*-musl-g++' | head -1)" [ -n "$CXX" ] || CXX="$(find "$ROOT" -maxdepth 2 -path '*/bin/clang++' | head -1)" -[ -n "$CXX" ] || { echo "SKIP: no g++/clang++ in $TARBALL"; exit 0; } +[ -n "$CXX" ] || skip "no g++/clang++ in $TARBALL — nothing here compiles anything" is_musl=0 case "$CXX" in *musl*) is_musl=1;; esac @@ -39,7 +49,7 @@ if [ "$is_musl" = 1 ]; then ML="$(find "$ROOT" -path '*-linux-musl/lib/libc.so' -o -name 'ld-musl-x86_64.so.1' 2>/dev/null | head -1)" [ -n "$ML" ] && LOADER="$ML" && GLIBC_LIB="$(dirname "$ML")" fi -[ -e "$LOADER" ] || { echo "SKIP: loader not available: $LOADER (cannot run on this machine)"; exit 0; } +[ -e "$LOADER" ] || skip "loader not available: $LOADER (cannot run on this machine)" echo "verify: $(basename "$TARBALL") flavor=$([ $is_musl = 1 ] && echo musl || echo glibc) loader=$LOADER" # patch interp + rpath on driver and backend executables (throwaway copy only). diff --git a/.github/scripts/dep-closure-check.sh b/.github/scripts/dep-closure-check.sh new file mode 100755 index 00000000..655eb40c --- /dev/null +++ b/.github/scripts/dep-closure-check.sh @@ -0,0 +1,262 @@ +#!/usr/bin/env bash +# Does what a payload NEEDS match what its recipe DECLARES? +# +# Invoked per package from posix-test.sh, after the install succeeded. +# +# dep-closure-check.sh [] +# +# THE DEFECT THIS EXISTS FOR +# -------------------------- +# A recipe's `deps` list is the only input to the RPATH closure xlings stamps +# onto the payload at install time (elfpatch's `closure_lib_paths`, which reads +# the DIRECT runtime deps -- not the transitive set). So a library the payload +# genuinely loads but the recipe never declared is simply absent from the +# closure. Nothing fails at install time. Nothing fails on a developer machine +# either, because the host happens to have a copy. It fails on a machine that +# does not, and the error names a soname rather than the package that forgot to +# declare it. +# +# That is how libxcb came to search for libXau with no libXau on any search +# path -- an entire index-wide sweep of 28 recipes was needed to repair it, and +# the declaration that would have prevented it is one line. This check is that +# one line, enforced. +# +# THE TWO ASSERTIONS +# ------------------ +# D1 every external soname that some INSTALLED package provides must be +# declared as a direct dep of this recipe. +# +# Transitivity does not save you here and that is the whole point: A -> B +# -> C puts only B's libdirs in A's closure, so if A itself names a C +# soname, A must declare C. Reading "B already depends on C" as sufficient +# is the mistake. +# +# D2 if this payload's interpreter points inside XLINGS_HOME, then EVERY +# external soname must have a provider -- a host-only soname is a hard +# failure, not a note. +# +# Because our glibc's ld.so carries the build machine's cache path +# (`/home/xlings/.xlings_data/.../etc/ld.so.cache`), which exists on no +# machine. On a multiarch distro the host's libraries are reachable ONLY +# through that cache, so switching PT_INTERP removes the host fallback +# outright. Measured on jdk-temurin 2026-08-08: headless Java is perfect +# and AWT dies with `libX11.so.6: cannot open shared object file`, against +# a working unpatched control on the same machine. "It is only dlopen'd, +# so it degrades no further than today" is false -- today it resolves off +# the host, afterwards it resolves nowhere. +# +# For a payload still on the host loader (`code`, the JDKs) host-only +# sonames are expected and reported as notes. +# +# A declared dep that nothing in the payload needs is a WARNING, never a +# failure: build-only tools, plugins loaded by path and data-only packages are +# all legitimate reasons, and failing on them would train people to delete +# correct declarations. +# +# EXIT CODES -- the contract in .agents/tools/README.md +# 0 proven 1 broken 2 inconclusive 3 could not be exercised here +set -uo pipefail + +PAYLOAD="${1:-}" +RECIPE="${2:-}" +PLATFORM="${3:-linux}" +XPKGS="${4:-${XLINGS_HOME:-$HOME/.xlings}/data/xpkgs}" + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "$HERE/../.." && pwd)" + +say() { echo " $*"; } +fail() { echo " [FAIL] $*" >&2; } + +# 3, not 0. A missing tool means this machine could not evaluate the assertion, +# which is a different statement from the assertion holding -- and 0 is what the +# caller reads as "checked, clean". +skip() { echo " [SKIP] $*"; exit 3; } + +[[ -n "$PAYLOAD" && -n "$RECIPE" ]] || { echo "usage: dep-closure-check.sh []" >&2; exit 2; } +[[ -d "$PAYLOAD" ]] || skip "payload dir does not exist: $PAYLOAD" +[[ -f "$RECIPE" ]] || skip "recipe does not exist: $RECIPE" +command -v readelf >/dev/null 2>&1 || skip "no readelf (binutils) on this machine" +command -v lua >/dev/null 2>&1 || LUA="" +LUA="$(command -v lua5.4 || command -v lua || true)" +[[ -n "$LUA" ]] || skip "no lua interpreter; cannot read deps structurally" +[[ -f "$HERE/check-dep-namespace.lua" ]] || skip "check-dep-namespace.lua not present next to this script" + +# ── what the recipe declares ──────────────────────────────────────────── +# Structurally, via the shared reader -- never by grepping the recipe. A dep +# list spans lines and appears under several xpm sections, so a line sweep +# both misses and mis-attributes entries, and both failures look clean. +# +# `*` is the descriptor-level list that applies to every platform. Version- +# scoped rows are included: a dep declared only under one version still governs +# that version's closure, and this runs against exactly one installed version. +rel_recipe="${RECIPE#"$ROOT"/}" +# A recipe outside this checkout produces no rows from --list, which would leave +# `declared` empty and make every soname look undeclared. That fails closed, so +# it is not dangerous -- but it reports a dependency defect for what is really +# a wrong argument, and someone would go edit the deps. Say which it is. +case "$rel_recipe" in + pkgs/*) ;; + *) echo " [SKIP] recipe is not inside this index ($RECIPE); cannot read its deps" + exit 3 ;; +esac +declared="" +while IFS=$'\t' read -r f plat _scope kind _idx dep; do + [[ "$f" == "$rel_recipe" ]] || continue + [[ "$plat" == "$PLATFORM" || "$plat" == "*" ]] || continue + # `build` deps are not in the runtime closure, so they cannot satisfy a + # DT_NEEDED. Positional (`list`) counts: the client copies it into both. + [[ "$kind" == "build" ]] && continue + name="${dep##*:}"; name="${name%%@*}" + declared+="$name"$'\n' +done < <("$LUA" "$HERE/check-dep-namespace.lua" --list "$ROOT" 2>/dev/null) + +is_declared() { [[ -n "$declared" ]] && grep -qxF "$1" <<<"$declared"; } + +# ── who provides what ─────────────────────────────────────────────────── +# Over every INSTALLED payload except the one under test. Store dir layout is +# /-x-//, so the package name comes off the path. +# +# Symlinks count, and that is not a detail: an soname is almost always the +# symlink (`libstdc++.so.6` -> `libstdc++.so.6.0.33`), so `-type f` alone finds +# the real file under a name nothing ever asks for and reports the actual +# soname as host-only. That mistake makes this check accuse correct recipes. +# +# ALL candidates are kept, not the first one found. Package names are not +# unique per soname -- `libgcc_s.so.1` is shipped by gcc-runtime and by every +# cross-toolchain in the home, including aarch64 ones -- and `find` order is +# alphabetical, so first-wins answers `aarch64-linux-musl-gcc` for an x86_64 +# payload. Resolving the ambiguity by preferring a DECLARED candidate is both +# correct and useful: it is the recipe that says which one it meant. +declare -A PROVIDERS +if [[ -d "$XPKGS" ]]; then + while IFS= read -r so; do + case "$so" in "$PAYLOAD"/*) continue ;; esac + b="${so##*/}" + rest="${so#"$XPKGS"/}"; store="${rest%%/*}" + name="${store#*-x-}" + case " ${PROVIDERS[$b]:-} " in + *" $name "*) ;; + *) PROVIDERS["$b"]="${PROVIDERS[$b]:-}${PROVIDERS[$b]:+ }$name" ;; + esac + done < <(find "$XPKGS" -mindepth 3 -maxdepth 6 -name '*.so*' \( -type f -o -type l \) 2>/dev/null) +fi + +# ── what the payload provides itself ──────────────────────────────────── +declare -A SELF +while IFS= read -r so; do SELF["${so##*/}"]=1; done \ + < <(find "$PAYLOAD" -name '*.so*' \( -type f -o -type l \) 2>/dev/null) + +# ── what the payload needs, and whether its loader is ours ────────────── +# +# `sealed` is the discriminator for how strict to be, and it has to be measured +# rather than assumed. Two ways a payload can be resolving from our tree: +# its executables point at our interpreter, or its objects carry an RPATH into +# xpkgs (what selfcontain.seal stamps on a library package, which has no +# interpreter to inspect at all). +# +# A payload that is neither is integrated with the host on purpose -- `code`, +# the JDKs -- and for it the host IS the provider. Applying D1 there would +# demand that every such package declare `xim:glibc` for a libc it does not use, +# and a check that fires on correct recipes gets switched off. +declare -A NEEDED_BY +ours_interp=""; sealed=0; scanned=0 +# Only executables and *.so* are considered, and that is a rule rather than a +# sample: DT_NEEDED is resolved by soname, so a shared object has to be +# `.so`-named for anything to ask for it, and a program has to be executable +# for anything to run it. A file that is neither cannot participate in the +# loader's work no matter what its ELF header says. Scanning everything instead +# meant an `od` per file over payloads like godot and llvm -- minutes each, in a +# check that runs per package in CI. +while IFS= read -r -d '' f; do + # `od`, not `head -c4` in a command substitution: a payload is full of + # non-ELF files (.jar, .png, .dat) whose first bytes contain NUL, and bash + # strips those with a warning on every one of them -- pages of noise around + # the actual result. Compare the hex instead. + [[ "$(od -An -tx1 -N4 "$f" 2>/dev/null | tr -d ' ')" == "7f454c46" ]] || continue + scanned=$((scanned + 1)) + interp="$(readelf -p .interp "$f" 2>/dev/null | grep -oE '/[^ ]*ld-[^ ]*' | head -1)" + case "$interp" in *"/xpkgs/"*) ours_interp="$interp"; sealed=1 ;; esac + rp="$(readelf -d "$f" 2>/dev/null | grep -oP '\((?:RPATH|RUNPATH)\).*\[\K[^\]]+')" + case "$rp" in *"/xpkgs/"*) sealed=1 ;; esac + while IFS= read -r n; do + # A DT_NEEDED may itself be a path (`$ORIGIN/../lib/libpython3.13.so.1.0`). + # It is the basename that has to be provided by someone. + n="${n##*/}" + [[ -n "${SELF[$n]:-}" ]] && continue + NEEDED_BY["$n"]="${NEEDED_BY[$n]:-}${NEEDED_BY[$n]:+ }${f##*/}" + done < <(readelf -d "$f" 2>/dev/null | grep -oP 'Shared library: \[\K[^\]]+') +done < <(find "$PAYLOAD" -type f ! -type l \ + \( -perm -u+x -o -name '*.so' -o -name '*.so.*' \) -print0 2>/dev/null) + +if [[ $scanned -eq 0 ]]; then + # Not a pass. A payload with no ELF in it (a header-only package, a data + # package, a pure script) has nothing for this check to say, and saying + # "clean" would be a claim it has not earned. + say "no ELF objects in this payload; dependency closure not evaluated" + exit 3 +fi + +# ── the assertions ────────────────────────────────────────────────────── +undeclared=(); hostonly=(); used=() +for n in $(printf '%s\n' "${!NEEDED_BY[@]}" | sort); do + cands="${PROVIDERS[$n]:-}" + if [[ -z "$cands" ]]; then + hostonly+=("$n (needed by ${NEEDED_BY[$n]%% *})") + continue + fi + hit="" + for c in $cands; do is_declared "$c" && { hit="$c"; break; }; done + if [[ -n "$hit" ]]; then + used+=("$hit") + else + undeclared+=("$n -> provided by {${cands// /, }} (needed by ${NEEDED_BY[$n]%% *})") + fi +done + +rc=0 + +if [[ $sealed -eq 0 ]]; then + say "this payload resolves from the host (no xlings interpreter, no xpkgs RPATH);" + say "D1/D2 are reported but not enforced -- the host is its provider by design." + [[ ${#undeclared[@]} -gt 0 ]] && say " would-be undeclared: ${#undeclared[@]}" + [[ ${#hostonly[@]} -gt 0 ]] && say " host-provided sonames: ${#hostonly[@]}" + exit 0 +fi + +if [[ ${#undeclared[@]} -gt 0 ]]; then + fail "D1: ${#undeclared[@]} soname(s) resolve to an installed package this recipe does not declare:" + for u in "${undeclared[@]}"; do echo " $u" >&2; done + echo " Add them to deps. A transitive dep does NOT put its libdir in this" >&2 + echo " payload's RPATH closure -- only direct deps do." >&2 + rc=1 +fi + +if [[ ${#hostonly[@]} -gt 0 ]]; then + if [[ -n "$ours_interp" ]]; then + fail "D2: this payload uses OUR loader ($ours_interp)" + fail " but ${#hostonly[@]} soname(s) have no provider in the index:" + for h in "${hostonly[@]}"; do echo " $h" >&2; done + echo " There is no host fallback behind our loader -- its ld.so.cache path" >&2 + echo " does not exist on any machine. These will not be found at runtime," >&2 + echo " including the ones only reached via dlopen." >&2 + rc=1 + else + say "note: ${#hostonly[@]} soname(s) come from the host; this payload is on the" + say " host loader, so that is the documented arrangement, not a leak:" + for h in "${hostonly[@]}"; do say " $h"; done + fi +fi + +# Warning only, deliberately -- see the header. +used_list="$(printf '%s\n' ${used[@]+"${used[@]}"} | sort -u)" +while IFS= read -r d; do + [[ -n "$d" ]] || continue + grep -qxF "$d" <<<"$used_list" \ + || say "warn: declares '$d', but nothing in the payload names a soname it provides" +done <<<"$declared" + +if [[ $rc -eq 0 ]]; then + say "dependency closure: $scanned ELF, ${#NEEDED_BY[@]} external soname(s), all accounted for" +fi +exit $rc diff --git a/.github/scripts/posix-test.sh b/.github/scripts/posix-test.sh index 371bfaf5..0eb2d63f 100755 --- a/.github/scripts/posix-test.sh +++ b/.github/scripts/posix-test.sh @@ -380,7 +380,15 @@ for rel_file in "${files[@]}"; do step "[$pkg] loader/libc same-source" split_found=0 while IFS= read -r -d '' elf; do - [[ "$(head -c4 "$elf" 2>/dev/null)" == $'\x7fELF' ]] || continue + # `od`, not `head -c4` in a command substitution. This walks the + # WHOLE store, which is full of non-ELF files whose first four + # bytes contain NUL, and bash strips those with + # + # warning: command substitution: ignored null byte in input + # + # once per file -- hundreds of lines of it, burying the actual + # result. Compare the magic as hex instead. + [[ "$(od -An -tx1 -N4 "$elf" 2>/dev/null | tr -d ' ')" == "7f454c46" ]] || continue interp="$(patchelf --print-interpreter "$elf" 2>/dev/null)" || continue [[ -n "$interp" ]] || continue payload_of() { sed -E 's#(.*/xpkgs/[^/]+/[^/]+)/.*#\1#' <<<"$1"; } @@ -407,6 +415,42 @@ for rel_file in "${files[@]}"; do log_pass "loader and libc come from one payload" fi + # Declared deps vs the payload's real DT_NEEDED. + # + # Runs here, on the freshly installed payload, because that is the only + # place both halves exist at once: the recipe's declaration and the bytes + # xlings produced from it. A static check of the recipe cannot see what the + # binaries need, and a check of the binaries alone cannot see what was + # promised. + # + # Exit 3 is "this machine could not evaluate it" (no readelf, no lua, a + # payload with no ELF in it) and must not be read as a pass -- so it is + # reported and skipped, not folded into log_pass. See .agents/tools/README.md. + if [[ "$HOST_OS" == "linux" && -n "$installed_version" ]]; then + step "[$pkg] declared deps vs DT_NEEDED" + # `examined` exists because the first version of this block could print + # its header and then nothing at all. A script-type package has no + # payload directory, so `install_dirs` is empty, `<<<` still feeds the + # loop one empty line, the -d test skips it, and the step reported + # neither pass nor skip -- a check announcing itself and going silent, + # which is the exact shape the check was added to remove. + examined=0 + while IFS= read -r dir; do + [[ -n "$dir" && -d "$dir/$installed_version" ]] || continue + examined=$((examined + 1)) + "$WORKSPACE_ROOT/.github/scripts/dep-closure-check.sh" \ + "$dir/$installed_version" "$lua_file" "$HOST_OS" "$XPKGS_DIR" + case $? in + 0) log_pass "dependency closure complete" ;; + 3) info "not evaluated on this machine (exit 3)" ;; + *) log_fail "dependency closure incomplete" + failures+=("$rel_file (dep-closure)") ;; + esac + done <<< "$install_dirs" + [[ $examined -eq 0 ]] \ + && info "no payload directory for '$pkg' (type=$pkg_type); nothing to check" + fi + # Remove exactly what this test installed, not "whatever is active" — a # bare removal resolves the ACTIVE version, which for a binding-group member # can carry a provider annotation that is a DISPLAY form, not a key. @@ -417,7 +461,55 @@ for rel_file in "${files[@]}"; do fi step "[$pkg] uninstall ($remove_spec)" - if ! "$XLINGS_CMD" remove "$remove_spec" -y; then + remove_out="$("$XLINGS_CMD" remove "$remove_spec" -y 2>&1)"; remove_rc=$? + printf '%s\n' "$remove_out" + if [[ $remove_rc -ne 0 ]]; then + # A `type = "config"` package configures the system and registers no + # xvm version of its own, so there is nothing for removal to select: + # + # uninstall failed: xvm removal selection failed for xim:cpp@gnu: + # removal version is not registered (target='cpp', version='gnu') + # + # 11 of the 12 config-type recipes in this index call xvm.add zero + # times, so this is the shape of the type rather than a fault in one + # recipe -- the same reason the `namespace = "config"` branch above + # skips the lifecycle assertion entirely. Whether `remove` should + # succeed as a no-op there is a real question, and an xlings-side one; + # it is not this test's to answer, and it is not what put these + # recipes in the changed set. + # + # Narrow on purpose: the config type AND this exact diagnostic. Keying + # on the type alone would also swallow a genuine removal bug in + # `xvm-sysdetect`, the one config package that DOES call xvm.add -- and + # a tolerance that hides the case it was not written for is how a + # skipped assertion becomes permanent. + if [[ "$pkg_type" == "config" ]] \ + && grep -q "removal version is not registered" <<<"$remove_out"; then + info "uninstall not asserted: type='config' registers no xvm version" + info " (${remove_spec} -- see the note in this script)" + continue + fi + + # Removing xlings when it is the only installed version is refused by + # design -- that binary is the one running the command, and there is a + # separate `xlings self uninstall` built for it. + # + # This only started happening to bump PRs after #543. Before it, a + # changed recipe was registered via `config --add-xpkg` and installed + # as `local:xlings`, which the running-binary guard does not match; + # #543 made CI overlay the recipe into the index instead, so it now + # installs as `xim:xlings` and the guard fires. PR #541 (the previous + # bump) shows `local:xlings@2026.8.7.1` in its log and passed; #548 + # shows `xim:xlings@2026.8.8.1` and did not. Nothing about xlings or + # about the recipe changed in between. + # + # Every future bump PR touches pkgs/x/xlings.lua, so this would be red + # on all of them. + if grep -q "cannot remove the running binary itself" <<<"$remove_out"; then + info "uninstall not asserted: this IS the running xlings, and it is" + info " the only installed version (use \`xlings self uninstall\`)" + continue + fi log_fail "uninstall failed"; failures+=("$rel_file (uninstall)"); continue fi diff --git a/.github/workflows/ci-test.yml b/.github/workflows/ci-test.yml index bd367d03..398f6b3c 100644 --- a/.github/workflows/ci-test.yml +++ b/.github/workflows/ci-test.yml @@ -190,10 +190,27 @@ jobs: with: files: pkgs/**/*.lua + # patchelf/binutils and lua5.4 are what the post-install dependency + # closure check needs (dep-closure-check.sh: readelf for DT_NEEDED, + # lua to read the recipe's deps structurally). + # + # This is a SEPARATE job from linux-test, on its own runner, so a lua5.4 + # installed there is not here. Without this step the check finds no + # interpreter and exits 3 -- correctly reported as "not evaluated", but a + # guard that never evaluates anything is not a guard, and the job still + # goes green. + - name: Install ELF + Lua tooling for the closure check + if: steps.changed-files.outputs.any_changed == 'true' + run: | + sudo apt-get update -qq + sudo apt-get install -y --no-install-recommends binutils patchelf lua5.4 + readelf --version | head -1 + lua5.4 -v + - name: Linux install/uninstall test (changed packages) if: steps.changed-files.outputs.any_changed == 'true' run: | - chmod +x .github/scripts/posix-test.sh + chmod +x .github/scripts/posix-test.sh .github/scripts/dep-closure-check.sh .github/scripts/posix-test.sh \ "${{ steps.changed-files.outputs.all_changed_files }}" \ "${{ github.workspace }}" \