diff --git a/docs/compatibility.rst b/docs/compatibility.rst index 5850cc0c74..fb50cf54a0 100644 --- a/docs/compatibility.rst +++ b/docs/compatibility.rst @@ -54,7 +54,8 @@ The following table lists the minimum requirements for running Hyperloom. | Python | >= 3.10 | +---------------------+--------------------------------------------------------+ | Inference Framework | SGLang (>= 0.5.12), vLLM (>= 0.21.0), | -| | ATOM (preinstalled; see below), plus ``custom`` | +| | ATOM (preinstalled or setup-installed; see below), | +| | plus ``custom`` | | | benchmark script | +---------------------+--------------------------------------------------------+ | Kernel Languages | HIP, Triton, FlyDSL | @@ -131,7 +132,7 @@ The following inference frameworks are supported: frameworks within one session * - ATOM - 7.2.4 (recorded image stack) - - AMD out-of-tree engine, launched as ``python3 -m atom.entrypoints.openai_server``. Supports Docker or direct execution in a preinstalled ATOM/ROCm torch environment. The recorded image-based stack used ``rocm/atom-dev:v0.1.7-rc0`` on MI355X, not a universal ATOM pip-version minimum. Other builds require local validation. An unset or empty ``KERNEL_OPT_BACKEND_ORDER`` resolves to GEAK here as it does for every framework, and explicit values are preserved. + - AMD out-of-tree engine, launched as ``python3 -m atom.entrypoints.openai_server``. Supports Docker, direct execution in a preinstalled ATOM/ROCm torch environment, or a bare-metal source install with ``--install-framework atom``. The recorded image-based stack used ``rocm/atom-dev:v0.1.7-rc0`` on MI355X, not a universal ATOM pip-version minimum. Other builds require local validation. An unset or empty ``KERNEL_OPT_BACKEND_ORDER`` resolves to GEAK here as it does for every framework, and explicit values are preserved. * - ``custom`` - Host-defined - Escape hatch for your own benchmark script; Hyperloom does not manage the server lifecycle. Requires ``HYPERLOOM_BENCHMARK_BACKEND=bypass`` plus ``--framework-path`` (or ``FRAMEWORK_REPO_PATH``) and ``--benchmark-scripts-dir`` (or ``HYPERLOOM_BYPASS_SCRIPTS_DIR``); the CLI exits with status 2 when any of the three is missing. @@ -188,8 +189,15 @@ Hyperloom does not install ROCm or torch itself. For ATOM, ``baremetal`` means running directly in the development machine's selected Python environment, including when the development platform itself is -a container; it does not start an additional Docker container. ATOM must already -be installed. Verify a real ``import atom`` and a non-empty ``torch.version.hip`` +a container; it does not start an additional Docker container. ATOM is either +already installed there or installed by setup with ``--install-framework atom``, +which builds AITER and then ATOM (default ``ATOM_REF``: the commit +``rocm/atom-dev:v0.1.7-rc0`` was built from) against the existing ROCm torch in +that Python. That Python must not also serve SGLang or vLLM: ATOM registers +plugins that both engines load by default, so setup refuses to install ATOM +where either imports (including a ROCm 10 vLLM venv built over that Python), +and refuses to install SGLang, shared vLLM or ROCm 10 source vLLM where ATOM +imports. Run ATOM in a separate container instead. Verify a real ``import atom`` and a non-empty ``torch.version.hip`` with that Python, keep its executable first on ``PATH`` for Magpie's ``python3`` launch, and use ``PYTHON`` with ``INFERENCE_OPTIMIZER_FORCE_PYTHON=1`` if pinning the interpreter. Keep any activated venv consistent; ``/opt/venv`` is not required. @@ -199,9 +207,9 @@ Run ``python -m hyperloom.inference_optimizer.setup --check-only -- selected interpreter first. Only after approval, repeat without ``--check-only``: ``none`` skips framework installation but can still write configuration and apply ROCm hotfixes. Preserve the selected ``USER_DATA_PATH``; an existing setup -need not be repeated. Hyperloom does not install ATOM and does not assert a -minimum ATOM package version derived from a Docker tag. Import checks establish -local prerequisites, not end-to-end validation of every stack. +need not be repeated. Hyperloom does not assert a minimum ATOM package version +derived from a Docker tag. Import checks establish local prerequisites, not +end-to-end validation of every stack. .. list-table:: :header-rows: 1 diff --git a/docs/install/install.md b/docs/install/install.md index e6f11e5c62..847ed2454f 100644 --- a/docs/install/install.md +++ b/docs/install/install.md @@ -132,7 +132,7 @@ The backend runs `install_baremetal.sh` in five phases: 1. **Base preflight**: Checks ROCm, GPU arch, ROCm torch, torch/triton alignment, and serving framework imports. -2. **Framework install**: Optionally installs the SGLang or vLLM framework layer. +2. **Framework install**: Optionally installs the SGLang, vLLM or ATOM framework layer. 3. **ROCm hotfix**: Applies the profiler hotfix when the ROCm stack is eligible, covering both `/opt/rocm/lib` and PyTorch's bundled `torch/lib/`. 4. **Credentials**: Resolves LLM gateway credentials into `.env`. @@ -396,7 +396,7 @@ set of accepted shapes, including split entrypoints and self-hosted gateways. Make sure the host already provides the required base environment: - ROCm runtime and a ROCm-built torch. -- A serving framework (SGLang or vLLM) importable in the active Python. +- A serving framework (SGLang, vLLM or ATOM) importable in the active Python. - `git` for the dependency checkouts the optimization skill performs. With that in place, open the repository root in the agent and paste a launch diff --git a/docs/reference/environment-variables.md b/docs/reference/environment-variables.md index edd66febad..1f5b15357d 100644 --- a/docs/reference/environment-variables.md +++ b/docs/reference/environment-variables.md @@ -721,8 +721,9 @@ The following variables configure framework source discovery and path overrides. | `INFERENCE_`
`OPTIMIZER`
`_STRICT_PATHS` | `1` when CLI bootstraps | When `1`, missing path env raises instead of falling back to discovery. Set by the CLI at session start; do not override unless debugging. | | `HYPERLOOM_`
`SGLANG_PA`
`TCH_EXACT`
`_VERSIONS` | Unset | Pin the sglang server-patch step to specific upstream versions; advanced compatibility option. | | `HYPERLOOM_`
`ENABLE`
`_PATCH` | `1` | Set to `0` to skip the in-place server patch step (useful when the upstream is already pre-patched). | -| `HYPERLOOM_`
`SKIP_FRAME`
`WORK_CHECK` | Unset (check enabled) | Truthy skips the `optimize` preflight gate that requires the selected serving framework to be importable and a ROCm build. Last resort: when the server runs elsewhere, set `BENCHMARK_BASE_URL` instead, which exempts the check and configures the supported path. The gate already stays out of the way for `xdit`/`custom` (server-less), external multi-node, and any framework `install_baremetal.sh` cannot install (`atom`), where it warns instead of blocking. | +| `HYPERLOOM_`
`SKIP_FRAME`
`WORK_CHECK` | Unset (check enabled) | Truthy skips the `optimize` preflight gate that requires the selected serving framework to be importable and a ROCm build. Last resort: when the server runs elsewhere, set `BENCHMARK_BASE_URL` instead, which exempts the check and configures the supported path. The gate already stays out of the way for `xdit`/`custom` (server-less) and external multi-node. | | `AITER_REF` | Unset | Optional bare-metal AITER install pin. When unset, the installer selects the newest tag compatible with the installed torch/triton stack. | +| `ATOM_REF` | `fe1099b15ddc52e6873e1932935f722864cdeee0` | Full commit SHA that `--install-framework atom` checks out from `ATOM_REPO` (default `https://github.com/ROCm/ATOM.git`) into `ATOM_ROOT` (default `/atom`). The default is the commit `rocm/atom-dev:v0.1.7-rc0` was built from. | --- diff --git a/examples/README.md b/examples/README.md index c4ddf85bc5..9448e71c12 100644 --- a/examples/README.md +++ b/examples/README.md @@ -124,7 +124,7 @@ The backend runs `install_baremetal.sh` in five phases: 1. **Base preflight**: checks ROCm, GPU arch, ROCm torch, torch/triton alignment, and serving framework imports. -2. **Framework install**: optionally installs the SGLang or vLLM framework layer. +2. **Framework install**: optionally installs the SGLang, vLLM or ATOM framework layer. 3. **ROCm hotfix**: applies the profiler hotfix when the ROCm stack is eligible. 4. **Credentials**: resolves LLM gateway credentials into `.env`. 5. **Runtime env**: persists bare-metal runtime vars (framework, ROCm/venv roots, @@ -175,21 +175,16 @@ run: - [`3h`](hyperloom-qwen3-8b-3h/SKILL.md) — Qwen3-8B, short no-kernel run; best for a first end-to-end check. -- [`12h`](hyperloom-qwen3-14b-fp8-12h/SKILL.md) — Qwen3-14B-FP8, medium-length FP8 run. -- [`12h forge`](hyperloom-qwen3-14b-fp8-12h-forge/SKILL.md) — the same run on the - KernelForge kernel backend. -- [`12h atom`](hyperloom-qwen3-14b-fp8-12h-atom/SKILL.md) — the same run on ATOM, - using the user's selected `docker` or `baremetal` mode from setup; if no mode is - selected, ask before continuing. `baremetal` uses the development machine's - existing ATOM/ROCm Python environment, even if that platform is a container - (no additional Docker). Setup uses `--install-framework none - --frameworks atom --require-frameworks`, first with `--check-only` and then - with approval if setup is needed. The KERNEL_AGENT phase runs GEAK, the same - backend the other frameworks get when none is named. -- [`12h atom forge`](hyperloom-qwen3-14b-fp8-12h-atom-forge/SKILL.md) — the ATOM - run with `KERNEL_OPT_BACKEND_ORDER=forge`, so the KERNEL_AGENT phase is the - per-kernel KernelForge backend. Same workload, budget and phase split as - `12h atom`, so the two are directly comparable. +- [`12h`](hyperloom-qwen3-14b-fp8-12h/SKILL.md) — Qwen3-14B-FP8, medium-length FP8 + run on SGLang, vLLM or ATOM. For ATOM it follows its ATOM section, using the + user's selected `docker` or `baremetal` mode from setup; if no mode is selected, + ask before continuing. `baremetal` uses the development machine's ATOM/ROCm + Python environment, even if that platform is a container (no additional + Docker); setup verifies an existing ATOM with `--install-framework none + --frameworks atom --require-frameworks` or, with approval, installs it with + `--install-framework atom`. +- [`12h forge`](hyperloom-qwen3-14b-fp8-12h-forge/SKILL.md) — the same run, on any + of the three frameworks, with the KernelForge kernel backend. - [`custom advanced`](hyperloom-custom-advanced/SKILL.md) — user-selected model, framework, TP/EP, concurrency, ISL/OSL, precision, budget, phase toggles, and advanced CLI flags. diff --git a/examples/hyperloom-qwen3-14b-fp8-12h-atom-forge/SKILL.md b/examples/hyperloom-qwen3-14b-fp8-12h-atom-forge/SKILL.md deleted file mode 100644 index b2fda6d6ce..0000000000 --- a/examples/hyperloom-qwen3-14b-fp8-12h-atom-forge/SKILL.md +++ /dev/null @@ -1,317 +0,0 @@ ---- -name: hyperloom-qwen3-14b-fp8-12h-atom-forge -description: Run a 12-hour Hyperloom Qwen3-14B-FP8 optimization session on ATOM with the per-kernel KernelForge backend instead of GEAK. Use when the user wants the ATOM demo and has asked for the forge kernel backend. ---- - -# Hyperloom Qwen3-14B-FP8 12h Run (ATOM Framework, Forge Kernel Backend) - -Load `.env` with the execution-shell preamble below and resolve -`HYPERLOOM_SKILL_PATH`. Follow `@${HYPERLOOM_SKILL_PATH}`; if unset, use -`@hyperloom/inference_optimizer/SKILL.md` (wheel installation) or -`@src/hyperloom/inference_optimizer/SKILL.md` (source checkout). This ATOM variant -uses the same workload and phase budgets as the -[12h SGLang/vLLM example](../hyperloom-qwen3-14b-fp8-12h/SKILL.md). - -This is the [12h ATOM example](../hyperloom-qwen3-14b-fp8-12h-atom/SKILL.md) with -**one** difference: the KERNEL_AGENT phase runs the per-kernel KernelForge -backend instead of GEAK. The workload, budget, and phase split are identical on -purpose, so the two runs stay directly comparable. - -## Run Mode - -Follow the setup skill's **Run Mode Resolution**, shared with the vLLM/SGLang -workflow. Reuse the user's `HYPERLOOM_RUN_MODE` selection (`baremetal` or `docker`) -from setup, the caller, or `.env`. If no valid selection is available, ask the user -to choose before setup, container creation, or launch. Do not default to either mode -or infer a preference from the framework or a previous validation run. - -- `docker`: run in an ATOM container on the selected development host. -- `baremetal`: run directly in that host's existing ATOM/ROCm Python environment. - A development platform that is itself a container still counts as baremetal - when no additional Docker container is started. - -Export the selected `HYPERLOOM_RUN_MODE` in the execution shell and run only its -matching entry below; both entries use the same Environment and Launch steps. - -### Execution shell - -In the chosen Hyperloom workspace, use the shared loader with caller exports -taking precedence. Repeat this preamble in each new execution shell, including -inside Docker, with the selected `HYPERLOOM_RUN_MODE` exported. Keep -`USER_DATA_PATH` unchanged; the loader handles readonly roots and Docker's host -Python isolation without a new shell. - -```bash -set -e -export REPO_ROOT="$(pwd -P)" -INSTALL_SH="${REPO_ROOT}/hyperloom/inference_optimizer/assets/install.sh" -if [ ! -f "$INSTALL_SH" ]; then - INSTALL_SH="${REPO_ROOT}/src/hyperloom/inference_optimizer/assets/install.sh" -fi -. "${INSTALL_SH%/*}/runtime_env.sh" -load_dotenv_no_clobber -export FRAMEWORK=atom -export KERNEL_OPT_BACKEND_ORDER=forge -export PYTHONPATH="${REPO_ROOT}:${REPO_ROOT}/src:${PYTHONPATH:-}" -``` - -### Baremetal - -After choosing direct execution, select the mode in that execution shell: - -```bash -export HYPERLOOM_RUN_MODE=baremetal -``` - -Activate the existing ATOM environment, or select its executable with `PYTHON`. -Continue with [Environment](#environment) below; do not run any Docker commands. - -### Docker container - -Only after the user selects `HYPERLOOM_RUN_MODE=docker`: use the approved -`HYPERLOOM_DOCKER_TARGET_HOST`, or the current development host if unset. Do not -create containers or run setup/optimize on a login host. Reuse completed setup -only in the actual execution environment, not a different host Python. - -Suggested MI355X image: `docker.io/rocm/atom-dev:v0.1.7-rc0`; preserve an explicit -`HYPERLOOM_IMAGE`. Choose an image compatible with the target GPU. -After approval, mount the workspace at the same absolute path. Add matching -mounts for `USER_DATA_PATH` and any model directory outside the workspace: - -```bash -export HYPERLOOM_RUN_MODE=docker -export REPO_ROOT="$(pwd -P)" -export HYPERLOOM_IMAGE="${HYPERLOOM_IMAGE:-docker.io/rocm/atom-dev:v0.1.7-rc0}" -docker run -d \ - --name "${HYPERLOOM_CONTAINER_NAME:-hyperloom-local}" \ - --shm-size "${HYPERLOOM_SHM_SIZE:-64g}" \ - --entrypoint tail \ - --device /dev/kfd \ - --device /dev/dri \ - --group-add video \ - -v "$REPO_ROOT:$REPO_ROOT" \ - "$HYPERLOOM_IMAGE" \ - -f /dev/null -``` - -Enter the container and feed the shared steps below to Bash. This non-TTY form -forwards exported selections by name without printing their values; use `-it` -instead of `-i` for an interactive terminal. Add any other required shell-only -credential/provider variables to the list. Mounted `.env` supplies missing values. -Do not forward host `PYTHON`, `PATH`, or venv settings: select the container's -existing ATOM environment after entry. - -```bash -set -e -_atom_container_env=() -for _atom_name in HYPERLOOM_RUN_MODE USER_DATA_PATH MODEL_PATH KERNEL_OPT_BACKEND_ORDER CLAW_SESSION_ID \ - FORGE_AGENT_BACKEND FORGE_AGENT_CLI CLAUDE_MODEL CODEX_MODEL HYPERLOOM_SKILL_PATH \ - ANTHROPIC_API_KEY ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKEN OPENAI_API_KEY OPENAI_BASE_URL; do - if printenv "$_atom_name" > /dev/null; then - _atom_container_env+=(--env "$_atom_name") - fi -done -unset _atom_name -docker exec -i -w "$REPO_ROOT" "${_atom_container_env[@]}" \ - "${HYPERLOOM_CONTAINER_NAME:-hyperloom-local}" bash -``` - -Inside that shell, repeat the [execution-shell preamble](#execution-shell), then -run Environment, Runtime Install, and Launch Requirements below. Repeat the -preamble and Python selection for each new `docker exec` shell. Stop this session's -container only with approval after the run finishes. - -### Prior workload cleanup (required) - -Before replacement launches, follow **IR-1 — Prior workload cleanup gate** in the -packaged optimizer skill, on the direct environment or Docker target as applicable. -ATOM worker command lines can omit the server entrypoint. Identify this session's -PIDs, process groups, ports and container before proposing cleanup; never broadly -kill Python workers or restart the machine. Check GPU usage before GPU work: - -```bash -rocm-smi --showmemuse -pgrep -af "openai_server|spawn_main" -``` - -## Environment - -Both modes use the following steps in the shell where ATOM will run. - -### Framework - -ATOM must already be installed with ROCm torch. Hyperloom does not install ATOM: -use `--install-framework none --frameworks atom --require-frameworks` below. -This example is single-node, fixes `FRAMEWORK=atom`, and lets Hyperloom detect the -GPU rather than passing `--gpu-type`. - -For additional serving settings, use the optimizer's `--server-args` option. -Do not launch a separate server or export `EXTRA_ATOM_ARGS`: Hyperloom materializes -that transport variable. Keep the initial configuration untuned; do not copy -another GPU's block/KV settings or the final settings of a previous optimization. -Persist `FRAMEWORK=atom` in `.env` only when requested. - -### Kernel Backend - -`KERNEL_OPT_BACKEND_ORDER=forge` in the [execution-shell -preamble](#execution-shell) is the only switch this example needs, and it is what -separates this run from the [12h ATOM -example](../hyperloom-qwen3-14b-fp8-12h-atom/SKILL.md). The opt-in is an **exact** -match on `forge`; every other value, including unset and `forge,geak`, leaves GEAK -owning the whole kernel phase. There is no backend CLI flag, so leaving the export -out silently produces a GEAK run that looks like this demo but is not — confirm the -value is actually set in the launching shell and report it before launch. - -Forge is included in Hyperloom; do not clone it or set `FORGE_PATH`. Preserve the -selected agent provider and `FORGE_AGENT_CLI`; the runtime installer prepares and -checks that CLI unconditionally, because the backend is chosen per session. - -### Selected Python and Setup - -Activate the existing ATOM venv or select an explicit `PYTHON`; otherwise use -`python3` from PATH. Do not create a fresh venv or require `/opt/venv`. - -```bash -export PYTHON="${PYTHON:-$(command -v python3)}" -export INFERENCE_OPTIMIZER_FORCE_PYTHON=1 -``` - -Setup checks the selected Python, ROCm torch and ATOM import/server readiness. -Run its read-only check even when setup previously completed: - -```bash -"$PYTHON" -m hyperloom.inference_optimizer.setup --check-only -- \ - --install-framework none --frameworks atom --require-frameworks \ - --user-data-path "${USER_DATA_PATH:?USER_DATA_PATH missing}" -``` - -Reuse successful setup in this environment. Only if setup is needed, explain its -changes and obtain approval before running: - -```bash -"$PYTHON" -m hyperloom.inference_optimizer.setup -- \ - --install-framework none --frameworks atom --require-frameworks \ - --user-data-path "${USER_DATA_PATH:?USER_DATA_PATH missing}" --yes -``` - -`none` skips framework installation only: actual setup writes `.env` and may -apply ROCm hotfixes; `--yes` is not user consent. Do not bypass base checks, repeat -onboarding unnecessarily, or install SGLang/vLLM to compensate for missing ATOM. - -### Model - -Ask the operator to choose the existing `MODEL_PATH`, a custom local directory, -or the demo default `Qwen/Qwen3-14B-FP8`. A chosen local directory must contain -`config.json`; resolve the path in the actual execution environment. - -For the demo default, use `${REPO_ROOT}/.cache/hyperloom-models/Qwen3-14B-FP8` when -no local path was selected. Use the selected Python, not an assumed Hugging Face -CLI. If `huggingface_hub` is missing, obtain approval before installing it with -`"$PYTHON" -m pip install huggingface_hub`. - -```bash -export MODEL_PATH="${MODEL_PATH:-${REPO_ROOT}/.cache/hyperloom-models/Qwen3-14B-FP8}" -"$PYTHON" - <<'PY' -import os -from pathlib import Path -from huggingface_hub import snapshot_download - -target = Path(os.environ["MODEL_PATH"]).expanduser() -if (target / "config.json").is_file(): - print(f"Using existing model at {target.resolve()}") -else: - snapshot_download( - repo_id="Qwen/Qwen3-14B-FP8", - local_dir=str(target), - ) -print(target.resolve()) -PY -``` - -## Pre-launch Runtime Install - -Prepare runtime in the same environment as ATOM, using the existing installer. -Reuse a prepared runtime; installation needs approval because it installs -dependencies and may start services. If shell and setup-written `.env` disagree -on `USER_DATA_PATH`, reconcile the selected root first: the installer treats -setup's `.env` as authoritative. Never silently replace the artifact root. - -Use the execution-shell preamble and selected Python above, then run: - -```bash -: "${USER_DATA_PATH:?USER_DATA_PATH missing}" "${PYTHON:?Select the existing ATOM Python first}" -export USER_DATA_PATH -ulimit -Sn 65536 || true -bash "$INSTALL_SH" -``` - -For every launch, including resume, the optimizer loads `kernel-agent.env.sh` -in process. Startup preflight checks the selected framework. Do not source the -generated runtime file in -the shell. Readiness failures require diagnosis and approval for repairs, not a provider switch. - -## Launch Requirements - -Use the packaged optimizer skill's **Launch a New Optimization** instructions to -prepare `RUN_LOG`, `PID_FILE`, `LAUNCH_INFO_FILE` and run-scoped metadata in this -execution environment. Keep the selected ATOM Python and backend for launch. - -### First launch - -After setup and runtime preparation, use this complete command for both modes. -Do not replace workload flags with environment-only settings or add -`--no-framework-agent` / `--no-kernel`: - -```bash -set -e -: "${PYTHON:?PYTHON missing}" "${MODEL_PATH:?MODEL_PATH missing}" -: "${RUN_LOG:?RUN_LOG missing}" "${LAUNCH_INFO_FILE:?LAUNCH_INFO_FILE missing}" -"$PYTHON" -m hyperloom.inference_optimizer.cli --verbose optimize \ - --model "$MODEL_PATH" \ - --framework atom \ - --tp 1 --conc 64 --isl 1024 --osl 1024 \ - --precision fp8 \ - --target-gain 50 --max-hours 12 \ - --max-minutes-framework-pct 0.43 --max-minutes-kernel-pct 0.42 \ - --launch-info-file "$LAUNCH_INFO_FILE" \ - > "$RUN_LOG" 2>&1 < /dev/null -``` - -Detach through the existing harness-aware launch path: when `CLAW_SESSION_ID` is -set and the Bash tool supports `run_in_background`, use it without shell-level -detachment; otherwise prefix the optimize command with `setsid nohup` and append -`&`. Follow the packaged skill's separate health check and launch-info/PID -reconciliation; a shell wrapper PID is not the optimizer PID. Do not create a -second launcher or watchdog for baremetal. - -### Resume - -After the first launch, never start another fresh `optimize` to recover a failed -run. Diagnose it, obtain explicit approval, and pass `--resume-from "$SESSION_DIR"` -for that same session instead of a new model launch. Repeat the execution-shell -preamble and Python selection, retain phase fractions `.43/.42`, and follow IR-1 -for any remaining ATOM workers before relaunching. Do not assume resume rewrites -launch-info: verify the current process and session state rather than using an old -PID. A final `stop_reason` ends the run; do not automatically restart it. - -## User-visible Progress - -Follow the packaged skill's **Monitoring** and **Report Back To User** rules. -Report the chosen mode, Python, model, framework/backend, workload and artifact -root before launch, then the real PID, session directory, log/launch-info paths -and initial health check. Never print credentials or custom header values. - -Confirm the recorded framework/backend immediately, not just the shell exports: - -```bash -grep -o '"framework": *"[^"]*"' "$SESSION_DIR/state.json" -grep -o '"kernel_optimizer": *"[^"]*"' "$SESSION_DIR/state.json" -``` - -Expect `atom` and `forge`. A recorded `geak` means the export did not reach the -launching shell: report the mismatch without silently replacing the session. On -requested checks report process state, phase, accepted throughput/gain and the -latest business outcome, not just heartbeats. - -At completion, report final throughput/gain and accuracy evidence, the final -report path, stop reason, incomplete phases, and this session's process/GPU state. diff --git a/examples/hyperloom-qwen3-14b-fp8-12h-atom/SKILL.md b/examples/hyperloom-qwen3-14b-fp8-12h-atom/SKILL.md deleted file mode 100644 index bd9538a7d8..0000000000 --- a/examples/hyperloom-qwen3-14b-fp8-12h-atom/SKILL.md +++ /dev/null @@ -1,301 +0,0 @@ ---- -name: hyperloom-qwen3-14b-fp8-12h-atom -description: Run a 12-hour Hyperloom Qwen3-14B-FP8 optimization session on ATOM in an existing development environment or Docker. ---- - -# Hyperloom Qwen3-14B-FP8 12h Run (ATOM Framework) - -Load `.env` with the execution-shell preamble below and resolve -`HYPERLOOM_SKILL_PATH`. Follow `@${HYPERLOOM_SKILL_PATH}`; if unset, use -`@hyperloom/inference_optimizer/SKILL.md` (wheel installation) or -`@src/hyperloom/inference_optimizer/SKILL.md` (source checkout). This ATOM variant -uses the same workload and phase budgets as the -[12h SGLang/vLLM example](../hyperloom-qwen3-14b-fp8-12h/SKILL.md). - -## Run Mode - -Follow the setup skill's **Run Mode Resolution**, shared with the vLLM/SGLang -workflow. Reuse the user's `HYPERLOOM_RUN_MODE` selection (`baremetal` or `docker`) -from setup, the caller, or `.env`. If no valid selection is available, ask the user -to choose before setup, container creation, or launch. Do not default to either mode -or infer a preference from the framework or a previous validation run. - -- `docker`: run in an ATOM container on the selected development host. -- `baremetal`: run directly in that host's existing ATOM/ROCm Python environment. - A development platform that is itself a container still counts as baremetal - when no additional Docker container is started. - -Export the selected `HYPERLOOM_RUN_MODE` in the execution shell and run only its -matching entry below; both entries use the same Environment and Launch steps. - -### Execution shell - -In the chosen Hyperloom workspace, use the shared loader with caller exports -taking precedence. Repeat this preamble in each new execution shell, including -inside Docker, with the selected `HYPERLOOM_RUN_MODE` exported. Keep -`USER_DATA_PATH` unchanged; the loader handles readonly roots and Docker's host -Python isolation without a new shell. - -```bash -set -e -export REPO_ROOT="$(pwd -P)" -INSTALL_SH="${REPO_ROOT}/hyperloom/inference_optimizer/assets/install.sh" -if [ ! -f "$INSTALL_SH" ]; then - INSTALL_SH="${REPO_ROOT}/src/hyperloom/inference_optimizer/assets/install.sh" -fi -. "${INSTALL_SH%/*}/runtime_env.sh" -load_dotenv_no_clobber -export FRAMEWORK=atom -export PYTHONPATH="${REPO_ROOT}:${REPO_ROOT}/src:${PYTHONPATH:-}" -``` - -### Baremetal - -After choosing direct execution, select the mode in that execution shell: - -```bash -export HYPERLOOM_RUN_MODE=baremetal -``` - -Activate the existing ATOM environment, or select its executable with `PYTHON`. -Continue with [Environment](#environment) below; do not run any Docker commands. - -### Docker container - -Only after the user selects `HYPERLOOM_RUN_MODE=docker`: use the approved -`HYPERLOOM_DOCKER_TARGET_HOST`, or the current development host if unset. Do not -create containers or run setup/optimize on a login host. Reuse completed setup -only in the actual execution environment, not a different host Python. - -Suggested MI355X image: `docker.io/rocm/atom-dev:v0.1.7-rc0`; preserve an explicit -`HYPERLOOM_IMAGE`. Choose an image compatible with the target GPU. -After approval, mount the workspace at the same absolute path. Add matching -mounts for `USER_DATA_PATH` and any model directory outside the workspace: - -```bash -export HYPERLOOM_RUN_MODE=docker -export REPO_ROOT="$(pwd -P)" -export HYPERLOOM_IMAGE="${HYPERLOOM_IMAGE:-docker.io/rocm/atom-dev:v0.1.7-rc0}" -docker run -d \ - --name "${HYPERLOOM_CONTAINER_NAME:-hyperloom-local}" \ - --shm-size "${HYPERLOOM_SHM_SIZE:-64g}" \ - --entrypoint tail \ - --device /dev/kfd \ - --device /dev/dri \ - --group-add video \ - -v "$REPO_ROOT:$REPO_ROOT" \ - "$HYPERLOOM_IMAGE" \ - -f /dev/null -``` - -Enter the container and feed the shared steps below to Bash. This non-TTY form -forwards exported selections by name without printing their values; use `-it` -instead of `-i` for an interactive terminal. Add any other required shell-only -credential/provider variables to the list. Mounted `.env` supplies missing values. -Do not forward host `PYTHON`, `PATH`, or venv settings: select the container's -existing ATOM environment after entry. - -```bash -set -e -_atom_container_env=() -for _atom_name in HYPERLOOM_RUN_MODE USER_DATA_PATH MODEL_PATH KERNEL_OPT_BACKEND_ORDER CLAW_SESSION_ID \ - FORGE_AGENT_BACKEND FORGE_AGENT_CLI CLAUDE_MODEL CODEX_MODEL HYPERLOOM_SKILL_PATH \ - ANTHROPIC_API_KEY ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKEN OPENAI_API_KEY OPENAI_BASE_URL; do - if printenv "$_atom_name" > /dev/null; then - _atom_container_env+=(--env "$_atom_name") - fi -done -unset _atom_name -docker exec -i -w "$REPO_ROOT" "${_atom_container_env[@]}" \ - "${HYPERLOOM_CONTAINER_NAME:-hyperloom-local}" bash -``` - -Inside that shell, repeat the [execution-shell preamble](#execution-shell), then -run Environment, Runtime Install, and Launch Requirements below. Repeat the -preamble and Python selection for each new `docker exec` shell. Stop this session's -container only with approval after the run finishes. - -### Prior workload cleanup (required) - -Before replacement launches, follow **IR-1 — Prior workload cleanup gate** in the -packaged optimizer skill, on the direct environment or Docker target as applicable. -ATOM worker command lines can omit the server entrypoint. Identify this session's -PIDs, process groups, ports and container before proposing cleanup; never broadly -kill Python workers or restart the machine. Check GPU usage before GPU work: - -```bash -rocm-smi --showmemuse -pgrep -af "openai_server|spawn_main" -``` - -## Environment - -Both modes use the following steps in the shell where ATOM will run. - -### Framework - -ATOM must already be installed with ROCm torch. Hyperloom does not install ATOM: -use `--install-framework none --frameworks atom --require-frameworks` below. -This example is single-node, fixes `FRAMEWORK=atom`, and lets Hyperloom detect the -GPU rather than passing `--gpu-type`. - -For additional serving settings, use the optimizer's `--server-args` option. -Do not launch a separate server or export `EXTRA_ATOM_ARGS`: Hyperloom materializes -that transport variable. Keep the initial configuration untuned; do not copy -another GPU's block/KV settings or the final settings of a previous optimization. -Persist `FRAMEWORK=atom` in `.env` only when requested. - -The KERNEL_AGENT phase runs GEAK, the same backend every other framework gets -when the operator names none. Preserve an explicit `KERNEL_OPT_BACKEND_ORDER` -from the caller or `.env` rather than clearing it; for the per-kernel KernelForge -backend use the [12h ATOM forge -example](../hyperloom-qwen3-14b-fp8-12h-atom-forge/SKILL.md). - -### Selected Python and Setup - -Activate the existing ATOM venv or select an explicit `PYTHON`; otherwise use -`python3` from PATH. Do not create a fresh venv or require `/opt/venv`. - -```bash -export PYTHON="${PYTHON:-$(command -v python3)}" -export INFERENCE_OPTIMIZER_FORCE_PYTHON=1 -``` - -Setup checks the selected Python, ROCm torch and ATOM import/server readiness. -Run its read-only check even when setup previously completed: - -```bash -"$PYTHON" -m hyperloom.inference_optimizer.setup --check-only -- \ - --install-framework none --frameworks atom --require-frameworks \ - --user-data-path "${USER_DATA_PATH:?USER_DATA_PATH missing}" -``` - -Reuse successful setup in this environment. Only if setup is needed, explain its -changes and obtain approval before running: - -```bash -"$PYTHON" -m hyperloom.inference_optimizer.setup -- \ - --install-framework none --frameworks atom --require-frameworks \ - --user-data-path "${USER_DATA_PATH:?USER_DATA_PATH missing}" --yes -``` - -`none` skips framework installation only: actual setup writes `.env` and may -apply ROCm hotfixes; `--yes` is not user consent. Do not bypass base checks, repeat -onboarding unnecessarily, or install SGLang/vLLM to compensate for missing ATOM. - -### Model - -Ask the operator to choose the existing `MODEL_PATH`, a custom local directory, -or the demo default `Qwen/Qwen3-14B-FP8`. A chosen local directory must contain -`config.json`; resolve the path in the actual execution environment. - -For the demo default, use `${REPO_ROOT}/.cache/hyperloom-models/Qwen3-14B-FP8` when -no local path was selected. Use the selected Python, not an assumed Hugging Face -CLI. If `huggingface_hub` is missing, obtain approval before installing it with -`"$PYTHON" -m pip install huggingface_hub`. - -```bash -export MODEL_PATH="${MODEL_PATH:-${REPO_ROOT}/.cache/hyperloom-models/Qwen3-14B-FP8}" -"$PYTHON" - <<'PY' -import os -from pathlib import Path -from huggingface_hub import snapshot_download - -target = Path(os.environ["MODEL_PATH"]).expanduser() -if (target / "config.json").is_file(): - print(f"Using existing model at {target.resolve()}") -else: - snapshot_download( - repo_id="Qwen/Qwen3-14B-FP8", - local_dir=str(target), - ) -print(target.resolve()) -PY -``` - -## Pre-launch Runtime Install - -Prepare runtime in the same environment as ATOM, using the existing installer. -Reuse a prepared runtime; installation needs approval because it installs -dependencies and may start services. If shell and setup-written `.env` disagree -on `USER_DATA_PATH`, reconcile the selected root first: the installer treats -setup's `.env` as authoritative. Never silently replace the artifact root. - -Use the execution-shell preamble and selected Python above, then run: - -```bash -: "${USER_DATA_PATH:?USER_DATA_PATH missing}" "${PYTHON:?Select the existing ATOM Python first}" -export USER_DATA_PATH -ulimit -Sn 65536 || true -bash "$INSTALL_SH" -``` - -For every launch, including resume, the optimizer loads `kernel-agent.env.sh` -in process. Startup preflight checks the selected framework. Do not source the -generated runtime file in -the shell. Readiness failures require diagnosis and approval for repairs, not a provider switch. - -## Launch Requirements - -Use the packaged optimizer skill's **Launch a New Optimization** instructions to -prepare `RUN_LOG`, `PID_FILE`, `LAUNCH_INFO_FILE` and run-scoped metadata in this -execution environment. Keep the selected ATOM Python and backend for launch. - -### First launch - -After setup and runtime preparation, use this complete command for both modes. -Do not replace workload flags with environment-only settings or add -`--no-framework-agent` / `--no-kernel`: - -```bash -set -e -: "${PYTHON:?PYTHON missing}" "${MODEL_PATH:?MODEL_PATH missing}" -: "${RUN_LOG:?RUN_LOG missing}" "${LAUNCH_INFO_FILE:?LAUNCH_INFO_FILE missing}" -"$PYTHON" -m hyperloom.inference_optimizer.cli --verbose optimize \ - --model "$MODEL_PATH" \ - --framework atom \ - --tp 1 --conc 64 --isl 1024 --osl 1024 \ - --precision fp8 \ - --target-gain 50 --max-hours 12 \ - --max-minutes-framework-pct 0.43 --max-minutes-kernel-pct 0.42 \ - --launch-info-file "$LAUNCH_INFO_FILE" \ - > "$RUN_LOG" 2>&1 < /dev/null -``` - -Detach through the existing harness-aware launch path: when `CLAW_SESSION_ID` is -set and the Bash tool supports `run_in_background`, use it without shell-level -detachment; otherwise prefix the optimize command with `setsid nohup` and append -`&`. Follow the packaged skill's separate health check and launch-info/PID -reconciliation; a shell wrapper PID is not the optimizer PID. Do not create a -second launcher or watchdog for baremetal. - -### Resume - -After the first launch, never start another fresh `optimize` to recover a failed -run. Diagnose it, obtain explicit approval, and pass `--resume-from "$SESSION_DIR"` -for that same session instead of a new model launch. Repeat the execution-shell -preamble and Python selection, retain phase fractions `.43/.42`, and follow IR-1 -for any remaining ATOM workers before relaunching. Do not assume resume rewrites -launch-info: verify the current process and session state rather than using an old -PID. A final `stop_reason` ends the run; do not automatically restart it. - -## User-visible Progress - -Follow the packaged skill's **Monitoring** and **Report Back To User** rules. -Report the chosen mode, Python, model, framework/backend, workload and artifact -root before launch, then the real PID, session directory, log/launch-info paths -and initial health check. Never print credentials or custom header values. - -Confirm the recorded framework/backend immediately, not just the shell exports: - -```bash -grep -o '"framework": *"[^"]*"' "$SESSION_DIR/state.json" -grep -o '"kernel_optimizer": *"[^"]*"' "$SESSION_DIR/state.json" -``` - -Expect `atom` and `geak`. Report mismatches without silently replacing the -session. On requested checks report process state, phase, accepted -throughput/gain and the latest business outcome, not just heartbeats. - -At completion, report final throughput/gain and accuracy evidence, the final -report path, stop reason, incomplete phases, and this session's process/GPU state. diff --git a/examples/hyperloom-qwen3-14b-fp8-12h-forge/SKILL.md b/examples/hyperloom-qwen3-14b-fp8-12h-forge/SKILL.md index 2664b2701d..3d8b9eb5bc 100644 --- a/examples/hyperloom-qwen3-14b-fp8-12h-forge/SKILL.md +++ b/examples/hyperloom-qwen3-14b-fp8-12h-forge/SKILL.md @@ -1,6 +1,6 @@ --- name: hyperloom-qwen3-14b-fp8-12h-forge -description: Run a 12-hour Hyperloom Qwen3-14B-FP8 optimization session with the per-kernel KernelForge backend instead of GEAK. Use when the user wants the medium-length Hyperloom demo and has asked for the forge kernel backend. +description: Run a 12-hour Hyperloom Qwen3-14B-FP8 optimization session on SGLang, vLLM or ATOM with the per-kernel KernelForge backend instead of GEAK. Use when the user wants the medium-length Hyperloom demo and has asked for the forge kernel backend. --- # Hyperloom Qwen3-14B-FP8 12h Run (Forge Kernel Backend) @@ -10,7 +10,8 @@ Load `.env` with the preamble below and resolve `HYPERLOOM_SKILL_PATH`. Read and This is the [`hyperloom-qwen3-14b-fp8-12h`](../hyperloom-qwen3-14b-fp8-12h/SKILL.md) demo with **one** difference: the KERNEL_AGENT phase runs the per-kernel KernelForge backend instead of GEAK. The workload, budget, and phase split are -identical on purpose, so the two runs stay directly comparable. +identical on purpose, so the two runs stay directly comparable. SGLang and vLLM +follow the sections below; for ATOM (`FRAMEWORK=atom`) follow [ATOM](#atom). ## Execution Shell @@ -286,3 +287,23 @@ and the stop reason. Never print API keys, tokens, or custom header values. 8. Inspect persisted state on requested status checks; report when work stops. 9. After diagnosing an unexpected crash and obtaining explicit resume approval, only run `optimize --resume-from "$SESSION_DIR"` against the same session dir, with `KERNEL_OPT_BACKEND_ORDER=forge` still set. After the first launch, never start a new `optimize`; that creates a new `` session and is forbidden. 10. If `stop_reason` in the current session `state.json` is final, stop and exit. + +## ATOM + +For `FRAMEWORK=atom`, run the +[ATOM section of the 12h demo](../hyperloom-qwen3-14b-fp8-12h/SKILL.md#atom) +(run mode, execution shell, Docker or baremetal entry, setup, runtime install, +first launch, resume and status checks) in place of the generic steps above, with +one addition: right after the ATOM execution-shell preamble, in every shell that +runs the runtime installer or `optimize` (inside the container in Docker mode), +select this demo's backend: + +```bash +export KERNEL_OPT_BACKEND_ORDER=forge +``` + +That section's Docker entry already forwards `KERNEL_OPT_BACKEND_ORDER` into the +container. Everything in [Kernel Backend](#kernel-backend) about the exact +`forge` opt-in and `FORGE_*` knobs applies. After launch, expect `atom` and +`forge` in `state.json`; a recorded `geak` means the export did not reach the +launching shell, so report the mismatch without silently replacing the session. diff --git a/examples/hyperloom-qwen3-14b-fp8-12h/SKILL.md b/examples/hyperloom-qwen3-14b-fp8-12h/SKILL.md index 4572c4a0b4..325d192f59 100644 --- a/examples/hyperloom-qwen3-14b-fp8-12h/SKILL.md +++ b/examples/hyperloom-qwen3-14b-fp8-12h/SKILL.md @@ -1,12 +1,20 @@ --- name: hyperloom-qwen3-14b-fp8-12h -description: Run a 12-hour Hyperloom Qwen3-14B-FP8 optimization session. Use when the user wants a medium-length Hyperloom demo on the local AMD ROCm environment. +description: Run a 12-hour Hyperloom Qwen3-14B-FP8 optimization session on SGLang, vLLM or ATOM. Use when the user wants a medium-length Hyperloom demo on the local AMD ROCm environment. --- # Hyperloom Qwen3-14B-FP8 12h Run Load `.env` with the preamble below and resolve `HYPERLOOM_SKILL_PATH`. Read and follow the optimizer skill at `@${HYPERLOOM_SKILL_PATH}` before launching. If `HYPERLOOM_SKILL_PATH` is missing, fall back to `@hyperloom/inference_optimizer/SKILL.md` (wheel install) or `@src/hyperloom/inference_optimizer/SKILL.md` (source checkout). This skill provides the concrete workload and launch constraints for a 12-hour Qwen3-14B-FP8 demo. +The serving framework is `sglang`, `vllm` or `atom`. SGLang and vLLM follow the +sections below as written. For ATOM (`FRAMEWORK=atom`, or the user asked for +ATOM, including a Docker run whose container has not been detected yet), use the +[ATOM](#atom) section for the execution shell, run mode, setup, runtime install +and launch command; the workload flags in [Environment](#environment), the model +choice, [User-visible Progress](#user-visible-progress) and +[Launch Requirements](#launch-requirements) apply to ATOM unchanged. + ## Execution Shell Run this in the current Hyperloom workspace before setup or runtime installation. @@ -54,6 +62,7 @@ Suggested Docker images: - `vllm`: `docker.io/rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0` - `sglang` MI300X: `docker.io/lmsysorg/sglang-rocm:v0.5.20-rocm10-mi30x-20260920` - `sglang` MI355X: `docker.io/lmsysorg/sglang-rocm:v0.5.20-rocm10-mi35x-20260920` +- `atom`: see [ATOM Docker container](#atom-docker-container) In Docker mode, start a long-running container on `HYPERLOOM_DOCKER_TARGET_HOST` (or the current host when it is unset) before running setup or optimize: @@ -105,7 +114,7 @@ docker stop "${HYPERLOOM_CONTAINER_NAME:-hyperloom-local}" ## Environment - `MODEL_PATH=` -- `FRAMEWORK=` +- `FRAMEWORK=` - `GPU_TYPE=` Required optimize CLI flags: @@ -229,3 +238,268 @@ and the stop reason. Never print API keys, tokens, or custom header values. 7. Inspect persisted state on requested status checks; report when work stops. 8. After diagnosing an unexpected crash and obtaining explicit resume approval, only run `optimize --resume-from "$SESSION_DIR"` against the same session dir. After the first launch, never start a new `optimize`; that creates a new `` session and is forbidden. 9. If `stop_reason` in the current session `state.json` is final, stop and exit. + +## ATOM + +ATOM is an AMD out-of-tree serving engine that Hyperloom launches as +`python3 -m atom.entrypoints.openai_server`. The subsections below replace the +generic execution shell, run mode, setup, runtime install and launch command for +`FRAMEWORK=atom`; everything else above applies unchanged. ATOM is single-node +(IR-8 in the optimizer skill). The KERNEL_AGENT phase runs GEAK, the same backend +every other framework gets when the operator names none; preserve an explicit +`KERNEL_OPT_BACKEND_ORDER` from the caller or `.env` rather than clearing it. For +the per-kernel KernelForge backend use the +[12h forge demo](../hyperloom-qwen3-14b-fp8-12h-forge/SKILL.md). + +### ATOM run mode + +Follow the setup skill's **Run Mode Resolution**, shared with the vLLM/SGLang +workflow. Reuse the user's `HYPERLOOM_RUN_MODE` selection (`baremetal` or `docker`) +from setup, the caller, or `.env`. If no valid selection is available, ask the user +to choose before setup, container creation, or launch. Do not default to either mode +or infer a preference from the framework or a previous validation run. + +- `docker`: run in an ATOM container on the selected development host. +- `baremetal`: run directly in that host's ATOM/ROCm Python environment. A + development platform that is itself a container still counts as baremetal + when no additional Docker container is started. + +Export the selected `HYPERLOOM_RUN_MODE` in the execution shell and run only its +matching entry below; both entries use the same setup, runtime install and +launch steps. + +### ATOM execution shell + +In the chosen Hyperloom workspace, use the shared loader with caller exports +taking precedence. Repeat this preamble in each new execution shell, including +inside Docker, with the selected `HYPERLOOM_RUN_MODE` exported. Keep +`USER_DATA_PATH` unchanged; the loader handles readonly roots and Docker's host +Python isolation without a new shell. + +```bash +set -e +export REPO_ROOT="$(pwd -P)" +INSTALL_SH="${REPO_ROOT}/hyperloom/inference_optimizer/assets/install.sh" +if [ ! -f "$INSTALL_SH" ]; then + INSTALL_SH="${REPO_ROOT}/src/hyperloom/inference_optimizer/assets/install.sh" +fi +. "${INSTALL_SH%/*}/runtime_env.sh" +load_dotenv_no_clobber +export FRAMEWORK=atom +export PYTHONPATH="${REPO_ROOT}:${REPO_ROOT}/src:${PYTHONPATH:-}" +``` + +### ATOM baremetal + +After choosing direct execution, select the mode in that execution shell: + +```bash +export HYPERLOOM_RUN_MODE=baremetal +``` + +Activate the existing ATOM environment, or select its executable with `PYTHON`; +if the environment has no ATOM yet, setup can install it (see +[ATOM Python and setup](#atom-python-and-setup)). Do not run any Docker commands. + +### ATOM Docker container + +Only after the user selects `HYPERLOOM_RUN_MODE=docker`: use the approved +`HYPERLOOM_DOCKER_TARGET_HOST`, or the current development host if unset. Do not +create containers or run setup/optimize on a login host. Reuse completed setup +only in the actual execution environment, not a different host Python. + +Suggested MI355X image: `docker.io/rocm/atom-dev:v0.1.7-rc0`; preserve an explicit +`HYPERLOOM_IMAGE`. Choose an image compatible with the target GPU. +After approval, mount the workspace at the same absolute path. Add matching +mounts for `USER_DATA_PATH` and any model directory outside the workspace: + +```bash +export HYPERLOOM_RUN_MODE=docker +export REPO_ROOT="$(pwd -P)" +export HYPERLOOM_IMAGE="${HYPERLOOM_IMAGE:-docker.io/rocm/atom-dev:v0.1.7-rc0}" +docker run -d \ + --name "${HYPERLOOM_CONTAINER_NAME:-hyperloom-local}" \ + --shm-size "${HYPERLOOM_SHM_SIZE:-64g}" \ + --entrypoint tail \ + --device /dev/kfd \ + --device /dev/dri \ + --group-add video \ + -v "$REPO_ROOT:$REPO_ROOT" \ + "$HYPERLOOM_IMAGE" \ + -f /dev/null +``` + +Enter the container and feed the ATOM steps below to Bash. This non-TTY form +forwards exported selections by name without printing their values; use `-it` +instead of `-i` for an interactive terminal. Add any other required shell-only +credential/provider variables to the list. Mounted `.env` supplies missing values. +Do not forward host `PYTHON`, `PATH`, or venv settings: select the container's +existing ATOM environment after entry. + +```bash +set -e +_atom_container_env=() +for _atom_name in HYPERLOOM_RUN_MODE USER_DATA_PATH MODEL_PATH KERNEL_OPT_BACKEND_ORDER CLAW_SESSION_ID \ + FORGE_AGENT_BACKEND FORGE_AGENT_CLI CLAUDE_MODEL CODEX_MODEL HYPERLOOM_SKILL_PATH \ + ANTHROPIC_API_KEY ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKEN OPENAI_API_KEY OPENAI_BASE_URL; do + if printenv "$_atom_name" > /dev/null; then + _atom_container_env+=(--env "$_atom_name") + fi +done +unset _atom_name +docker exec -i -w "$REPO_ROOT" "${_atom_container_env[@]}" \ + "${HYPERLOOM_CONTAINER_NAME:-hyperloom-local}" bash +``` + +Inside that shell, repeat the [ATOM execution shell](#atom-execution-shell) +preamble, then run the setup, runtime install and launch steps below. Repeat the +preamble and Python selection for each new `docker exec` shell. Stop this +session's container only with approval after the run finishes. + +### ATOM prior workload cleanup + +Before replacement launches, follow **IR-1 — Prior workload cleanup gate** in the +packaged optimizer skill, on the direct environment or Docker target as applicable. +ATOM worker command lines can omit the server entrypoint. Identify this session's +PIDs, process groups, ports and container before proposing cleanup; never broadly +kill Python workers or restart the machine. Check GPU usage before GPU work: + +```bash +rocm-smi --showmemuse +pgrep -af "openai_server|spawn_main" +``` + +### ATOM Python and setup + +Activate the existing ATOM venv or select an explicit `PYTHON`; otherwise use +`python3` from PATH. Do not create a fresh venv or require `/opt/venv`. + +```bash +export PYTHON="${PYTHON:-$(command -v python3)}" +export INFERENCE_OPTIMIZER_FORCE_PYTHON=1 +``` + +Setup checks the selected Python, ROCm torch and ATOM import/server readiness. +Run its read-only check even when setup previously completed: + +```bash +"$PYTHON" -m hyperloom.inference_optimizer.setup --check-only -- \ + --install-framework none --frameworks atom --require-frameworks \ + --user-data-path "${USER_DATA_PATH:?USER_DATA_PATH missing}" +``` + +Reuse successful setup in this environment. Only if setup is needed, explain its +changes and obtain approval before running: + +```bash +"$PYTHON" -m hyperloom.inference_optimizer.setup -- \ + --install-framework none --frameworks atom --require-frameworks \ + --user-data-path "${USER_DATA_PATH:?USER_DATA_PATH missing}" --yes +``` + +If the check reports ATOM missing in a baremetal environment and the operator +approves installing it, run setup with `atom` instead of `none`. It installs +AITER and ATOM from source into the selected Python, keeping its ROCm torch. +Setup refuses when SGLang or vLLM also imports from that Python, because those +engines would then load ATOM's plugins; report the refusal and propose a +separate container for ATOM rather than removing the other engine. Do not do +this in Docker mode, where ATOM comes from the image: + +```bash +"$PYTHON" -m hyperloom.inference_optimizer.setup -- \ + --install-framework atom \ + --user-data-path "${USER_DATA_PATH:?USER_DATA_PATH missing}" --yes +``` + +`none` skips framework installation only: actual setup writes `.env` and may +apply ROCm hotfixes; `--yes` is not user consent. Do not bypass base checks, repeat +onboarding unnecessarily, or install SGLang/vLLM to compensate for missing ATOM. + +For additional serving settings, use the optimizer's `--server-args` option. +Do not launch a separate server or export `EXTRA_ATOM_ARGS`: Hyperloom materializes +that transport variable. Keep the initial configuration untuned; do not copy +another GPU's block/KV settings or the final settings of a previous optimization. +Persist `FRAMEWORK=atom` in `.env` only when requested. + +Resolve the model with the decision flow in [Environment](#environment), in the +actual execution environment, running its Python steps with the selected +`"$PYTHON"`. If `huggingface_hub` is missing, obtain approval before installing +it with `"$PYTHON" -m pip install huggingface_hub`. + +### ATOM runtime install + +Prepare runtime in the same environment as ATOM, using the existing installer. +Reuse a prepared runtime; installation needs approval because it installs +dependencies and may start services. If shell and setup-written `.env` disagree +on `USER_DATA_PATH`, reconcile the selected root first: the installer treats +setup's `.env` as authoritative. Never silently replace the artifact root. + +Use the ATOM execution-shell preamble and selected Python above, then run: + +```bash +: "${USER_DATA_PATH:?USER_DATA_PATH missing}" "${PYTHON:?Select the existing ATOM Python first}" +export USER_DATA_PATH +ulimit -Sn 65536 || true +bash "$INSTALL_SH" +``` + +For every launch, including resume, the optimizer loads `kernel-agent.env.sh` +in process. Startup preflight checks the selected framework. Do not source the +generated runtime file in the shell. Readiness failures require diagnosis and +approval for repairs, not a provider switch. + +### ATOM first launch + +Use the packaged optimizer skill's **Launch a New Optimization** instructions to +prepare `RUN_LOG`, `PID_FILE`, `LAUNCH_INFO_FILE` and run-scoped metadata in this +execution environment. Keep the selected ATOM Python and backend for launch. +After setup and runtime preparation, use this complete command for both modes. +Do not replace workload flags with environment-only settings or add +`--no-framework-agent` / `--no-kernel`: + +```bash +set -e +: "${PYTHON:?PYTHON missing}" "${MODEL_PATH:?MODEL_PATH missing}" +: "${RUN_LOG:?RUN_LOG missing}" "${LAUNCH_INFO_FILE:?LAUNCH_INFO_FILE missing}" +"$PYTHON" -m hyperloom.inference_optimizer.cli --verbose optimize \ + --model "$MODEL_PATH" \ + --framework atom \ + --tp 1 --conc 64 --isl 1024 --osl 1024 \ + --precision fp8 \ + --target-gain 50 --max-hours 12 \ + --max-minutes-framework-pct 0.43 --max-minutes-kernel-pct 0.42 \ + --launch-info-file "$LAUNCH_INFO_FILE" \ + > "$RUN_LOG" 2>&1 < /dev/null +``` + +Detach through the existing harness-aware launch path: when `CLAW_SESSION_ID` is +set and the Bash tool supports `run_in_background`, use it without shell-level +detachment; otherwise prefix the optimize command with `setsid nohup` and append +`&`. Follow the packaged skill's separate health check and launch-info/PID +reconciliation; a shell wrapper PID is not the optimizer PID. Do not create a +second launcher or watchdog for baremetal. + +### ATOM resume + +After the first launch, never start another fresh `optimize` to recover a failed +run. Diagnose it, obtain explicit approval, and pass `--resume-from "$SESSION_DIR"` +for that same session instead of a new model launch. Repeat the ATOM +execution-shell preamble and Python selection, retain phase fractions `.43/.42`, +and follow the ATOM prior workload cleanup for any remaining ATOM workers before +relaunching. Do not assume resume rewrites launch-info: verify the current +process and session state rather than using an old PID. A final `stop_reason` +ends the run; do not automatically restart it. + +### ATOM status checks + +Report the chosen mode and Python together with the launch plan. Confirm the +recorded framework and backend immediately after launch, not just the shell +exports: + +```bash +grep -o '"framework": *"[^"]*"' "$SESSION_DIR/state.json" +grep -o '"kernel_optimizer": *"[^"]*"' "$SESSION_DIR/state.json" +``` + +Expect `atom` and `geak` (or the backend the operator explicitly kept). Report +mismatches without silently replacing the session. diff --git a/pyproject.toml b/pyproject.toml index 5995a14b96..7aa7242848 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -213,18 +213,6 @@ exclude = [ ".agents/skills/hyperloom-qwen3-14b-fp8-12h-forge" = ["examples/hyperloom-qwen3-14b-fp8-12h-forge/SKILL.md"] ".claude/skills/hyperloom-qwen3-14b-fp8-12h-forge" = ["examples/hyperloom-qwen3-14b-fp8-12h-forge/SKILL.md"] ".cursor/skills/hyperloom-qwen3-14b-fp8-12h-forge" = ["examples/hyperloom-qwen3-14b-fp8-12h-forge/SKILL.md"] -".agents/skills/hyperloom-qwen3-14b-fp8-12h-atom" = ["examples/hyperloom-qwen3-14b-fp8-12h-atom/SKILL.md"] -".claude/skills/hyperloom-qwen3-14b-fp8-12h-atom" = ["examples/hyperloom-qwen3-14b-fp8-12h-atom/SKILL.md"] -".cursor/skills/hyperloom-qwen3-14b-fp8-12h-atom" = ["examples/hyperloom-qwen3-14b-fp8-12h-atom/SKILL.md"] -".agents/skills/hyperloom-qwen3-14b-fp8-12h-atom-forge" = [ - "examples/hyperloom-qwen3-14b-fp8-12h-atom-forge/SKILL.md", -] -".claude/skills/hyperloom-qwen3-14b-fp8-12h-atom-forge" = [ - "examples/hyperloom-qwen3-14b-fp8-12h-atom-forge/SKILL.md", -] -".cursor/skills/hyperloom-qwen3-14b-fp8-12h-atom-forge" = [ - "examples/hyperloom-qwen3-14b-fp8-12h-atom-forge/SKILL.md", -] ".agents/skills/hyperloom-custom-advanced" = ["examples/hyperloom-custom-advanced/SKILL.md"] ".claude/skills/hyperloom-custom-advanced" = ["examples/hyperloom-custom-advanced/SKILL.md"] ".cursor/skills/hyperloom-custom-advanced" = ["examples/hyperloom-custom-advanced/SKILL.md"] diff --git a/src/hyperloom/agents/kernel/tests/test_source_resolution_artifact.py b/src/hyperloom/agents/kernel/tests/test_source_resolution_artifact.py index c0f23d0134..25d6f63875 100644 --- a/src/hyperloom/agents/kernel/tests/test_source_resolution_artifact.py +++ b/src/hyperloom/agents/kernel/tests/test_source_resolution_artifact.py @@ -147,6 +147,40 @@ def test_projection_classifies_each_resolution_tier(): assert by_id["k5"]["method"] != ksc.METHOD_LLM +def test_active_finder_non_patchable_verdict_is_written(tmp_path): + """A symbol-detected non-patchable kernel names no source, so it must not be recorded as located.""" + out = tmp_path / ksc.SOURCE_RESOLUTION_FILENAME + written = tl.write_source_resolution_artifact( + [ + { + "kernel_id": "k1", + "name": "a", + "gpu_pct": 5.0, + "source_file": "/x/a.py", + "source_resolution_method": "trace_python_stack", + }, + { + "kernel_id": "k2", + "name": "asm_gemm", + "gpu_pct": 4.0, + "source_file": "", + "source_resolution_method": "active_finder", + "source_resolution_reason": "non-patchable kernel (symbol-detected)", + "skip_reason": "non-patchable kernel name marker", + }, + ], + out, + framework="atom", + ) + assert written == out + doc = json.loads(out.read_text(encoding="utf-8")) + assert ksc.validate_document(doc) == [] + by_id = {e["kernel_id"]: e for e in doc["entries"]} + assert by_id["k2"]["method"] == ksc.METHOD_UNRESOLVED + assert by_id["k2"]["reason"] == "non-patchable kernel (symbol-detected)" + assert by_id["k2"]["reason_class"] == ksc.CLASS_NON_PATCHABLE_NAME + + def test_written_artifact_satisfies_its_own_contract(tmp_path): out = tmp_path / ksc.SOURCE_RESOLUTION_FILENAME tl.write_source_resolution_artifact( diff --git a/src/hyperloom/agents/kernel/tests/test_tracelens_csv.py b/src/hyperloom/agents/kernel/tests/test_tracelens_csv.py index 7fdd3a498c..987c3e6c00 100644 --- a/src/hyperloom/agents/kernel/tests/test_tracelens_csv.py +++ b/src/hyperloom/agents/kernel/tests/test_tracelens_csv.py @@ -1193,6 +1193,65 @@ async def _fake_query(*, prompt, options): assert res.runner == "claude_agent_sdk" +@pytest.mark.parametrize( + ("tool_idle_env", "expected_ceiling_ms"), + [(None, "3600000"), ("900", "900000"), ("0", "0")], +) +def test_run_tracelens_skill_aligns_cli_background_wait_with_tool_idle_bound( + tmp_path, monkeypatch, tool_idle_env, expected_ceiling_ms +): + """The Claude CLI must not kill background sub-agents before the runner's own in-flight bound does.""" + import asyncio + from dataclasses import dataclass + from typing import Any + + @dataclass + class _TextBlock: + text: str + + @dataclass + class _Message: + content: list[Any] + + class _FakeOptions: + def __init__(self, **kwargs): + self.kwargs = kwargs + + monkeypatch.delenv("CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS", raising=False) + if tool_idle_env is None: + monkeypatch.delenv("HYPERLOOM_TRACELENS_TOOL_IDLE_TIMEOUT_SEC", raising=False) + else: + monkeypatch.setenv("HYPERLOOM_TRACELENS_TOOL_IDLE_TIMEOUT_SEC", tool_idle_env) + output_dir = tmp_path / "out" + captured: dict[str, Any] = {} + + async def _fake_query(*, prompt, options): + captured["options"] = options.kwargs + output_dir.mkdir(parents=True, exist_ok=True) + (output_dir / "analysis.md").write_text("# report\n", encoding="utf-8") + yield _Message(content=[_TextBlock("done")]) + + asyncio.run( + tlr.run_tracelens_skill( + skill_path=tmp_path / "skill.md", + trace_path=tmp_path / "trace.json.gz", + output_dir=output_dir, + tracelens_root=tmp_path, + tracelens_internal_root=tmp_path / "TraceLens-internal", + platform="MI355X", + framework="vllm", + analysis_mode="inference", + capture_folder=None, + budget_minutes=1, + model="claude-sonnet-4-5-20250929", + sdk_query_factory=_fake_query, + sdk_options_cls=_FakeOptions, + ) + ) + + assert captured["options"]["env"]["CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS"] == expected_ceiling_ms + + def test_run_tracelens_skill_books_its_requests_on_the_trajectory(tmp_path): import asyncio from dataclasses import dataclass diff --git a/src/hyperloom/agents/kernel/tools/tracelens_analysis.py b/src/hyperloom/agents/kernel/tools/tracelens_analysis.py index 29462890b6..6535e26be0 100755 --- a/src/hyperloom/agents/kernel/tools/tracelens_analysis.py +++ b/src/hyperloom/agents/kernel/tools/tracelens_analysis.py @@ -5353,11 +5353,16 @@ def _candidate_resolution_method(item: dict[str, Any]) -> str: The candidate carries the method only when a tier stamped one; a path that arrived from grep has none, and an absent path means nothing resolved it. + A tier that ruled on the kernel without locating it (the active finder's + non-patchable verdict) is not a location method, so it maps to unresolved; + its verdict survives in ``reason`` and ``reason_class``. """ stamped = str(item.get("source_resolution_method") or "").strip() - if stamped in getattr(_KSC, "KNOWN_METHODS", frozenset()): + has_source = bool(str(item.get("source_file") or "").strip()) + pathless = {getattr(_KSC, "METHOD_UNRESOLVED", "unresolved"), getattr(_KSC, "METHOD_REJECTED", "")} + if stamped in getattr(_KSC, "KNOWN_METHODS", frozenset()) and (has_source or stamped in pathless): return stamped - if str(item.get("source_file") or "").strip(): + if has_source: # No tier claimed it but a path is present: grep is the only tier that # resolves without stamping. return getattr(_KSC, "METHOD_GREP", "name_grep") diff --git a/src/hyperloom/agents/kernel/tools/tracelens_skill_runner.py b/src/hyperloom/agents/kernel/tools/tracelens_skill_runner.py index b2c3fd03d4..9427b0c549 100644 --- a/src/hyperloom/agents/kernel/tools/tracelens_skill_runner.py +++ b/src/hyperloom/agents/kernel/tools/tracelens_skill_runner.py @@ -158,6 +158,59 @@ def _tool_call_transition(message: Any) -> str | None: return transition +# For SDK builds that do not export TERMINAL_TASK_STATUSES. +_TERMINAL_TASK_STATUSES = frozenset({"completed", "failed", "stopped", "killed", "cancelled"}) + + +def _terminal_task_statuses() -> frozenset[str]: + try: + from claude_agent_sdk import TERMINAL_TASK_STATUSES # type: ignore[import-not-found] + except ImportError: + return _TERMINAL_TASK_STATUSES + return frozenset(str(s) for s in TERMINAL_TASK_STATUSES) + + +class _InFlight: + """Whether the run is waiting on work that emits no SDK message until it ends. + + That is a foreground tool call, or a background task (an async sub-agent) + whose launching tool call already returned. A background task is tracked by + id from ``TaskStartedMessage`` until a ``TaskNotificationMessage`` or a + ``TaskUpdatedMessage`` with a terminal status, which is how the SDK says + active task ids must be cleared. + """ + + def __init__(self) -> None: + self.tool = False + self.tasks: set[str] = set() + + @property + def busy(self) -> bool: + return self.tool or bool(self.tasks) + + def observe(self, message: Any) -> None: + transition = _tool_call_transition(message) + if transition == "start": + self.tool = True + elif transition == "end": + self.tool = False + name = type(message).__name__ + task_id = str(getattr(message, "task_id", "") or "") + if name == "ResultMessage": + self.tasks.clear() + elif not task_id: + return + elif name == "TaskStartedMessage": + self.tasks.add(task_id) + elif name == "TaskNotificationMessage": + self.tasks.discard(task_id) + elif name == "TaskUpdatedMessage": + patch = getattr(message, "patch", None) + status = patch.get("status") if isinstance(patch, dict) else getattr(patch, "status", None) + if str(status or getattr(message, "status", "") or "") in _terminal_task_statuses(): + self.tasks.discard(task_id) + + # Strips a ``Kernel N:`` label prefix from a kernel-name cell piece. _KERNEL_LABEL_RE = re.compile(r"^\s*Kernel\s+\d+\s*:\s*", re.IGNORECASE) @@ -528,15 +581,15 @@ async def _drive_sdk_stream( ) -> str: """Consume the SDK stream, appending its text to ``chunks``; return the SDK error, ``""`` when there was none. - Each next message is bounded by an idle timeout (inactivity, not a total budget), widened while a tool call is in - flight because the SDK emits nothing between a tool's launch and its result. + Each next message is bounded by an idle timeout (inactivity, not a total budget), widened while a tool call or a + background task is in flight because the SDK emits nothing between its launch and its result. """ sdk_error = "" - tool_in_flight = False + in_flight = _InFlight() stream_iter = stream.__aiter__() if hasattr(stream, "__aiter__") else stream try: while True: - wait_for = tool_idle_timeout if tool_in_flight else idle_timeout + wait_for = tool_idle_timeout if in_flight.busy else idle_timeout try: if wait_for > 0: message = await asyncio.wait_for(stream_iter.__anext__(), timeout=wait_for) @@ -549,7 +602,12 @@ async def _drive_sdk_stream( # down so its transport/subprocess does not leak. Name the phase — # silence during a tool call means the tool overran its bound, not # that the gateway died. - phase = "while a tool call was in flight" if tool_in_flight else "with no tool call in flight" + if in_flight.tool: + phase = "while a tool call was in flight" + elif in_flight.tasks: + phase = f"while {len(in_flight.tasks)} background task(s) were running" + else: + phase = "with no tool call in flight" sdk_error = f"stream idle timeout: no SDK message for {wait_for:.0f}s {phase}" if log: log(f"[claude-sdk] WARNING: {sdk_error}") @@ -561,11 +619,7 @@ async def _drive_sdk_stream( pass break observe(message) - transition = _tool_call_transition(message) - if transition == "start": - tool_in_flight = True - elif transition == "end": - tool_in_flight = False + in_flight.observe(message) for text in _iter_message_text(message): chunks.append(text) if log: @@ -785,6 +839,13 @@ async def run_tracelens_skill( operation="analyze_trace", ) ) + idle_timeout = _resolve_stream_idle_timeout_sec() + tool_idle_timeout = _resolve_tool_idle_timeout_sec(idle_timeout) + # In print mode the CLI terminates background sub-agents still running 600s + # after the main turn goes quiet, before the in-flight bound below applies. + kwargs.setdefault("env", {})["CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS"] = os.environ.get( + "CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS", str(int(tool_idle_timeout * 1000)) + ) try: options = sdk_options_cls(**kwargs) @@ -798,13 +859,12 @@ async def run_tracelens_skill( # Drive the SDK stream manually so each next message is bounded by a # per-message idle timeout (inactivity, not a total budget); the in-process # SDK has no client-side read timeout and would otherwise block on a stall. - idle_timeout = _resolve_stream_idle_timeout_sec() with _TrajectoryCall(resolved_model) as trajectory: sdk_error = await _drive_sdk_stream( sdk_query_factory(prompt=prompt, options=options), chunks=chunks, idle_timeout=idle_timeout, - tool_idle_timeout=_resolve_tool_idle_timeout_sec(idle_timeout), + tool_idle_timeout=tool_idle_timeout, observe=trajectory.observe, log=log, ) diff --git a/src/hyperloom/inference_optimizer/assets/install_baremetal.sh b/src/hyperloom/inference_optimizer/assets/install_baremetal.sh index 42136f13b9..546b364054 100755 --- a/src/hyperloom/inference_optimizer/assets/install_baremetal.sh +++ b/src/hyperloom/inference_optimizer/assets/install_baremetal.sh @@ -6,10 +6,10 @@ # # Runs Hyperloom directly on a host that already provides the ROCm framework # base (ROCm runtime + a ROCm-built torch + a serving framework). For bare-metal -# installs, the script can optionally install SGLang or vLLM ROCm framework layers. +# installs, the script can optionally install SGLang, vLLM or ATOM ROCm framework layers. # # Phase 1 base preflight — ROCm / GPU arch / ROCm torch / serving framework -# Phase 2 framework — optional bare-metal SGLang/vLLM install +# Phase 2 framework — optional bare-metal SGLang/vLLM/ATOM install # Phase 3 ROCm hotfix — install ROCclr HIP runtime + roctracer profiler fix # Phase 4 credentials — resolve Anthropic/DeepSeek LLM creds into .env # Phase 5 runtime env — persist bare-metal runtime vars into .env @@ -81,6 +81,10 @@ VLLM_SOURCE_REF="${VLLM_SOURCE_REF:-98dff2a81d747d1dba01a47f939f48c3526d4206}" # The source checkout is a depth-1 fetch of a commit SHA and carries no tags, so # setuptools_scm would otherwise stamp the build 0.1.dev1 instead of the release. VLLM_PRETEND_VERSION="${VLLM_PRETEND_VERSION:-${VLLM_VERSION}}" +# ATOM publishes no release tags. ATOM_REF is the ATOM_COMMIT build argument of +# rocm/atom-dev:v0.1.7-rc0, the only ATOM stack docs/compatibility.rst records. +ATOM_REPO="${ATOM_REPO:-https://github.com/ROCm/ATOM.git}" +ATOM_REF="${ATOM_REF:-fe1099b15ddc52e6873e1932935f722864cdeee0}" VLLM_ROOT="${VLLM_ROOT:-/opt/hyperloom/vllm}" # Index publishing the TheRock ROCm SDK wheels these images are built from; # rocm-sdk-devel is pulled from here to supply source-build headers. @@ -101,8 +105,8 @@ usage() { Usage: src/hyperloom/inference_optimizer/assets/install_baremetal.sh [options] Set up a bare-metal host with ROCm + ROCm torch for Hyperloom. Verifies the base, -optionally installs SGLang/vLLM, resolves credentials, and writes the combined -runtime env. Stops BEFORE launching. +optionally installs SGLang/vLLM/ATOM, resolves credentials, and writes the +combined runtime env. Stops BEFORE launching. Options: --user-data-path PATH Writable artifact root (default: /workspace/hyperloom) @@ -111,7 +115,7 @@ Options: sglang,vllm,atom). Phase 1 passes when at least one entry imports. --install-framework FW Install a missing bare-metal framework layer. - Supported: none, sglang, vllm. Default: none. + Supported: none, sglang, vllm, atom. Default: none. --framework-env MODE Install target for framework packages: shared or isolated. Default: shared, except vLLM which defaults to isolated so it never replaces the shared ROCm @@ -141,7 +145,7 @@ SGLANG_REPO, SGLANG_REF, SGLANG_ROOT, SGLANG_ROCM_PYPI_VERSION, SGLANG_ROCM_EXTRA, SGLANG_BUILD_RUST_EXTS, AITER_REPO, AITER_REF, AITER_ROOT, ROCM_PATH, HIP_PATH, LD_LIBRARY_PATH, ROCM_SDK_INDEX_URL, VLLM_VERSION, VLLM_ROCM_VARIANT, VLLM_ROCM_INDEX, VLLM_INSTALL_METHOD, VLLM_REPO, VLLM_SOURCE_REF, VLLM_ROOT, VLLM_VENV_ROOT, -HYPERLOOM_WHEEL_REPO, HYPERLOOM_WHEEL_TAG. +ATOM_REPO, ATOM_REF, ATOM_ROOT, HYPERLOOM_WHEEL_REPO, HYPERLOOM_WHEEL_TAG. EOF } @@ -155,8 +159,8 @@ while [ "$#" -gt 0 ]; do shift INSTALL_FRAMEWORK="${1:-}" case "$INSTALL_FRAMEWORK" in - none|sglang|vllm) ;; - *) echo "[install-baremetal] ERROR: --install-framework must be one of: none, sglang, vllm" >&2; exit 2 ;; + none|sglang|vllm|atom) ;; + *) echo "[install-baremetal] ERROR: --install-framework must be one of: none, sglang, vllm, atom" >&2; exit 2 ;; esac ;; --framework-env) @@ -229,13 +233,28 @@ framework_probe_python() { fi } +# ATOM registers vllm.platform_plugins, vllm.general_plugins and sglang.srt.plugins +# entry points, and both engines load every registered plugin unless told +# otherwise, so a vLLM or SGLang that can import ATOM serves through ATOM's +# platform, model classes and loader patches. Die before `incoming` would put +# ATOM and either engine into the interpreter `py`. +refuse_atom_beside_vllm_or_sglang() { + local py="$1" incoming="$2" engines="" fw + for fw in sglang vllm; do + if [ "$incoming" = "$fw" ] || _py_has "$py" "$fw"; then engines="${engines:+${engines}, }${fw}"; fi + done + [ -n "$engines" ] || return 0 + [ "$incoming" = atom ] || _py_has "$py" atom || return 0 + die "ATOM cannot share ${py} with ${engines}: those engines load ATOM's plugins by default, which replace their platform, model classes and loader code. Install ATOM in a separate container, or in a Python that imports neither vLLM nor SGLang (pin it with PYTHON and INFERENCE_OPTIMIZER_FORCE_PYTHON=1)." +} + # Print the serving framework to record for downstream skills, or nothing when # none is importable. Walks $FRAMEWORKS in order — the same list Phase 1 probes # — so an engine that passes preflight is always the one written to .env. resolve_installed_framework() { - if [ "$INSTALL_FRAMEWORK" = "sglang" ] || [ "$INSTALL_FRAMEWORK" = "vllm" ]; then - printf '%s' "$INSTALL_FRAMEWORK"; return 0 - fi + case "$INSTALL_FRAMEWORK" in + sglang|vllm|atom) printf '%s' "$INSTALL_FRAMEWORK"; return 0 ;; + esac local py fw probe_py _rif_arr py="$(resolve_python)" || return 0 IFS=',' read -r -a _rif_arr <<< "$FRAMEWORKS" @@ -860,6 +879,7 @@ install_sglang_from_wheel() { install_sglang_framework() { local py deps_root aiter_root py_mm py="$(resolve_python)" || die "no usable Python found for SGLang install" + refuse_atom_beside_vllm_or_sglang "$py" sglang deps_root="$(framework_deps_root)" aiter_root="${AITER_ROOT:-${deps_root}/aiter}" py_mm="$("$py" - <<'PY' @@ -956,6 +976,84 @@ PY log "SGLang framework install complete (SGLANG_USE_AITER=${SGLANG_USE_AITER})" } +# Editable, as in the rocm/atom-dev image, so framework-agent patches land in +# the tree the server imports. The Rust atomesh build stays off (ATOM_MESH_BUILD +# unset): the optimizer launches atom.entrypoints.openai_server, not atomesh. +install_atom_from_source() { + local py="$1" deps_root="$2" atom_root constraint_file + atom_root="${ATOM_ROOT:-${deps_root}/atom}" + + log "installing ATOM from source at ${atom_root} (ref=${ATOM_REF})" + if [ ! -d "${atom_root}/.git" ]; then + mkdir -p "$atom_root" + git init -q "$atom_root" + git -C "$atom_root" remote add origin "$ATOM_REPO" + fi + # Fetch the ref rather than `clone --branch`: ATOM_REF is a commit, which + # GitHub serves directly only as a full 40-char SHA. + git -C "$atom_root" fetch --depth 1 origin "$ATOM_REF" + git -C "$atom_root" checkout -q FETCH_HEAD + + constraint_file="$(mktemp)" + write_rocm_torch_constraints "$py" "$constraint_file" + "$py" -m pip install --constraint "$constraint_file" -e "$atom_root" \ + || { rm -f "$constraint_file"; die "ATOM ${ATOM_REF} install failed under the current torch/triton constraints"; } + rm -f "$constraint_file" +} + +# ATOM runs its kernels through AITER, so AITER goes in first, selected the same +# way as for SGLang: the newest tag compatible with the installed torch/triton. +install_atom_framework() { + local py deps_root aiter_root + py="$(resolve_python)" || die "no usable Python found for ATOM install" + deps_root="$(framework_deps_root)" + aiter_root="${AITER_ROOT:-${deps_root}/aiter}" + refuse_atom_beside_vllm_or_sglang "$py" atom + if vllm_overlay_is_valid "$VLLM_VENV_ROOT"; then + refuse_atom_beside_vllm_or_sglang "${VLLM_VENV_ROOT}/bin/python" atom + fi + + log "Phase 2: installing ATOM framework layer" + log "framework python: ${py}" + log "ATOM source: ${ATOM_REPO}@${ATOM_REF}" + log "AITER_ROOT=${aiter_root}" + if [ -n "$AITER_REF" ]; then + log "AITER_REF=${AITER_REF}" + else + log "AITER_REF=auto (newest tag compatible with installed torch/triton)" + fi + + if [ "$CHECK_ONLY" -eq 1 ]; then + _py_has "$py" atom && log "atom import OK" \ + || warn "atom missing (check-only; would install ${ATOM_REPO}@${ATOM_REF})" + _py_has "$py" aiter && log "aiter import OK" \ + || warn "aiter missing (check-only; would install ${AITER_REF:-the newest compatible tag} from ${AITER_REPO})" + return 0 + fi + + if [ "$DRY_RUN" -eq 1 ]; then + log "would install AITER ${AITER_REF:-(newest compatible tag)} from ${AITER_REPO} at ${aiter_root} with current torch/triton constraints" + log "would clone ${ATOM_REPO}@${ATOM_REF} at ${ATOM_ROOT:-${deps_root}/atom} and install it editable with current torch/triton constraints" + return 0 + fi + + ensure_aiter_for_python "$py" "$aiter_root" + if _py_has "$py" atom; then + log "atom already importable; skipping ATOM source install" + else + install_atom_from_source "$py" "$deps_root" + fi + + "$py" -c "import aiter" >/dev/null || die "aiter not importable after install" + "$py" -c "import atom" >/dev/null || die "atom not importable after install" + # Import the server module rather than run `--help`: ATOM's argparse help + # text carries unescaped "%" (e.g. at fe1099b15), so rendering help raises + # even though the server itself starts and serves. + "$py" -c "import atom.entrypoints.openai.api_server" >/dev/null \ + || die "atom.entrypoints.openai.api_server not importable after install" + log "ATOM framework install complete" +} + # Verify that the installed vLLM package resolves to a ROCm runtime. verify_vllm_rocm() { local py="$1" @@ -1305,6 +1403,10 @@ install_vllm_framework() { local py base_py py_mm constraint_file package_spec rocm_torch_ver base_py="$(resolve_python)" || die "no usable Python found for vLLM install" route_vllm_install_method "$base_py" || return $? + # The source overlay imports the base site-packages; only the wheel venv is self-contained. + if [ "$FRAMEWORK_ENV" = shared ] || [ "$VLLM_INSTALL_METHOD" = source ]; then + refuse_atom_beside_vllm_or_sglang "$base_py" vllm + fi if [ "$VLLM_INSTALL_METHOD" = source ]; then if [ "$FRAMEWORK_ENV" != isolated ]; then die "vLLM source install requires --framework-env isolated" @@ -1451,6 +1553,7 @@ install_requested_framework() { none) log "Phase 2: framework install skipped (--install-framework none)" ;; sglang) install_sglang_framework ;; vllm) install_vllm_framework ;; + atom) install_atom_framework ;; esac } @@ -2609,7 +2712,7 @@ main() { shared|isolated) ;; *) die "FRAMEWORK_ENV must be one of: shared, isolated" ;; esac - if [ "$FRAMEWORK_ENV" = "isolated" ] && [ "$INSTALL_FRAMEWORK" = "sglang" ]; then + if [ "$FRAMEWORK_ENV" = "isolated" ] && { [ "$INSTALL_FRAMEWORK" = "sglang" ] || [ "$INSTALL_FRAMEWORK" = "atom" ]; }; then die "--framework-env isolated is currently supported for vLLM only" fi diff --git a/src/hyperloom/inference_optimizer/cli/preflight.py b/src/hyperloom/inference_optimizer/cli/preflight.py index 362f77bdae..773082b34d 100644 --- a/src/hyperloom/inference_optimizer/cli/preflight.py +++ b/src/hyperloom/inference_optimizer/cli/preflight.py @@ -759,7 +759,7 @@ def _ensure_framework_deps(args, python_exe: str, pip_extra: list[str]) -> dict[ #: Frameworks ``install_baremetal.sh --install-framework`` accepts; it exits 2 on #: anything else. A test asserts this stays equal to the installer's own list. -_SETUP_INSTALLABLE_FRAMEWORKS = frozenset({"sglang", "vllm"}) +_SETUP_INSTALLABLE_FRAMEWORKS = frozenset({"sglang", "vllm", "atom"}) def _setup_install_command(framework: str) -> str: @@ -1022,12 +1022,17 @@ def _check_serving_framework(args, benchmark_python: str) -> dict[str, Any]: interpreters = _framework_probe_interpreters(framework, benchmark_python) found, probe = _resolve_framework_build(framework, interpreters) if framework == "atom" and (not found or probe.verdict is not True): + remedy = ( + "Select the existing ATOM Python and put its bin directory first on PATH, or install ATOM into it:\n" + f" {_setup_install_command(framework)}" + if not found + else "Select an ATOM Python whose torch is a ROCm build and put its bin directory first on PATH." + ) print( f"Preflight: ERROR — atom runtime check failed in python3 ({found or ', '.join(interpreters) or 'not on PATH'}). " "ATOM must import with a ROCm torch build (torch.version.hip) in the selected Python environment." f"{_probe_detail_block(probe.detail)}\n" - "Select the existing ATOM Python and put its bin directory first on PATH; " - "setup does not install ATOM.", + f"{remedy}", file=sys.stderr, ) raise SystemExit(2) diff --git a/src/hyperloom/inference_optimizer/framework_paths.py b/src/hyperloom/inference_optimizer/framework_paths.py index 72361310bb..3365fdc00d 100644 --- a/src/hyperloom/inference_optimizer/framework_paths.py +++ b/src/hyperloom/inference_optimizer/framework_paths.py @@ -23,10 +23,14 @@ import importlib.util import logging import os +import subprocess import sys from collections.abc import Sequence +from dataclasses import dataclass from pathlib import Path +from hyperloom.common.git_safety import safe_directory_args + log = logging.getLogger(__name__) #: Framework-agnostic way to name the source tree a session may patch. Accepted in @@ -64,6 +68,9 @@ #: Backwards-compatible private alias. _FRAMEWORK_PACKAGES: tuple[str, ...] = FRAMEWORK_SOURCE_PACKAGES +#: Frameworks whose Python package is not named after them. +_FRAMEWORK_IMPORT_NAMES: dict[str, str] = {"xdit": "xfuser"} + #: Packages an isolated vLLM venv may hold. Deliberately narrower than #: :data:`FRAMEWORK_SOURCE_PACKAGES`: that tree exists because vLLM needs its #: own interpreter, so only vLLM and the kernel library it links against are @@ -464,6 +471,96 @@ def resolve_session_framework_root() -> str: return generic[0] if generic else "" +def enclosing_checkout(path: str) -> Path | None: + """Return the git checkout that contains ``path``, or ``None``. + + A framework is named by its package dir (``/python/sglang``), not + the repo root, so the checkout is found by walking up from it. ``.git`` may + be a file (worktree) or a dir (repo). + + Args: + path: Directory inside the checkout, or the checkout root itself. + + Returns: + Path | None: The checkout root, or ``None`` when ``path`` is not a + directory inside one. + """ + if not str(path or "").strip(): + return None + p = Path(path) + if not p.is_dir(): + return None + for candidate in (p, *p.parents): + if (candidate / ".git").exists(): + return candidate + return None + + +@dataclass(frozen=True) +class FrameworkTree: + """The tree a session optimises, and where an edit to it lands. + + Attributes: + tree: The tree as the session names it: a checkout, or a package dir. + root: Where patches against the tree apply: the checkout that tracks + ``tree``, or ``tree`` itself when no checkout does, as for a + pip-installed package. The latter is the form the framework source + roots already name such a package in. + checkout: Whether ``root`` is a git checkout tracking ``tree``. + """ + + tree: Path + root: Path + checkout: bool + + +def _tracks(checkout: Path, tree: Path) -> bool: + """Whether ``checkout`` tracks any file under ``tree``. + + A venv inside some unrelated repository sits under that repository's + ``.git`` without being part of it, so a ``.git`` above the tree is not + enough. + """ + try: + rel = tree.resolve().relative_to(checkout.resolve()).as_posix() or "." + except (OSError, ValueError): + return False + try: + cp = subprocess.run( + ["git", *safe_directory_args(["-C", str(checkout), "ls-files", "--error-unmatch", "--", rel])], + capture_output=True, + text=True, + timeout=30, + check=False, + ) + except (OSError, subprocess.SubprocessError): + return False + return cp.returncode == 0 + + +def framework_apply_tree(tree: str) -> FrameworkTree | None: + """Classify the tree a session optimises by where an edit to it lands. + + Args: + tree: The tree as named by :func:`resolve_framework_tree`, a session + root or an explicit declaration. + + Returns: + FrameworkTree | None: The classification, or ``None`` when ``tree`` is + unset or not a directory here. + """ + text = str(tree or "").strip().rstrip("/") + if not text: + return None + path = Path(text) + if not path.is_dir(): + return None + checkout = enclosing_checkout(text) + if checkout is not None and _tracks(checkout, path): + return FrameworkTree(tree=path, root=checkout, checkout=True) + return FrameworkTree(tree=path, root=path, checkout=False) + + def resolve_framework_tree(framework: str) -> str: """Return the source tree belonging to ``framework``, or ``""``. @@ -478,18 +575,18 @@ def resolve_framework_tree(framework: str) -> str: Returns: str: The normalised tree root, or ``""`` when the framework has none. """ - pkg = str(framework or "").strip().lower() - if not pkg: + name = str(framework or "").strip().lower() + if not name: return "" - for key in (f"{pkg.upper()}_REPO_PATH", f"{pkg.upper()}_DIR", GENERIC_FRAMEWORK_ROOT_ENV): + for key in (f"{name.upper()}_REPO_PATH", f"{name.upper()}_DIR", GENERIC_FRAMEWORK_ROOT_ENV): candidate = os.environ.get(key, "").strip() if candidate and Path(candidate).is_dir(): return _normalize_root(candidate) - origin = _find_spec_origin(pkg) + origin = _find_spec_origin(_FRAMEWORK_IMPORT_NAMES.get(name, name)) if origin is not None: return _normalize_root(str(origin)) for default in _DEFAULT_SOURCE_ROOTS: - if default.rstrip("/").endswith(f"/{pkg}") and Path(default).is_dir(): + if Path(default).name.lower() == name and Path(default).is_dir(): return _normalize_root(default) return "" @@ -638,6 +735,8 @@ def resolved_within(value: str, root: str) -> bool: __all__ = [ "FRAMEWORK_SOURCE_PACKAGES", + "FrameworkTree", + "framework_apply_tree", "probe_framework_source_roots_for_env", "resolve_framework_tree", "resolve_kernel_search_roots", diff --git a/src/hyperloom/inference_optimizer/tests/test_dotenv_precedence_recipes.py b/src/hyperloom/inference_optimizer/tests/test_dotenv_precedence_recipes.py index ddc4febf53..5ac200ec08 100644 --- a/src/hyperloom/inference_optimizer/tests/test_dotenv_precedence_recipes.py +++ b/src/hyperloom/inference_optimizer/tests/test_dotenv_precedence_recipes.py @@ -24,7 +24,9 @@ PKG_ROOT = Path(__file__).resolve().parents[1] REPO_ROOT = Path(__file__).resolve().parents[4] -ATOM_DOC = REPO_ROOT / "examples" / "hyperloom-qwen3-14b-fp8-12h-atom" / "SKILL.md" +# ATOM is a framework option of the 12h demo; its procedure lives in that skill's "## ATOM" section. +ATOM_DOC = REPO_ROOT / "examples" / "hyperloom-qwen3-14b-fp8-12h" / "SKILL.md" +FORGE_DOC = REPO_ROOT / "examples" / "hyperloom-qwen3-14b-fp8-12h-forge" / "SKILL.md" SETUP_DOC = REPO_ROOT / "src" / "hyperloom" / "skills" / "hyperloom-setup" / "SKILL.md" # In-package docs ship in the wheel; examples/ only exists in a source checkout. @@ -33,10 +35,8 @@ PKG_ROOT / "references" / "operations.md", REPO_ROOT / "examples" / "hyperloom-custom-advanced" / "SKILL.md", REPO_ROOT / "examples" / "hyperloom-qwen3-8b-3h" / "SKILL.md", - REPO_ROOT / "examples" / "hyperloom-qwen3-14b-fp8-12h" / "SKILL.md", - REPO_ROOT / "examples" / "hyperloom-qwen3-14b-fp8-12h-forge" / "SKILL.md", ATOM_DOC, - REPO_ROOT / "examples" / "hyperloom-qwen3-14b-fp8-12h-atom-forge" / "SKILL.md", + FORGE_DOC, ) CREDENTIAL_ONLY_DOC = REPO_ROOT / "docs" / "how-to" / "optimize-custom-workload.md" @@ -109,33 +109,63 @@ def _run_recipe( return dict(zip(observed, proc.stdout.splitlines()[-len(observed) :])) -def _atom_section_blocks(heading: str) -> list[list[str]]: +def _atom_section_text(heading: str) -> str: + """One ``### heading`` subsection of the 12h demo's ``## ATOM`` section.""" + atom = re.search(r"^## ATOM\n(.*?)(?=^## |\Z)", ATOM_DOC.read_text(encoding="utf-8"), re.MULTILINE | re.DOTALL) + assert atom is not None, "missing ## ATOM section in the 12h demo" section = re.search( rf"^### {re.escape(heading)}\n" + r"(.*?)(?=^#{2,3} |\Z)", - ATOM_DOC.read_text(encoding="utf-8"), + atom.group(1), re.MULTILINE | re.DOTALL, ) assert section is not None, f"missing ATOM section: {heading}" - return _bash_blocks(section.group(1)) + return section.group(1) + + +def _atom_section_blocks(heading: str) -> list[list[str]]: + return _bash_blocks(_atom_section_text(heading)) + + +def _atom_shell() -> str: + return "\n".join(_atom_section_blocks("ATOM execution shell")[0]) def test_atom_run_mode_requires_user_choice() -> None: - text = ATOM_DOC.read_text(encoding="utf-8") - run_mode = " ".join(text.split("## Run Mode\n", 1)[1].split("### Execution shell", 1)[0].split()) + run_mode = " ".join(_atom_section_text("ATOM run mode").split()) assert "Run Mode Resolution" in run_mode assert "ask the user to choose" in run_mode assert "Do not default to either mode" in run_mode - assert "or an unset/empty mode" not in text + assert "or an unset/empty mode" not in ATOM_DOC.read_text(encoding="utf-8") readme = (REPO_ROOT / "examples" / "README.md").read_text(encoding="utf-8") - readme_entry = readme.split("- [`12h atom`]", 1)[1].split("- [`", 1)[0] + readme_entry = readme.split("- [`12h`]", 1)[1].split("- [`", 1)[0] + assert "ATOM" in readme_entry assert "Docker as the default" not in readme_entry +def test_atom_has_no_separate_demo_skills() -> None: + """ATOM is a framework option of the 12h pair, not a separate pair of skills.""" + assert not (REPO_ROOT / "examples" / "hyperloom-qwen3-14b-fp8-12h-atom").exists() + assert not (REPO_ROOT / "examples" / "hyperloom-qwen3-14b-fp8-12h-atom-forge").exists() + pyproject = (REPO_ROOT / "pyproject.toml").read_text(encoding="utf-8") + assert "hyperloom-qwen3-14b-fp8-12h-atom" not in pyproject + setup = SETUP_DOC.read_text(encoding="utf-8") + assert "hyperloom-qwen3-14b-fp8-12h-atom" not in setup + + +def test_atom_forge_demo_opts_into_forge_in_the_atom_shell() -> None: + """The forge demo reuses the 12h ATOM procedure and adds only the backend export.""" + text = FORGE_DOC.read_text(encoding="utf-8") + atom = re.search(r"^## ATOM\n(.*?)(?=^## |\Z)", text, re.MULTILINE | re.DOTALL) + assert atom is not None, "missing ## ATOM section in the 12h forge demo" + assert "../hyperloom-qwen3-14b-fp8-12h/SKILL.md#atom" in atom.group(1) + assert "export KERNEL_OPT_BACKEND_ORDER=forge" in atom.group(1) + + @pytest.mark.parametrize("mode", ["baremetal", "docker"]) def test_atom_first_launch_runs_in_selected_context(tmp_path: Path, mode: str) -> None: """Execute mode/selection/launch blocks up to the optimizer process boundary.""" - mode_blocks = _atom_section_blocks("Baremetal" if mode == "baremetal" else "Docker container") - launch = "\n".join(_atom_section_blocks("First launch")[0]) + mode_blocks = _atom_section_blocks("ATOM baremetal" if mode == "baremetal" else "ATOM Docker container") + launch = "\n".join(_atom_section_blocks("ATOM first launch")[0]) workspace = tmp_path / "workspace with spaces" workspace.mkdir() data = workspace / "user data" @@ -217,8 +247,8 @@ def test_atom_first_launch_runs_in_selected_context(tmp_path: Path, mode: str) - if mode == "docker": payload += '[ -z "${PYTHON:-}" ]\n[ -z "${VIRTUAL_ENV:-}" ]\n' payload += '[ -z "${INFERENCE_OPTIMIZER_FORCE_PYTHON:-}" ]\n[[ "$PATH" != /host-only/bin:* ]]\n' - payload += "\n".join(_atom_section_blocks("Execution shell")[0]) + "\n" - payload += "\n".join(_atom_section_blocks("Selected Python and Setup")[0]) + "\n" + payload += _atom_shell() + "\n" + payload += "\n".join(_atom_section_blocks("ATOM Python and setup")[0]) + "\n" for key, value in {"RUN_LOG": run_log, "LAUNCH_INFO_FILE": launch_info}.items(): payload += f"export {key}={shlex.quote(value.as_posix())}\n" payload += launch + "\n: > launch-finished\n" @@ -290,7 +320,7 @@ def test_atom_first_launch_runs_in_selected_context(tmp_path: Path, mode: str) - @pytest.mark.parametrize("missing", ["MODEL_PATH", "PYTHON", "RUN_LOG", "LAUNCH_INFO_FILE"]) def test_atom_first_launch_requires_prepared_paths(tmp_path: Path, missing: str) -> None: - launch = "\n".join(_atom_section_blocks("First launch")[0]) + launch = "\n".join(_atom_section_blocks("ATOM first launch")[0]) exported = { "MODEL_PATH": (tmp_path / "model files").as_posix(), "PYTHON": "launch_probe", @@ -307,9 +337,9 @@ def test_atom_first_launch_requires_prepared_paths(tmp_path: Path, missing: str) def test_atom_runtime_install_accepts_readonly_user_data_path(tmp_path: Path) -> None: """The install invocation exports the fixed platform path without assigning it.""" - blocks = _bash_blocks(ATOM_DOC.read_text(encoding="utf-8")) + blocks = _atom_section_blocks("ATOM runtime install") install = "\n".join(next(b for b in blocks if 'bash "$INSTALL_SH"' in b)) - fragment = _dotenv_loads(ATOM_DOC)[0] + "\nreadonly USER_DATA_PATH\nbash() { INSTALLER_REACHED=yes; }\n" + install + fragment = _atom_shell() + "\nreadonly USER_DATA_PATH\nbash() { INSTALLER_REACHED=yes; }\n" + install result = _run_recipe( fragment, tmp_path, @@ -391,7 +421,7 @@ def test_atom_docker_recipe_passes_selected_python_to_setup(tmp_path: Path, doc: if selection in {"valid-pin", "invalid-pin"}: pin = python.as_posix() if selection == "valid-pin" else "/explicit-missing/python3" fragment += f"export PYTHON={shlex.quote(pin)} INFERENCE_OPTIMIZER_FORCE_PYTHON=0\n" - fragment += _dotenv_loads(ATOM_DOC)[0] + "\n" + fragment += _atom_shell() + "\n" if selection == "activate-after": fragment += activation blocks = _bash_blocks(doc.read_text(encoding="utf-8")) @@ -444,7 +474,7 @@ def test_atom_recipe_preserves_backend_selection( tmp_path: Path, shell_backend: str | None, dotenv_backend: str | None, expected: str ) -> None: """The workload selection must not discard an explicit kernel backend choice.""" - fragment = "\n".join(_atom_section_blocks("Execution shell")[0]) + fragment = _atom_shell() exported = {"USER_DATA_PATH": tmp_path.as_posix(), "PYTHON": "/selected/bin/python3"} if shell_backend is not None: exported["KERNEL_OPT_BACKEND_ORDER"] = shell_backend @@ -473,6 +503,7 @@ def test_atom_recipe_provides_direct_setup_commands(tmp_path: Path, doc: Path) - for block in blocks if any("hyperloom.inference_optimizer.setup" in line for line in block) and not any("docker" in line for line in block) + and any("--install-framework none" in line for line in block) and (doc == ATOM_DOC or any("--frameworks atom" in line for line in block)) ] assert len(setup_blocks) == 2, "direct mode needs a check-only command and a separate approved setup command" @@ -495,6 +526,33 @@ def test_atom_recipe_provides_direct_setup_commands(tmp_path: Path, doc: Path) - assert ("--yes" in args) == (key == "INSTALL_ARGS") +def test_atom_recipe_provides_an_approved_atom_install_command(tmp_path: Path) -> None: + """The baremetal install path must run with the selected Python and artifact root.""" + blocks = _atom_section_blocks("ATOM Python and setup") + install_blocks = [ + "\n".join(block) + for block in blocks + if any("hyperloom.inference_optimizer.setup" in line for line in block) + and any("--install-framework atom" in line for line in block) + ] + assert len(install_blocks) == 1 + fragment = """ +setup_args=() +python_probe() { setup_args+=("$*"); } +export PYTHON=python_probe +""" + install_blocks[0] + fragment += '\nINSTALL_ARGS="${setup_args[0]}"' + result = _run_recipe(fragment, tmp_path, {"USER_DATA_PATH": "/selected/data"}, observed=("INSTALL_ARGS",)) + + args = result["INSTALL_ARGS"] + assert "-m hyperloom.inference_optimizer.setup" in args + assert "--install-framework atom" in args + assert "--user-data-path /selected/data" in args + assert "--yes" in args + assert "--check-only" not in args + assert "docker" not in install_blocks[0] + + @pytest.mark.parametrize("doc", RECIPE_DOCS, ids=lambda p: p.parent.name + "/" + p.name) @pytest.mark.parametrize("layout", [".", "src"], ids=["wheel", "source"]) def test_recipe_keeps_caller_user_data_path(doc: Path, tmp_path: Path, layout: str) -> None: diff --git a/src/hyperloom/inference_optimizer/tests/test_framework_paths_units.py b/src/hyperloom/inference_optimizer/tests/test_framework_paths_units.py index 6e181b4943..da2e9396ec 100644 --- a/src/hyperloom/inference_optimizer/tests/test_framework_paths_units.py +++ b/src/hyperloom/inference_optimizer/tests/test_framework_paths_units.py @@ -836,3 +836,59 @@ def test_unknown_framework_resolves_to_nothing(self, monkeypatch): def test_empty_name_resolves_to_nothing(self): assert fp.resolve_framework_tree("") == "" + + def test_a_framework_is_found_by_its_import_name(self, monkeypatch, tmp_path): + """xDiT ships as the ``xfuser`` package.""" + xfuser = tmp_path / "xDiT" / "xfuser" + xfuser.mkdir(parents=True) + monkeypatch.delenv("FRAMEWORK_REPO_PATH", raising=False) + monkeypatch.setattr(fp, "_find_spec_origin", lambda name: xfuser if name == "xfuser" else None) + assert fp.resolve_framework_tree("xdit") == f"{xfuser}/" + + def test_a_default_root_matches_its_framework_case_insensitively(self, monkeypatch, tmp_path): + xdit = tmp_path / "app" / "xDiT" + xdit.mkdir(parents=True) + monkeypatch.delenv("FRAMEWORK_REPO_PATH", raising=False) + monkeypatch.setattr(fp, "_find_spec_origin", lambda name: None) + monkeypatch.setattr(fp, "_DEFAULT_SOURCE_ROOTS", (f"{xdit}/",)) + assert fp.resolve_framework_tree("xdit") == f"{xdit}/" + + +def _git_tracking(checkout: Path, *files: str) -> Path: + import subprocess + + subprocess.run(["git", "init", "-q", str(checkout)], check=True) + for rel in files: + target = checkout / rel + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text("x = 1\n", encoding="utf-8") + subprocess.run(["git", "-C", str(checkout), "add", "-A"], check=True) + subprocess.run( + ["git", "-C", str(checkout), "-c", "user.email=t@t", "-c", "user.name=t", "commit", "-qm", "base"], + check=True, + ) + return checkout + + +class TestFrameworkApplyTree: + def test_a_package_in_a_checkout_is_edited_at_the_checkout(self, tmp_path): + checkout = _git_tracking(tmp_path / "sglang", "python/sglang/__init__.py") + tree = fp.framework_apply_tree(f"{checkout}/python/sglang/") + assert tree == fp.FrameworkTree(tree=checkout / "python" / "sglang", root=checkout, checkout=True) + + def test_a_pip_installed_package_is_edited_in_place(self, tmp_path): + package = tmp_path / "site-packages" / "vllm" + package.mkdir(parents=True) + (package / "__init__.py").write_text("", encoding="utf-8") + assert fp.framework_apply_tree(f"{package}/") == fp.FrameworkTree(tree=package, root=package, checkout=False) + + def test_an_untracked_package_under_some_repository_is_not_that_repository(self, tmp_path): + project = _git_tracking(tmp_path / "project", "README.md") + package = project / ".venv" / "site-packages" / "vllm" + package.mkdir(parents=True) + (package / "__init__.py").write_text("", encoding="utf-8") + assert fp.framework_apply_tree(str(package)) == fp.FrameworkTree(tree=package, root=package, checkout=False) + + def test_nothing_named_is_nothing(self, tmp_path): + assert fp.framework_apply_tree("") is None + assert fp.framework_apply_tree(str(tmp_path / "absent")) is None diff --git a/src/hyperloom/inference_optimizer/tests/test_kill_spawned_server.py b/src/hyperloom/inference_optimizer/tests/test_kill_spawned_server.py index f83a5483a5..f2043a3ea5 100644 --- a/src/hyperloom/inference_optimizer/tests/test_kill_spawned_server.py +++ b/src/hyperloom/inference_optimizer/tests/test_kill_spawned_server.py @@ -1187,7 +1187,7 @@ def test_scan_logs_increment_tells_the_childs_own_log_from_the_servers(tmp_path) _scan_logs_increment(passed, offsets) with server_log.open("a") as f: - f.write('INFO: 127.0.0.1:0 - "GET /health HTTP/1.1" 200 OK\n') + f.write("[atom 22:29:41] Scheduled prefill batch: 4 reqs, 3707 new tokens\n") server_only = _scan_logs_increment(passed, offsets) assert server_only.grew is True and server_only.child_spoke is False @@ -1197,6 +1197,30 @@ def test_scan_logs_increment_tells_the_childs_own_log_from_the_servers(tmp_path) assert child_only.grew is True and child_only.child_spoke is True +def test_health_probe_lines_are_not_server_activity(tmp_path): + """An engine that died behind a live HTTP front end still answers /metrics. + + ATOM's ModelRunner crashed on a GPU memory fault while its API server kept + serving the monitor's /metrics polls, so the log grew every few seconds and + the silence gate never fired: lm_eval waited on 64 requests nothing would + serve until the 7800s hard cap. + """ + log_path = tmp_path / "server.log" + log_path.write_text("Application startup complete\n", encoding="utf-8") + offsets: dict[str, int] = {} + _scan_logs_increment(str(log_path), offsets) + + with log_path.open("a", encoding="utf-8") as f: + f.write('INFO: 127.0.0.1:45518 - "GET /metrics HTTP/1.1" 200 OK\n') + f.write('INFO: 127.0.0.1:45520 - "GET /health HTTP/1.1" 200 OK\n') + assert _scan_logs_increment(str(log_path), offsets).grew is False + + with log_path.open("a", encoding="utf-8") as f: + f.write('INFO: 127.0.0.1:45522 - "GET /metrics HTTP/1.1" 200 OK\n') + f.write("[atom 23:29:42] Request 639 arrived, input tokens: 1295, pending requests: 64\n") + assert _scan_logs_increment(str(log_path), offsets).grew is True + + def test_run_with_session_kill_detok_stall_reaps_ready_but_silent_server(tmp_path): """A server that reports ready then produces no generation progress is reaped with ``DETOKENIZER_STALL_RETURNCODE`` well before the hard timeout.""" diff --git a/src/hyperloom/inference_optimizer/tests/test_preflight_serving_framework.py b/src/hyperloom/inference_optimizer/tests/test_preflight_serving_framework.py index 6ef89058d8..1e4e20fbb5 100644 --- a/src/hyperloom/inference_optimizer/tests/test_preflight_serving_framework.py +++ b/src/hyperloom/inference_optimizer/tests/test_preflight_serving_framework.py @@ -569,8 +569,7 @@ def test_in_container_prefers_container_when_no_signal_is_readable(monkeypatch): _SETUP_INSTALLER = "inference_optimizer/assets/install_baremetal.sh" -def test_atom_missing_build_is_blocked_without_an_install_remedy(monkeypatch, capsys): - """ATOM must be usable in its existing environment, not installed by setup.""" +def test_atom_missing_build_is_blocked_with_the_setup_install_remedy(monkeypatch, capsys): _probe_result(monkeypatch, importable=False) monkeypatch.setattr(preflight.shutil, "which", lambda _name: sys.executable) @@ -580,7 +579,7 @@ def test_atom_missing_build_is_blocked_without_an_install_remedy(monkeypatch, ca assert failure.value.code == 2 output = capsys.readouterr() assert "atom" in output.err.lower() - assert "--install-framework atom" not in output.out + output.err + assert preflight._setup_install_command("atom") in output.err @pytest.fixture diff --git a/src/hyperloom/inference_optimizer/tests/test_proposal_identity_unit.py b/src/hyperloom/inference_optimizer/tests/test_proposal_identity_unit.py index 4b183f548b..78cfa560c0 100644 --- a/src/hyperloom/inference_optimizer/tests/test_proposal_identity_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_proposal_identity_unit.py @@ -130,6 +130,29 @@ def test_list_form_args_are_space_joined_not_repr(value, expected): assert normalize_proposal({"extra_args": value})["extra_args"] == expected +@pytest.mark.parametrize( + "value,expected", + [ + ({"--kv-cache-dtype": "fp8_e4m3"}, "--kv-cache-dtype fp8_e4m3"), + ( + {"--mem-fraction-static": 0.9, "--max-running-requests": 128}, + "--mem-fraction-static 0.9 --max-running-requests 128", + ), + ({"--enable-mixed-chunk": True, "--disable-radix-cache": False}, "--enable-mixed-chunk"), + ({"--served-model-name": "a b"}, "--served-model-name 'a b'"), + ], +) +def test_mapping_form_args_render_as_flags_not_repr(value, expected): + assert normalize_proposal({"extra_args": value})["extra_args"] == expected + + +def test_mapping_form_args_pass_the_grid_argv_gate(): + from hyperloom.inference_optimizer.grid_server_args import validate_server_args_shell_safe + + args = normalize_proposal({"extra_args": {"--kv-cache-dtype": "fp8_e4m3"}})["extra_args"] + assert validate_server_args_shell_safe(args) == args + + def test_the_queue_and_the_grid_parser_agree_on_a_list_form_variant(): from hyperloom.orchestrator.actions.executors.explore import _grid_variants_from_payload diff --git a/src/hyperloom/inference_optimizer/tests/test_server_patcher_vllm_isolated_venv.py b/src/hyperloom/inference_optimizer/tests/test_server_patcher_vllm_isolated_venv.py index c2135038bc..d4d22f93c6 100644 --- a/src/hyperloom/inference_optimizer/tests/test_server_patcher_vllm_isolated_venv.py +++ b/src/hyperloom/inference_optimizer/tests/test_server_patcher_vllm_isolated_venv.py @@ -5,6 +5,7 @@ from __future__ import annotations +import sys from pathlib import Path from hyperloom.orchestrator.actions.executors import _server_patcher as sp @@ -56,6 +57,30 @@ def test_isolated_venv_missing_returns_none(tmp_path: Path, monkeypatch): assert sp._discover_vllm_plan(_make_tracelens_root(tmp_path)) is None +def test_isolated_venv_editable_source_install_discovers_plan(tmp_path: Path, monkeypatch): + """A source build installs vLLM editable: site-packages holds only an egg-link and the package lives in the checkout.""" + tracelens_root = _make_tracelens_root(tmp_path) + venv_root = tmp_path / "vllm-venv" + site = venv_root / "lib" / "python3.12" / "site-packages" + site.mkdir(parents=True) + checkout = tmp_path / "vllm-src" + vllm_pkg = checkout / "vllm" + (vllm_pkg / "config").mkdir(parents=True) + (vllm_pkg / "__init__.py").write_text('__version__ = "0.21.0"\n', encoding="utf-8") + (vllm_pkg / "config" / "profiler.py").write_text("# profiler\n", encoding="utf-8") + (site / "vllm.egg-link").write_text(f"{checkout}\n.\n", encoding="utf-8") + monkeypatch.setenv("VLLM_VENV_ROOT", str(venv_root)) + monkeypatch.setenv("VLLM_PYTHON", sys.executable) + monkeypatch.setenv("PYTHONPATH", str(checkout)) + + plan = sp._discover_vllm_plan(tracelens_root) + + assert plan is not None + assert plan.version == "0.21.0" + assert plan.apply_root == checkout.resolve() + assert plan.sentinel_file == plan.apply_root / "vllm" / "config" / "profiler.py" + + def test_isolated_venv_no_profiler_sentinel_returns_none(tmp_path: Path, monkeypatch): tracelens_root = _make_tracelens_root(tmp_path) venv_root = _make_isolated_vllm(tmp_path, with_profiler=False) diff --git a/src/hyperloom/inference_optimizer/tests/test_setup_cli.py b/src/hyperloom/inference_optimizer/tests/test_setup_cli.py index 37a236266c..688d0ee595 100644 --- a/src/hyperloom/inference_optimizer/tests/test_setup_cli.py +++ b/src/hyperloom/inference_optimizer/tests/test_setup_cli.py @@ -1343,6 +1343,382 @@ def test_baremetal_atom_only_host_writes_framework_atom(tmp_path: Path): assert "FRAMEWORK=atom" in _dotenv_lines(dotenv) +_ATOM_DEV_IMAGE_COMMIT = "fe1099b15ddc52e6873e1932935f722864cdeee0" + + +def _logging_python( + tmp_path: Path, + *, + atom_importable: bool, + atom_import_after_install: bool = True, + server_module_imports: bool = True, +) -> Path: + """A python stub that records every invocation and answers the ATOM probes.""" + log = tmp_path / "python-calls.log" + stub = tmp_path / "logging-python" + stub.write_text( + "\n".join( + [ + "#!/usr/bin/env bash", + f'printf "%s\\n" "$*" >> "{log.as_posix()}"', + 'case "$*" in', + f""" *"find_spec('atom')"*) exit {0 if atom_importable else 1} ;;""", + " *find_spec*) exit 1 ;;", + f' "-c import atom") exit {0 if atom_import_after_install else 1} ;;', + f' "-c import atom.entrypoints.openai.api_server") exit {0 if server_module_imports else 1} ;;', + # ATOM's own --help crashes: its argparse help text carries an unescaped "%". + ' *"atom.entrypoints.openai_server --help"*) exit 1 ;;', + "esac", + "exit 0", + ] + ) + + "\n", + encoding="utf-8", + newline="\n", + ) + stub.chmod(0o755) + return stub + + +def _drive_atom_installer(tmp_path: Path, python: Path, body: str) -> subprocess.CompletedProcess: + install_script = Path(setup.__file__).resolve().parent / "assets" / "install_baremetal.sh" + lib = _sourceable_installer(install_script, tmp_path) + runner = tmp_path / "atom-runner.sh" + runner.write_text( + "\n".join( + [ + "#!/usr/bin/env bash", + f'source "{lib.as_posix()}"', + f'REPO_ROOT="{tmp_path.as_posix()}"', + f'HYPERLOOM_CACHE_DIR="{tmp_path.as_posix()}/deps"', + "INSTALL_FRAMEWORK=atom", + "FRAMEWORK_ENV=shared", + f'VLLM_VENV_ROOT="{tmp_path.as_posix()}/absent"', + "DRY_RUN=0", + "CHECK_ONLY=0", + f'resolve_python() {{ printf "%s" "{python.as_posix()}"; }}', + body, + ] + ) + + "\n", + encoding="utf-8", + newline="\n", + ) + return subprocess.run(["bash", runner.as_posix()], text=True, capture_output=True) + + +def test_installer_accepts_install_framework_atom(): + """Argument parsing must not reject atom before --help is reached.""" + install_script = Path(setup.__file__).resolve().parent / "assets" / "install_baremetal.sh" + + res = subprocess.run( + ["bash", install_script.as_posix(), "--install-framework", "atom", "--help"], + text=True, + capture_output=True, + ) + + assert res.returncode == 0, res.stderr + assert "none, sglang, vllm, atom" in res.stdout + + +def test_install_requested_framework_dispatches_atom(tmp_path: Path): + res = _drive_installer( + tmp_path, + importable=set(), + dotenv=tmp_path / ".env", + install_framework="atom", + body="install_atom_framework() { echo ATOM_INSTALLER_CALLED; }\ninstall_requested_framework", + ) + + assert res.returncode == 0, res.stderr + assert "ATOM_INSTALLER_CALLED" in res.stdout + + +def test_atom_install_defaults_to_the_atom_dev_image_commit(tmp_path: Path): + """The default ATOM source is the commit rocm/atom-dev:v0.1.7-rc0 was built from.""" + python = _logging_python(tmp_path, atom_importable=False) + + res = _drive_atom_installer(tmp_path, python, "DRY_RUN=1\ninstall_atom_framework") + + assert res.returncode == 0, res.stderr + assert f"https://github.com/ROCm/ATOM.git@{_ATOM_DEV_IMAGE_COMMIT}" in res.stdout + assert "AITER" in res.stdout + + +def test_atom_check_only_reports_a_missing_atom_without_installing(tmp_path: Path): + python = _logging_python(tmp_path, atom_importable=False) + + res = _drive_atom_installer( + tmp_path, + python, + "CHECK_ONLY=1\n" + "ensure_aiter_for_python() { echo STEP aiter; }\n" + "install_atom_from_source() { echo STEP atom-source; }\n" + "install_atom_framework", + ) + + assert res.returncode == 0, res.stderr + assert "STEP" not in res.stdout + assert "atom missing" in res.stderr + + +def test_atom_install_puts_aiter_first_and_verifies_the_server(tmp_path: Path): + python = _logging_python(tmp_path, atom_importable=False) + + res = _drive_atom_installer( + tmp_path, + python, + "ensure_aiter_for_python() { echo STEP aiter; }\n" + "install_atom_from_source() { echo STEP atom-source; }\n" + "install_atom_framework", + ) + + assert res.returncode == 0, res.stderr + steps = [line for line in res.stdout.splitlines() if line.startswith("STEP ")] + assert steps == ["STEP aiter", "STEP atom-source"] + calls = (tmp_path / "python-calls.log").read_text(encoding="utf-8") + assert "-c import atom.entrypoints.openai.api_server" in calls + + +def test_atom_install_does_not_depend_on_atom_help_rendering(tmp_path: Path): + """ATOM fe1099b15 crashes printing --help, yet its server imports and serves.""" + python = _logging_python(tmp_path, atom_importable=False) + + res = _drive_atom_installer( + tmp_path, + python, + "ensure_aiter_for_python() { :; }\ninstall_atom_from_source() { :; }\ninstall_atom_framework", + ) + + assert res.returncode == 0, res.stderr + assert "ATOM framework install complete" in res.stdout + + +def test_atom_install_fails_when_the_server_module_does_not_import(tmp_path: Path): + python = _logging_python(tmp_path, atom_importable=False, server_module_imports=False) + + res = _drive_atom_installer( + tmp_path, + python, + "ensure_aiter_for_python() { :; }\ninstall_atom_from_source() { :; }\ninstall_atom_framework", + ) + + assert res.returncode != 0 + assert "atom.entrypoints.openai.api_server" in res.stderr + + +def test_atom_install_fails_when_atom_still_does_not_import(tmp_path: Path): + python = _logging_python(tmp_path, atom_importable=False, atom_import_after_install=False) + + res = _drive_atom_installer( + tmp_path, + python, + "ensure_aiter_for_python() { :; }\ninstall_atom_from_source() { :; }\ninstall_atom_framework", + ) + + assert res.returncode != 0 + assert "atom not importable after install" in res.stderr + + +def test_atom_install_skips_the_source_build_when_atom_already_imports(tmp_path: Path): + python = _logging_python(tmp_path, atom_importable=True) + + res = _drive_atom_installer( + tmp_path, + python, + "ensure_aiter_for_python() { echo STEP aiter; }\n" + "install_atom_from_source() { echo STEP atom-source; }\n" + "install_atom_framework", + ) + + assert res.returncode == 0, res.stderr + assert "STEP atom-source" not in res.stdout + + +def test_atom_source_install_fetches_the_pinned_commit_under_rocm_torch_constraints(tmp_path: Path): + """Without the constraint file pip is free to replace ROCm torch with a PyPI CUDA build.""" + python = _logging_python(tmp_path, atom_importable=False) + git_log = tmp_path / "git-calls.log" + + res = _drive_atom_installer( + tmp_path, + python, + "\n".join( + [ + f'git() {{ printf "%s\\n" "$*" >> "{git_log.as_posix()}"; }}', + "write_rocm_torch_constraints() { printf 'torch==2.11.0\\n' > \"$2\"; }", + f'install_atom_from_source "{python.as_posix()}" "{tmp_path.as_posix()}/deps"', + ] + ), + ) + + assert res.returncode == 0, res.stderr + git_calls = git_log.read_text(encoding="utf-8") + assert f"fetch --depth 1 origin {_ATOM_DEV_IMAGE_COMMIT}" in git_calls + pip_calls = [ + line + for line in (tmp_path / "python-calls.log").read_text(encoding="utf-8").splitlines() + if "pip install" in line + ] + atom_install = [line for line in pip_calls if f"{tmp_path.as_posix()}/deps/atom" in line] + assert atom_install, pip_calls + assert "--constraint" in atom_install[0] + assert " -e " in f" {atom_install[0]} " + + +def test_atom_install_records_framework_atom_when_frameworks_omits_it(tmp_path: Path): + """The engine the operator asked setup to install is the one downstream skills must use.""" + dotenv = tmp_path / ".env" + + res = _drive_installer( + tmp_path, + importable={"atom"}, + dotenv=dotenv, + install_framework="atom", + body="FRAMEWORKS=sglang,vllm\nwrite_runtime_dotenv", + ) + + assert res.returncode == 0, res.stderr + assert "FRAMEWORK=atom" in _dotenv_lines(dotenv) + + +def test_atom_install_rejects_an_isolated_framework_env(tmp_path: Path, monkeypatch): + monkeypatch.setenv("REPO_ROOT", tmp_path.as_posix()) + install_script = Path(setup.__file__).resolve().parent / "assets" / "install_baremetal.sh" + + res = subprocess.run( + [ + "bash", + install_script.as_posix(), + "--install-framework", + "atom", + "--framework-env", + "isolated", + "--dry-run", + "--yes", + ], + text=True, + capture_output=True, + ) + + assert res.returncode != 0 + assert "isolated is currently supported for vLLM only" in res.stderr + + +def _vllm_venv(tmp_path: Path, *, system_site_packages: bool) -> Path: + """A VLLM_VENV_ROOT whose python imports vllm; an overlay also sees the base site-packages.""" + root = tmp_path / "vllm-venv" + (root / "bin").mkdir(parents=True) + _fake_python(root / "bin", {"vllm"}).rename(root / "bin" / "python") + (root / "pyvenv.cfg").write_text( + f"include-system-site-packages = {'true' if system_site_packages else 'false'}\n", encoding="utf-8" + ) + return root + + +_ATOM_INSTALL_STEPS = ( + "ensure_aiter_for_python() { echo STEP aiter; }\ninstall_atom_from_source() { echo STEP atom-source; }\n" +) + + +@pytest.mark.parametrize("engine", ["sglang", "vllm"]) +def test_atom_install_refuses_a_python_that_serves_vllm_or_sglang(tmp_path: Path, engine: str): + """vLLM and SGLang load ATOM's entry-point plugins by default, so a shared Python would serve ATOM code.""" + res = _drive_installer( + tmp_path, + importable={engine}, + dotenv=tmp_path / ".env", + install_framework="atom", + body=f"DRY_RUN=1\n{_ATOM_INSTALL_STEPS}install_atom_framework", + ) + + assert res.returncode != 0 + assert "would install" not in res.stdout + assert f"ATOM cannot share {tmp_path.as_posix()}/fake-python with {engine}" in res.stderr + + +def test_atom_install_refuses_a_base_python_under_a_vllm_overlay(tmp_path: Path): + """The ROCm 10 vLLM overlay imports the base site-packages, and with them ATOM's plugins.""" + overlay = _vllm_venv(tmp_path, system_site_packages=True) + + res = _drive_installer( + tmp_path, + importable=set(), + dotenv=tmp_path / ".env", + install_framework="atom", + body=f'VLLM_VENV_ROOT="{overlay.as_posix()}"\n{_ATOM_INSTALL_STEPS}install_atom_framework', + ) + + assert res.returncode != 0 + assert "STEP" not in res.stdout + assert f"ATOM cannot share {overlay.as_posix()}/bin/python with vllm" in res.stderr + + +def test_atom_install_proceeds_beside_a_self_contained_vllm_venv(tmp_path: Path): + venv = _vllm_venv(tmp_path, system_site_packages=False) + + res = _drive_installer( + tmp_path, + importable=set(), + dotenv=tmp_path / ".env", + install_framework="atom", + body=f'VLLM_VENV_ROOT="{venv.as_posix()}"\n{_ATOM_INSTALL_STEPS}install_atom_framework', + ) + + assert res.returncode == 0, res.stderr + assert [line for line in res.stdout.splitlines() if line.startswith("STEP ")] == ["STEP aiter", "STEP atom-source"] + + +_VLLM_ROUTE_STUBS = "assert_vllm_glibc_compatible() { :; }\nensure_openmpi_runtime() { :; }\n" + + +@pytest.mark.parametrize( + ("installer", "framework"), + [ + pytest.param("install_sglang_framework", "sglang", id="sglang"), + pytest.param( + f"{_VLLM_ROUTE_STUBS}route_vllm_install_method() {{ VLLM_INSTALL_METHOD=wheel; }}\ninstall_vllm_framework", + "vllm", + id="vllm-shared", + ), + pytest.param( + "FRAMEWORK_ENV=isolated\n" + f"{_VLLM_ROUTE_STUBS}route_vllm_install_method() {{ VLLM_INSTALL_METHOD=source; }}\ninstall_vllm_framework", + "vllm", + id="vllm-source-overlay", + ), + ], +) +def test_engine_install_refuses_a_python_that_imports_atom(tmp_path: Path, installer: str, framework: str): + """ATOM installed first must not end up inside the interpreter a later vLLM/SGLang serves from.""" + res = _drive_installer( + tmp_path, + importable={"atom"}, + dotenv=tmp_path / ".env", + install_framework=framework, + body=f"DRY_RUN=1\n{installer}", + ) + + assert res.returncode != 0 + assert "would " not in res.stdout + assert f"ATOM cannot share {tmp_path.as_posix()}/fake-python with {framework}" in res.stderr + + +def test_isolated_vllm_wheel_install_proceeds_on_an_atom_host(tmp_path: Path): + """The wheel venv is created without the system site-packages, so it never sees ATOM.""" + res = _drive_installer( + tmp_path, + importable={"atom"}, + dotenv=tmp_path / ".env", + install_framework="vllm", + body="DRY_RUN=1\nFRAMEWORK_ENV=isolated\n" + f"{_VLLM_ROUTE_STUBS}route_vllm_install_method() {{ VLLM_INSTALL_METHOD=wheel; }}\ninstall_vllm_framework", + ) + + assert res.returncode == 0, res.stderr + assert "would create/update isolated vLLM venv" in res.stdout + + def test_baremetal_clears_stale_framework_when_none_importable(tmp_path: Path): """A re-imaged host must not keep pointing at an engine that is gone.""" dotenv = tmp_path / ".env" @@ -2335,6 +2711,7 @@ def test_baremetal_sglang_installs_aiter_when_find_spec_succeeds_but_import_fail "warn() { :; }", 'die() { echo "$*" >&2; exit 99; }', '_py_has() { [ "$2" = aiter ] && return 0; return 0; }', + "refuse_atom_beside_vllm_or_sglang() { :; }", "install_sglang_from_wheel() { :; }", "install_sglang_from_source() { :; }", "sglang_rocm_extra_for_torch() { printf 'rocm724\\n'; }", diff --git a/src/hyperloom/inference_optimizer/tests/test_specialist_runner_helpers_unit.py b/src/hyperloom/inference_optimizer/tests/test_specialist_runner_helpers_unit.py index cdd3de4d81..cfba4ccf88 100644 --- a/src/hyperloom/inference_optimizer/tests/test_specialist_runner_helpers_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_specialist_runner_helpers_unit.py @@ -32,6 +32,22 @@ def _runner(**over): return SpecialistRunner(**kwargs) +def _checkout(path, *files): + """A git checkout tracking ``files``.""" + import subprocess + + subprocess.run(["git", "init", "-q", str(path)], check=True) + for rel in files: + target = path / rel + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text("", encoding="utf-8") + subprocess.run(["git", "-C", str(path), "add", "-A"], check=True) + subprocess.run( + ["git", "-C", str(path), "-c", "user.email=t@t", "-c", "user.name=t", "commit", "-qm", "base"], check=True + ) + return path + + def test_safe_redact(): line = "export ANTHROPIC_API_KEY=redact_me and GITHUB_TOKEN=redact_me_too" out = sr._safe_redact(line) @@ -373,12 +389,8 @@ def test_maybe_setup_worktree_bases_on_the_framework_being_optimised(tmp_path, m one as ``missing_target`` — leaving an env-only proposal that toggled a switch with no code behind it and measured 0.0% five rounds running. """ - aiter = tmp_path / "aiter" - aiter.mkdir() - (aiter / ".git").mkdir() - worldplay = tmp_path / "HY-WorldPlay" - worldplay.mkdir() - (worldplay / ".git").mkdir() + aiter = _checkout(tmp_path / "aiter", "aiter/__init__.py") + worldplay = _checkout(tmp_path / "HY-WorldPlay", "hyvideo/__init__.py") monkeypatch.setenv("WORLDPLAY_REPO_PATH", str(worldplay)) cfg = sr.SpecialistSubprocessConfig( @@ -399,28 +411,54 @@ def _fake_setup(base, worktree_path, branch): ) ) - _wt, base, err = r._maybe_setup_worktree(ctx, workspace=tmp_path) + _wt, source, err = r._maybe_setup_worktree(ctx, workspace=tmp_path) assert err == "" - assert base == worldplay, f"specialist would patch {seen.get('base')}, not the framework" + assert source.root == worldplay and seen["base"] == worldplay, f"specialist would patch {seen.get('base')}" -def test_maybe_setup_worktree_falls_back_when_the_framework_is_not_a_checkout(tmp_path, monkeypatch): - """A pip-installed framework must not cost the specialist its isolation.""" - aiter = tmp_path / "aiter" - aiter.mkdir() - (aiter / ".git").mkdir() - monkeypatch.setenv("WORLDPLAY_REPO_PATH", str(tmp_path / "not-a-checkout")) +def test_maybe_setup_worktree_snapshots_a_framework_that_is_not_a_checkout(tmp_path, monkeypatch): + """A pip-installed framework keeps its isolation without borrowing whatever checkout sorts first.""" + aiter = _checkout(tmp_path / "aiter", "aiter/__init__.py") + package = tmp_path / "site-packages" / "worldplay" + package.mkdir(parents=True) + (package / "__init__.py").write_text("", encoding="utf-8") + monkeypatch.setenv("WORLDPLAY_REPO_PATH", str(package)) cfg = sr.SpecialistSubprocessConfig(framework_source_roots=(str(aiter),)) - r = _runner(backend_factory=None, subprocess_config=cfg) - monkeypatch.setattr(sr, "_setup_worktree", lambda base, path, branch: (path, "")) + r = _runner(backend_factory=None, subprocess_config=cfg, session_dir=tmp_path / "session") + seen: dict = {} + + def _fake_setup(base, worktree_path, branch): + seen["base"] = base + return worktree_path, "" + + monkeypatch.setattr(sr, "_setup_worktree", _fake_setup) ctx = SimpleNamespace(task=SimpleNamespace(task_id="t", params={"framework": "worldplay"})) - _wt, base, err = r._maybe_setup_worktree(ctx, workspace=tmp_path) + _wt, source, err = r._maybe_setup_worktree(ctx, workspace=tmp_path) assert err == "" - assert base == aiter + assert source.root == package and not source.checkout + assert seen["base"].parent == tmp_path / "session" / "tree_snapshots" + assert not (package / ".git").exists() + + +def test_maybe_setup_worktree_has_nothing_to_isolate_without_a_named_tree(tmp_path, monkeypatch): + aiter = _checkout(tmp_path / "aiter", "aiter/__init__.py") + monkeypatch.setenv("WORLDPLAY_REPO_PATH", str(tmp_path / "absent")) + monkeypatch.delenv("FRAMEWORK_REPO_PATH", raising=False) + monkeypatch.setattr(sr, "resolve_framework_tree", lambda framework: "") + + cfg = sr.SpecialistSubprocessConfig(framework_source_roots=(str(aiter),)) + r = _runner(backend_factory=None, subprocess_config=cfg) + ctx = SimpleNamespace( + task=SimpleNamespace(task_id="t", params={"framework": "worldplay", "domain": "framework_rewrite_specialist"}) + ) + + wt, source, err = r._maybe_setup_worktree(ctx, workspace=tmp_path) + + assert (wt, source, err) == (None, None, sr.NO_GIT_FRAMEWORK_SOURCE_ROOT) def test_patch_path_within_bases_accepts_sandbox_paths(tmp_path): diff --git a/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess.py b/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess.py index cbc2ff5326..009cfd4843 100644 --- a/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess.py +++ b/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess.py @@ -31,7 +31,6 @@ SpecialistSubprocessConfig, SpecialistSubprocessDispatcher, _build_specialist_env, - _pick_worktree_base, _setup_worktree, ) from hyperloom.orchestrator.loop.sub_agent_runner import RunnerContext @@ -277,9 +276,11 @@ def _make_fake_claude( @pytest.fixture -def fake_framework_repo(tmp_path: Path) -> Path: +def fake_framework_repo(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path: + """The checkout the session optimises, named the way a session names it.""" repo = tmp_path / "framework" init_git_repo(repo) + monkeypatch.setenv("FRAMEWORK_REPO_PATH", str(repo)) return repo @@ -292,6 +293,7 @@ def _make_runner_ctx(task_id: str = "t-spec-1") -> RunnerContext: "domain": "serving_specialist", "gap_canonical_id": "gap.test.example", "max_turns": 2, + "framework": "sglang", }, idempotency_key=task_id, requires_lanes=tuple(), @@ -329,22 +331,55 @@ def test_kb_mcp_tools_not_in_denylist(): assert denylisted_kb_mcp == [], f"stale KB MCP entries in the denylist: {denylisted_kb_mcp}" -def test_pick_worktree_base_picks_first_git_root( +@pytest.mark.asyncio +async def test_worktree_of_a_pip_installed_framework_is_its_snapshot_not_another_checkout( tmp_path: Path, - fake_framework_repo: Path, + monkeypatch: pytest.MonkeyPatch, ): - nonrepo = tmp_path / "not-a-repo" - nonrepo.mkdir() - base = _pick_worktree_base((str(nonrepo), str(fake_framework_repo))) - assert base is not None - assert base.samefile(fake_framework_repo) - - -def test_pick_worktree_base_returns_none_when_no_repo(tmp_path: Path): - nonrepo = tmp_path / "not-a-repo" - nonrepo.mkdir() - base = _pick_worktree_base((str(nonrepo),)) - assert base is None + """With no checkout of its own, the framework still hands its specialist its own code -- never InferenceX's.""" + harness = tmp_path / "InferenceX" + init_git_repo(harness, seed_file="benchmark_lib.sh", seed_text="run\n") + package = tmp_path / "site-packages" / "vllm" + package.mkdir(parents=True) + (package / "__init__.py").write_text("", encoding="utf-8") + (package / "envs.py").write_text("VLLM_USE_X = 0\n", encoding="utf-8") + monkeypatch.setenv("INFERENCEX_PATH", str(harness)) + session_dir = tmp_path / "session" + session_dir.mkdir() + runner = SpecialistRunner( + subprocess_config=SpecialistSubprocessConfig(framework_source_roots=(str(harness), str(package))), + session_dir=session_dir, + ) + ctx = _make_runner_ctx("t-spec-pip") + ctx.task.params["session_framework_tree"] = f"{package}/" + workspace = session_dir / "runs" / "specialist" / "t-spec-pip" + workspace.mkdir(parents=True) + + worktree, source, err = runner._maybe_setup_worktree(ctx, workspace=workspace) + + assert err == "" + assert source is not None and source.root == package and not source.checkout + assert worktree is not None and (worktree / "envs.py").read_text(encoding="utf-8") == "VLLM_USE_X = 0\n" + assert not (worktree / "benchmark_lib.sh").exists() + assert not (package / ".git").exists() + listed = subprocess.run( + ["git", "-C", str(harness), "worktree", "list"], capture_output=True, text=True, check=True + ).stdout + assert str(worktree) not in listed + + +def test_an_integrate_in_flight_holds_the_snapshot(tmp_path: Path): + from hyperloom.orchestrator.state.shared_state import SharedState + + session_dir = tmp_path / "session" + session_dir.mkdir() + runner = SpecialistRunner(subprocess_config=SpecialistSubprocessConfig(), session_dir=session_dir) + assert runner._integrate_in_flight() is False + + state = SharedState.load_or_init(session_dir) + state.pending_integrate = {"task_id": "t-integrate", "patches": []} + state.save(session_dir) + assert runner._integrate_in_flight() is True def test_setup_worktree_creates_branch_off_base( diff --git a/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess_helpers.py b/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess_helpers.py index f614427450..4435579e8d 100644 --- a/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess_helpers.py +++ b/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess_helpers.py @@ -19,9 +19,9 @@ from hyperloom.orchestrator.specialists.subprocess_ import ( SpecialistSubprocessConfig, SpecialistSubprocessDispatcher, - _pick_worktree_base, _setup_worktree, ) +from hyperloom.orchestrator.specialists.tree_snapshot import snapshot_tree class _CP: @@ -31,20 +31,6 @@ def __init__(self, returncode: int = 0, stdout: str = "", stderr: str = "") -> N self.stderr = stderr -# -- _pick_worktree_base --------------------------------------------------- -def test_pick_worktree_base_none(tmp_path: Path) -> None: - # directory without .git yields None - (tmp_path / "plain").mkdir() - assert _pick_worktree_base((str(tmp_path / "plain"), str(tmp_path / "absent"))) is None - - -def test_pick_worktree_base_finds_git(tmp_path: Path) -> None: - repo = tmp_path / "repo" - repo.mkdir() - (repo / ".git").mkdir() - assert _pick_worktree_base(("/nonexistent", str(repo))) == repo - - # -- _setup_worktree ------------------------------------------------------- def test_setup_worktree_reuses_existing(tmp_path: Path) -> None: wt = tmp_path / "wt" @@ -246,6 +232,19 @@ def test_collect_patches_work_artifacts_only_produces_no_delivery(tmp_path: Path assert (wt / rel).read_bytes() == content +def test_collect_patches_heartbeat_in_worktree_produces_no_delivery(tmp_path: Path) -> None: + """A heartbeat the agent writes at the worktree top is its liveness signal, not a change to the tree.""" + base, wt = _make_harvest_worktree(tmp_path) + heartbeat = wt / "heartbeat.json" + heartbeat.write_text('{"ts": "2026-09-28T19:12:00Z", "status": "running"}\n', encoding="utf-8") + + assert SpecialistSubprocessDispatcher._collect_patches(wt, tmp_path / "ws", worktree_base=base) == ([], {}) + assert not (wt / "patches" / "_worktree_diff.patch").exists() + subprocess.run(["git", "-C", str(wt), "add", "-N", "heartbeat.json"], check=True) + diff = subprocess.run(["git", "-C", str(wt), "diff", "HEAD"], check=True, capture_output=True, text=True).stdout + assert ps.patch_work_artifact_targets(diff) == ("heartbeat.json",) + + def test_collect_patches_annotation_only_produces_no_delivery(tmp_path: Path) -> None: base, wt = _make_harvest_worktree(tmp_path) (wt / "runtime.py").write_text("# Runtime selection, unchanged.\nBLOCK_SIZE = 64\n", encoding="utf-8") @@ -275,6 +274,70 @@ def test_collect_patches_fallback_cannot_deliver_work_artifact_patch(tmp_path: P assert manual.read_text(encoding="utf-8") == diff +def _installed_package(tmp_path: Path) -> Path: + """A pip-installed package: source, a shared object and a bytecode cache, no git anywhere.""" + package = tmp_path / "site-packages" / "vllm" + (package / "model_executor" / "models").mkdir(parents=True) + (package / "__init__.py").write_text("", encoding="utf-8") + (package / "envs.py").write_text("VLLM_USE_X = 0\n", encoding="utf-8") + (package / "model_executor" / "models" / "qwen3.py").write_text("BLOCK = 64\n", encoding="utf-8") + (package / "_C.abi3.so").write_bytes(b"\x7fELF shared object") + (package / "__pycache__").mkdir() + (package / "__pycache__" / "envs.cpython-314.pyc").write_bytes(b"\x00bytecode") + return package + + +def test_harvest_off_an_installed_package_snapshot_applies_to_the_package(tmp_path: Path) -> None: + """The worktree of a pip-installed framework holds the package itself, so its diff lands where the server imports.""" + package = _installed_package(tmp_path) + repo, err = snapshot_tree(package, tmp_path / "session" / "tree_snapshots" / "vllm.git", refresh=True) + assert err == "" and repo is not None + wt, err = _setup_worktree(repo, tmp_path / "ws" / "worktree", "specialist-t1") + assert err == "" and wt is not None + assert (wt / "model_executor" / "models" / "qwen3.py").read_text(encoding="utf-8") == "BLOCK = 64\n" + assert not (wt / "_C.abi3.so").exists() + assert not (wt / "__pycache__").exists() + + (wt / "model_executor" / "models" / "qwen3.py").write_text("BLOCK = 128\n", encoding="utf-8") + (wt / "envs.py").write_text("VLLM_USE_X = 1\n", encoding="utf-8") + patches, roots = SpecialistSubprocessDispatcher._collect_patches( + wt, tmp_path / "ws", worktree_base=package, worktree_base_commit=ss.head_commit(wt) + ) + + assert len(patches) == 1 + assert roots == {patches[0]: str(package)} + patch_text = Path(patches[0]).read_text(encoding="utf-8") + assert set(ps.parse_patch_targets(patch_text).all) == {"envs.py", "model_executor/models/qwen3.py"} + assert ps.ground_patch_text(patch_text, base_checkout=None, explicit_root=package).verdict == ps.GROUND_APPLIES + assert not (package / ".git").exists() + assert (package / "model_executor" / "models" / "qwen3.py").read_text(encoding="utf-8") == "BLOCK = 64\n" + + +def test_snapshot_refresh_follows_the_package_and_a_held_snapshot_does_not(tmp_path: Path) -> None: + """A KEEP lands in the package between dispatches; an integrate still deciding on a candidate must not.""" + package = _installed_package(tmp_path) + repo = tmp_path / "session" / "tree_snapshots" / "vllm.git" + snapshot_tree(package, repo, refresh=True) + first = ss.head_commit(repo) + + (package / "envs.py").write_text("VLLM_USE_X = 2\n", encoding="utf-8") + held, err = snapshot_tree(package, repo, refresh=False) + assert err == "" and held == repo + assert ss.head_commit(repo) == first + + snapshot_tree(package, repo, refresh=True) + assert ss.head_commit(repo) != first + wt, _ = _setup_worktree(repo, tmp_path / "ws" / "worktree", "specialist-t2") + assert wt is not None and (wt / "envs.py").read_text(encoding="utf-8") == "VLLM_USE_X = 2\n" + + +def test_snapshot_is_not_invented_while_the_package_is_mid_integrate(tmp_path: Path) -> None: + package = _installed_package(tmp_path) + repo, err = snapshot_tree(package, tmp_path / "session" / "tree_snapshots" / "vllm.git", refresh=False) + assert repo is None + assert "being integrated into" in err + + def test_collect_patches_does_not_rediscover_an_obsolete_harvest(tmp_path: Path) -> None: base, wt = _make_harvest_worktree(tmp_path) before = (wt / "runtime.py").read_text(encoding="utf-8") diff --git a/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess_helpers_coverage_unit.py b/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess_helpers_coverage_unit.py index 4615e9309a..ad664a4316 100644 --- a/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess_helpers_coverage_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess_helpers_coverage_unit.py @@ -16,7 +16,6 @@ from hyperloom.orchestrator.specialists.subprocess_ import ( SpecialistSubprocessConfig, SpecialistSubprocessDispatcher, - _pick_worktree_base, _setup_worktree, ) @@ -28,56 +27,6 @@ def __init__(self, returncode: int = 0, stdout: str = "", stderr: str = "") -> N self.stderr = stderr -# -- _pick_worktree_base --------------------------------------------------- -def test_pick_worktree_base_none(tmp_path: Path) -> None: - # directory without .git yields None - (tmp_path / "plain").mkdir() - assert _pick_worktree_base((str(tmp_path / "plain"), str(tmp_path / "absent"))) is None - - -def test_pick_worktree_base_finds_git(tmp_path: Path) -> None: - repo = tmp_path / "repo" - repo.mkdir() - (repo / ".git").mkdir() - assert _pick_worktree_base(("/nonexistent", str(repo))) == repo - - -def test_pick_worktree_base_prefers_the_framework_under_optimisation(tmp_path: Path) -> None: - """The session's own framework wins over whatever trusted root sorts first.""" - other = tmp_path / "aiter" - other.mkdir() - (other / ".git").mkdir() - framework = tmp_path / "HY-WorldPlay" - framework.mkdir() - (framework / ".git").mkdir() - - picked = _pick_worktree_base((str(other), str(framework)), preferred=str(framework)) - - assert picked == framework - - -def test_pick_worktree_base_ignores_a_preferred_root_that_is_not_a_checkout( - tmp_path: Path, -) -> None: - """A framework that is pip-installed rather than checked out must not disable isolation; the allowlist order still supplies a usable base.""" - other = tmp_path / "aiter" - other.mkdir() - (other / ".git").mkdir() - - picked = _pick_worktree_base((str(other),), preferred=str(tmp_path / "absent")) - - assert picked == other - - -def test_pick_worktree_base_without_a_preference_is_unchanged(tmp_path: Path) -> None: - """Domains with no framework checkout of their own keep the old behaviour.""" - repo = tmp_path / "repo" - repo.mkdir() - (repo / ".git").mkdir() - - assert _pick_worktree_base(("/nonexistent", str(repo)), preferred="") == repo - - # -- _setup_worktree ------------------------------------------------------- def test_setup_worktree_reuses_existing(tmp_path: Path) -> None: wt = tmp_path / "wt" diff --git a/src/hyperloom/inference_optimizer/tests/test_tracelens_stream_idle_timeout.py b/src/hyperloom/inference_optimizer/tests/test_tracelens_stream_idle_timeout.py index 2d39ba8b30..4297e1221a 100644 --- a/src/hyperloom/inference_optimizer/tests/test_tracelens_stream_idle_timeout.py +++ b/src/hyperloom/inference_optimizer/tests/test_tracelens_stream_idle_timeout.py @@ -28,7 +28,20 @@ def __init__(self, *blocks: object) -> None: class TaskStartedMessage: - pass + def __init__(self, task_id: str = "") -> None: + self.task_id = task_id + + +class TaskNotificationMessage: + def __init__(self, task_id: str, status: str = "completed") -> None: + self.task_id = task_id + self.status = status + + +class TaskUpdatedMessage: + def __init__(self, task_id: str, status: str) -> None: + self.task_id = task_id + self.patch = {"status": status} class ResultMessage: @@ -88,6 +101,59 @@ def test_silence_before_any_tool_stays_tightly_bounded(self): assert not _drive([AssistantMessage(TextBlock("reading the skill file"))]) +def _busy_after(messages: list[object]) -> bool: + in_flight = tr._InFlight() + for msg in messages: + in_flight.observe(msg) + return in_flight.busy + + +class TestBackgroundTasks: + def test_async_agents_still_running_keep_the_run_busy(self): + """Launch returns at once while the analyzer keeps running; the killed roofline sat exactly here.""" + assert _busy_after( + [ + AssistantMessage(ToolUseBlock("Agent")), + TaskStartedMessage("gemm"), + AssistantMessage(ToolResultBlock()), + AssistantMessage(TextBlock("10 of 11 subagents complete. Only the gemm analyzer is still running.")), + ] + ) + + def test_notification_releases_the_task(self): + assert not _busy_after( + [ + AssistantMessage(ToolUseBlock("Agent")), + TaskStartedMessage("gemm"), + AssistantMessage(ToolResultBlock()), + TaskNotificationMessage("gemm", "completed"), + ] + ) + + def test_terminal_update_releases_the_task(self): + assert not _busy_after( + [TaskStartedMessage("gemm"), AssistantMessage(ToolResultBlock()), TaskUpdatedMessage("gemm", "completed")] + ) + + def test_non_terminal_update_keeps_the_task(self): + assert _busy_after( + [TaskStartedMessage("gemm"), AssistantMessage(ToolResultBlock()), TaskUpdatedMessage("gemm", "running")] + ) + + def test_one_of_two_tasks_finishing_stays_busy(self): + assert _busy_after( + [ + TaskStartedMessage("a"), + TaskStartedMessage("b"), + AssistantMessage(ToolResultBlock()), + TaskNotificationMessage("a"), + ] + ) + + def test_silence_with_nothing_in_flight_stays_tightly_bounded(self): + assert not _busy_after([AssistantMessage(TextBlock("reading the skill file"))]) + + class TestToolIdleTimeoutResolution: def test_defaults_above_the_between_message_bound(self, monkeypatch): monkeypatch.delenv("HYPERLOOM_TRACELENS_TOOL_IDLE_TIMEOUT_SEC", raising=False) diff --git a/src/hyperloom/orchestrator/actions/executors/_proposal_identity.py b/src/hyperloom/orchestrator/actions/executors/_proposal_identity.py index 6b2181ca75..8762ac4633 100644 --- a/src/hyperloom/orchestrator/actions/executors/_proposal_identity.py +++ b/src/hyperloom/orchestrator/actions/executors/_proposal_identity.py @@ -5,6 +5,7 @@ from __future__ import annotations +import shlex from typing import Any, Mapping from hyperloom.common.coerce import to_str_list @@ -22,9 +23,23 @@ def coerce_args(value: Any) -> str: - """Coerce a payload ``extra_args`` / ``extra_server_args`` value to a shell-arg string.""" + """Coerce a payload ``extra_args`` / ``extra_server_args`` value to a shell-arg string. + + A mapping is read as flag -> value: ``True`` (or an empty value) is a bare + flag and ``False`` drops it. + """ if isinstance(value, (list, tuple)): return " ".join(str(v).strip() for v in value if str(v).strip()) + if isinstance(value, Mapping): + parts: list[str] = [] + for flag, flag_value in value.items(): + name = str(flag).strip() + if not name or flag_value is False: + continue + parts.append(name) + if flag_value is not True and str(flag_value if flag_value is not None else "").strip(): + parts.append(shlex.quote(str(flag_value).strip())) + return " ".join(parts) return str(value or "").strip() diff --git a/src/hyperloom/orchestrator/actions/executors/_server_patcher.py b/src/hyperloom/orchestrator/actions/executors/_server_patcher.py index 85fcbee5f4..6faa8c57aa 100644 --- a/src/hyperloom/orchestrator/actions/executors/_server_patcher.py +++ b/src/hyperloom/orchestrator/actions/executors/_server_patcher.py @@ -456,26 +456,34 @@ def _probe_isolated_vllm() -> tuple[str, Path] | None: if match.is_dir(): site = match.parent break - if site is None: - return None version = "" - vllm_python = os.environ.get("VLLM_PYTHON", "").strip() - if vllm_python and Path(vllm_python).exists(): + imported_root: Path | None = None + vllm_python = os.environ.get("VLLM_PYTHON", "").strip() or str(Path(venv_root) / "bin" / "python") + if Path(vllm_python).exists(): try: proc = subprocess.run( - [vllm_python, "-c", "import vllm; print(vllm.__version__)"], + [vllm_python, "-c", "import vllm; print(vllm.__version__); print(vllm.__file__)"], check=False, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=30, ) - if proc.returncode == 0: - version = proc.stdout.strip() + lines = proc.stdout.strip().splitlines() + if proc.returncode == 0 and len(lines) >= 2: + version = lines[-2].strip() + imported_root = Path(lines[-1].strip()).resolve().parent.parent except (OSError, subprocess.SubprocessError) as e: log.info("_server_patcher: VLLM_PYTHON version probe failed (%s)", e) + # A source build installs vLLM editable, so site-packages holds only an egg-link or .pth and the package lives + # in the checkout; only the venv's own interpreter can say where that is. + if site is None: + site = imported_root + if site is None: + return None + if not version: for dist in sorted(site.glob("vllm-*.dist-info")): name = dist.name diff --git a/src/hyperloom/orchestrator/actions/executors/_subprocess_kill.py b/src/hyperloom/orchestrator/actions/executors/_subprocess_kill.py index bea0c7c8d8..99a57f6b12 100644 --- a/src/hyperloom/orchestrator/actions/executors/_subprocess_kill.py +++ b/src/hyperloom/orchestrator/actions/executors/_subprocess_kill.py @@ -665,6 +665,12 @@ class _IncrementScan(NamedTuple): saw_warmup_begin: bool = False saw_measured_begin: bool = False residual: str = "" + active: bool = True + + +#: Access lines for the endpoints monitors poll on their own tick. An HTTP front end whose engine died keeps answering +#: them, so a log that grew only by these says nothing about whether the server is still serving. +_HEALTH_PROBE_LINE = re.compile(r'"GET /(?:metrics|health|healthz|ping)(?:[/?][^" ]*)? HTTP/[\d.]+"') # Cap on the partial line carried between scans. A log that appends a very long line without a newline -- or none at @@ -721,7 +727,7 @@ def _scan_logs_increment( saw_warmup_begin = saw_warmup_begin or scan.saw_warmup_begin saw_measured_begin = saw_measured_begin or scan.saw_measured_begin if scan.offset > prev: - grew = True + grew = grew or scan.active child_spoke = child_spoke or Path(path).name == _EVAL_LOG_NAME return _LogScan(saw_ready, saw_progress, saw_eval_start, grew, child_spoke, saw_warmup_begin, saw_measured_begin) @@ -759,6 +765,8 @@ def _scan_server_log_increment(path: str, from_offset: int, residual: str = "") saw_eval_start = any(marker in chunk for marker in _EVAL_START_MARKERS) saw_warmup_begin = any(marker in chunk for marker in _AGENTX_WARMUP_BEGIN_MARKERS) saw_measured_begin = any(marker in chunk for marker in _AGENTX_MEASURED_BEGIN_MARKERS) + complete = [line for line in chunk[: tail_at + 1].splitlines() if line.strip()] if tail_at >= 0 else [] + active = not complete or any(not _HEALTH_PROBE_LINE.search(line) for line in complete) return _IncrementScan( size, saw_ready, @@ -767,6 +775,7 @@ def _scan_server_log_increment(path: str, from_offset: int, residual: str = "") saw_warmup_begin, saw_measured_begin, residual=next_residual, + active=active, ) diff --git a/src/hyperloom/orchestrator/actions/executors/integrate_patch.py b/src/hyperloom/orchestrator/actions/executors/integrate_patch.py index 85be3be658..2e61745bb4 100644 --- a/src/hyperloom/orchestrator/actions/executors/integrate_patch.py +++ b/src/hyperloom/orchestrator/actions/executors/integrate_patch.py @@ -36,6 +36,8 @@ from hyperloom.inference_optimizer.gpu_types import amd_gpu_dispatch_identity from hyperloom.inference_optimizer.session.session_paths import runs_dir from hyperloom.inference_optimizer.framework_paths import ( + framework_apply_tree, + resolve_framework_tree, resolve_kernel_search_roots, resolve_session_framework_root, ) @@ -982,7 +984,9 @@ def _resolve_framework_root( Without a recorded root, the decision falls through to :func:`~...specialists.patch_safety.resolve_patch_apply_root`. Without any - patches to place, the session's declared root wins. + patches to place, the declared root wins, then the session's root, then + the root the session's framework tree is edited at: its checkout, or the + install root of a pip-installed package. Args: explicit: Declared framework root. @@ -994,8 +998,6 @@ def _resolve_framework_root( Returns: The resolved root, or ``None`` when the patches name no single tree. """ - roots = [Path(root) for root in resolve_kernel_search_roots()] - if recorded_root: return resolved_explicit_root(recorded_root) @@ -1016,7 +1018,8 @@ def _resolve_framework_root( # tree under optimisation into a non-candidate, and default_root cannot # stand in -- that is consulted only for a create-only set, which has no # pre-image to match. - candidates = [Path(session_root), *roots] if session_root else list(roots) + roots = [Path(root) for root in resolve_kernel_search_roots()] + candidates = [Path(session_root), *roots] if session_root else roots resolution = resolve_patch_apply_root( texts, explicit_root=explicit_path, @@ -1036,13 +1039,8 @@ def _resolve_framework_root( session_root = resolve_session_framework_root() if session_root and Path(session_root).is_dir(): return Path(session_root) - for root in roots: - if root.is_dir() and (root / ".git").exists(): - return root - for root in roots: - if root.is_dir(): - return root - return None + tree = framework_apply_tree(resolve_framework_tree(os.environ.get("FRAMEWORK", ""))) + return tree.root if tree is not None else None def _run_git_apply( diff --git a/src/hyperloom/orchestrator/actions/executors/tests/test_integrate_patch_helpers_unit.py b/src/hyperloom/orchestrator/actions/executors/tests/test_integrate_patch_helpers_unit.py index 2103207b17..ecae7f3156 100644 --- a/src/hyperloom/orchestrator/actions/executors/tests/test_integrate_patch_helpers_unit.py +++ b/src/hyperloom/orchestrator/actions/executors/tests/test_integrate_patch_helpers_unit.py @@ -171,14 +171,98 @@ def test_resolve_framework_root_explicit_missing_rejected(): assert ip._resolve_framework_root("/no/such/dir") is None -def test_resolve_framework_root_non_git_fallback(tmp_path, monkeypatch): +def _commit_tree(checkout: Path, *files: str) -> None: + """Make ``checkout`` a git repository tracking ``files``.""" + import subprocess + + subprocess.run(["git", "init", "-q", str(checkout)], check=True) + for rel in files: + target = checkout / rel + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text("x = 1\n", encoding="utf-8") + subprocess.run(["git", "-C", str(checkout), "add", "-A"], check=True) + subprocess.run( + ["git", "-C", str(checkout), "-c", "user.email=t@t", "-c", "user.name=t", "commit", "-qm", "base"], + check=True, + ) + + +def test_resolve_framework_root_non_git_framework_tree_is_its_own_root(tmp_path, monkeypatch): plain = tmp_path / "plain" plain.mkdir() monkeypatch.setattr(ip, "resolve_kernel_search_roots", lambda: [str(plain)]) monkeypatch.setattr(ip, "resolve_session_framework_root", lambda: "") + monkeypatch.setattr(ip, "resolve_framework_tree", lambda framework: str(plain) if framework == "vllm" else "") + monkeypatch.setenv("FRAMEWORK", "vllm") assert ip._resolve_framework_root(None) == plain +def test_resolve_framework_root_without_a_named_tree_takes_no_discovered_root(tmp_path, monkeypatch): + """A discovery order is not a name: with no tree named, no search root stands in for one.""" + checkout = tmp_path / "InferenceX" + _commit_tree(checkout, "benchmarks/benchmark_lib.sh") + monkeypatch.setattr(ip, "resolve_kernel_search_roots", lambda: [str(checkout)]) + monkeypatch.setattr(ip, "resolve_session_framework_root", lambda: "") + monkeypatch.setattr(ip, "resolve_framework_tree", lambda framework: "") + monkeypatch.delenv("FRAMEWORK", raising=False) + assert ip._resolve_framework_root(None, patch_paths=[]) is None + + +def test_resolve_framework_root_artifact_only_prefers_the_framework_checkout(tmp_path, monkeypatch): + """An artifact-only integrate must not land on the first git root discovered (the InferenceX checkout).""" + inferencex = tmp_path / "InferenceX" + _commit_tree(inferencex, "benchmarks/benchmark_lib.sh") + checkout = tmp_path / "sglang" + _commit_tree(checkout, "python/sglang/__init__.py") + package = checkout / "python" / "sglang" + monkeypatch.setattr(ip, "resolve_kernel_search_roots", lambda: [str(inferencex), str(package)]) + monkeypatch.setattr(ip, "resolve_session_framework_root", lambda: "") + monkeypatch.setattr(ip, "resolve_framework_tree", lambda framework: str(package) if framework == "sglang" else "") + monkeypatch.setenv("FRAMEWORK", "sglang") + + assert ip._resolve_framework_root(None, patch_paths=[]) == checkout + + +@pytest.mark.parametrize("image_has_source_checkout", [True, False]) +def test_resolve_framework_root_without_patches_takes_the_installed_package_itself( + tmp_path, monkeypatch, image_has_source_checkout +): + """A pip-installed framework is edited where the server imports it, never in a checkout that happens to be first.""" + inferencex = tmp_path / "InferenceX" + _commit_tree(inferencex, "benchmarks/benchmark_lib.sh") + site_packages = tmp_path / "site-packages" / "vllm" + site_packages.mkdir(parents=True) + (site_packages / "__init__.py").write_text("", encoding="utf-8") + roots = [inferencex] + if image_has_source_checkout: + source_checkout = tmp_path / "app" / "vllm" + _commit_tree(source_checkout, "vllm/__init__.py") + roots.append(source_checkout) + roots.append(site_packages) + monkeypatch.setattr(ip, "resolve_kernel_search_roots", lambda: [str(r) for r in roots]) + monkeypatch.setattr(ip, "resolve_session_framework_root", lambda: "") + monkeypatch.setattr(ip, "resolve_framework_tree", lambda framework: str(site_packages)) + monkeypatch.setenv("FRAMEWORK", "vllm") + monkeypatch.setenv("INFERENCEX_PATH", str(inferencex)) + + assert ip._resolve_framework_root(None, patch_paths=[]) == site_packages + + +def test_resolve_framework_root_ignores_a_repository_that_does_not_track_the_package(tmp_path, monkeypatch): + """A venv inside an unrelated repository sits under its ``.git`` without being part of it.""" + project = tmp_path / "project" + _commit_tree(project, "README.md") + package = project / ".venv" / "lib" / "site-packages" / "vllm" + package.mkdir(parents=True) + (package / "__init__.py").write_text("", encoding="utf-8") + monkeypatch.setattr(ip, "resolve_kernel_search_roots", lambda: [str(package)]) + monkeypatch.setattr(ip, "resolve_session_framework_root", lambda: "") + monkeypatch.setattr(ip, "resolve_framework_tree", lambda framework: str(package)) + monkeypatch.setenv("FRAMEWORK", "vllm") + + assert ip._resolve_framework_root(None, patch_paths=[]) == package + + def test_resolve_framework_root_none(monkeypatch): monkeypatch.setattr(ip, "resolve_kernel_search_roots", lambda: []) monkeypatch.setattr(ip, "resolve_session_framework_root", lambda: "") diff --git a/src/hyperloom/orchestrator/prompts/specialist_prompt_builder.py b/src/hyperloom/orchestrator/prompts/specialist_prompt_builder.py index 8f252e5a85..eb116f43d8 100644 --- a/src/hyperloom/orchestrator/prompts/specialist_prompt_builder.py +++ b/src/hyperloom/orchestrator/prompts/specialist_prompt_builder.py @@ -905,8 +905,9 @@ class SpecialistPromptInputs: # Extra knowledge-domain tags; each contributes a focus block to Section 1. extra_focus_tags: tuple[str, ...] = () - # Local source navigation hint. ``worktree_base`` is empty when the - # framework is pip-installed rather than a checkout. + # Local source navigation hint. ``worktree_base`` is the tree the worktree + # stands for -- the checkout it was cut from, or the installed tree it holds + # a snapshot of -- and is empty when the specialist has no worktree. session_framework_tree: str = "" framework_source_roots: tuple[str, ...] = () worktree_base: str = "" @@ -1966,15 +1967,17 @@ def _source_root_row(root: str, *, worktree_base: str) -> str: Args: root (str): The source root to render. - worktree_base (str): The checkout the specialist's worktree was cut - from, when there is one. + worktree_base (str): The tree the specialist's worktree stands for, + when there is one. Returns: str: A markdown list row for the root. """ base = (worktree_base or "").rstrip("/") - if base and root.rstrip("/") == base: - return f"- {root} — git checkout; your worktree was cut from it" + if base and Path(root.rstrip("/")).is_relative_to(base): + if (Path(base) / ".git").exists(): + return f"- {root} — git checkout; your worktree was cut from it" + return f"- {root} — installed package; your worktree holds a git snapshot of it, so edit its files there" if (Path(root) / ".git").is_dir(): return f"- {root} — git checkout" return f"- {root} — installed package, no git tree: re-author upstream diffs against it, never apply them" diff --git a/src/hyperloom/orchestrator/specialists/patch_safety.py b/src/hyperloom/orchestrator/specialists/patch_safety.py index 8b8a371411..d5257d2076 100644 --- a/src/hyperloom/orchestrator/specialists/patch_safety.py +++ b/src/hyperloom/orchestrator/specialists/patch_safety.py @@ -837,6 +837,7 @@ def strip_forbidden_proposal_fields(payload: dict[str, Any]) -> list[str]: _SPECIALIST_WORK_ARTIFACT_ROOTS = ( "patches", + "heartbeat.json", "specialist_done.json", "specialist_done.partial.json", "scratch/rebench", diff --git a/src/hyperloom/orchestrator/specialists/runner.py b/src/hyperloom/orchestrator/specialists/runner.py index f9ac3e67b2..9cf39d0966 100644 --- a/src/hyperloom/orchestrator/specialists/runner.py +++ b/src/hyperloom/orchestrator/specialists/runner.py @@ -29,7 +29,7 @@ from hyperloom.common.env_safety import BENCHMARK_SECRET_ENV_NAMES, redact_secret_values from hyperloom.common.timeutil import now_iso -from hyperloom.inference_optimizer.session.session_paths import runs_dir, specialist_intel_path +from hyperloom.inference_optimizer.session.session_paths import fs_safe_id, runs_dir, specialist_intel_path from ..roles.base import BackendError, LLMCallFailed from hyperloom.inference_optimizer.protocol.intent import Intent, IntentType from hyperloom.inference_optimizer.trace.conversation_trace import ConversationRecord, append_conversation @@ -55,12 +55,16 @@ SpecialistSubprocessConfig, SpecialistSubprocessDispatcher, SpecialistSubprocessResult, - _pick_worktree_base, _setup_worktree, ) from . import patch_safety as _patch_safety from .profile import MODE_PATCH, SpecialistProfile, resolve_specialist_profile -from hyperloom.inference_optimizer.framework_paths import resolve_framework_tree +from .tree_snapshot import snapshot_tree +from hyperloom.inference_optimizer.framework_paths import ( + FrameworkTree, + framework_apply_tree, + resolve_framework_tree, +) from ..loop.sub_agent_runner import RunnerContext from ..prompts.specialist_prompt_builder import ( SpecialistPromptInputs, @@ -73,29 +77,23 @@ NO_GIT_FRAMEWORK_SOURCE_ROOT = "no_git_framework_source_root" -def specialist_patch_preflight_error( - params: dict[str, Any] | None, - *, - framework_repo_path: str = "", - framework_source_roots: tuple[str, ...] = (), -) -> str: +def specialist_patch_preflight_error(params: dict[str, Any] | None, *, framework_repo_path: str = "") -> str: """Return the deterministic source-root error for a patch specialist.""" task_params = params or {} domain = get_domain(str(task_params.get("domain") or "")) if resolve_specialist_profile(task_params, domain=domain).mode != MODE_PATCH: return "" - roots = tuple( - path - for path in ( - str(framework_repo_path or "").strip(), - *(framework_source_roots or tuple(task_params.get("framework_source_roots") or ())), - ) - if path - ) - preferred = str(task_params.get("session_framework_tree") or "").strip() or resolve_framework_tree( - str(task_params.get("framework") or "") + source = _worktree_source(task_params, framework_repo_path=framework_repo_path) + return "" if source is not None else NO_GIT_FRAMEWORK_SOURCE_ROOT + + +def _worktree_source(params: dict[str, Any], *, framework_repo_path: str = "") -> FrameworkTree | None: + """The tree the session optimises, which the preflight and the worktree must both read the same way.""" + return framework_apply_tree( + str(params.get("session_framework_tree") or "").strip() + or resolve_framework_tree(str(params.get("framework") or "")) + or str(framework_repo_path or "").strip() ) - return "" if _pick_worktree_base(roots, preferred=preferred) is not None else NO_GIT_FRAMEWORK_SOURCE_ROOT def _subprocess_call_outcome(result: SpecialistSubprocessResult) -> tuple[str, dict[str, Any]]: @@ -628,7 +626,7 @@ async def _prepare( notes.append(f"domain={domain.key!r} is outside the domain catalogue; using generic prompt template") # Worktree — created only under subprocess dispatch; surfaced via ``workspace_path``. - worktree, worktree_base, worktree_err = await asyncio.to_thread( + worktree, worktree_source, worktree_err = await asyncio.to_thread( self._maybe_setup_worktree, ctx, workspace=workspace, @@ -636,6 +634,7 @@ async def _prepare( ) if worktree_err: notes.append(f"worktree_setup_failed:{worktree_err}") + worktree_base = worktree_source.root if worktree_source is not None else None workspace_for_prompt = worktree or workspace allocated_gpu_ids = tuple(int(g) for g in (ctx.extra.get("gpu_ids") or [])) @@ -661,7 +660,7 @@ async def _prepare( framework=str(params.get("framework") or ""), session_framework_tree=str(params.get("session_framework_tree") or ""), framework_source_roots=tuple(params.get("framework_source_roots") or ()), - worktree_base=str(worktree_base or ""), + worktree_base=str(worktree_base) if worktree is not None and worktree_base is not None else "", source_hint_directories=tuple(params.get("source_hint_directories") or ()), model_info=dict(params.get("model_info") or {}), static_recon_checklist=str(params.get("static_recon_checklist") or ""), @@ -1545,11 +1544,15 @@ def _maybe_setup_worktree( *, workspace: Path | None, profile: SpecialistProfile | None = None, - ) -> tuple[Path | None, Path | None, str]: + ) -> tuple[Path | None, FrameworkTree | None, str]: """Provision a per-task git worktree when in subprocess mode. Blocking (``git worktree add``, 60s); call via ``asyncio.to_thread``. + A tree no checkout tracks, such as a pip-installed package, is branched + off a session-owned snapshot of it instead, so the specialist still edits + the code the server runs. + Best-effort: the specialist still dispatches without isolation and the reason lands in ``notes``. @@ -1559,30 +1562,51 @@ def _maybe_setup_worktree( profile: Resolved dispatch profile; non-patch mode skips worktree. Returns: - A ``(worktree_dir, worktree_base, error)`` tuple; ``worktree_dir`` - is ``None`` in in-process mode or on git failure. + A ``(worktree_dir, source, error)`` tuple; ``worktree_dir`` is + ``None`` in in-process mode or on git failure, and ``source`` is + the tree the worktree stands for. """ if self.subprocess_config is None or workspace is None: return None, None, "" if profile is not None: if profile.mode != MODE_PATCH: return None, None, "" - preflight_error = specialist_patch_preflight_error( - ctx.task.params, - framework_source_roots=self.subprocess_config.framework_source_roots, - ) + preflight_error = specialist_patch_preflight_error(ctx.task.params) if preflight_error: return None, None, preflight_error - base = _pick_worktree_base( - self.subprocess_config.framework_source_roots, - preferred=resolve_framework_tree(str((ctx.task.params or {}).get("framework") or "")), - ) + source = _worktree_source(ctx.task.params or {}) + if source is None: + return None, None, "" + base = source.root + if not source.checkout: + if self.session_dir is None: + return None, source, f"{source.tree} has no checkout and no session dir to snapshot it into" + repo = self.session_dir / "tree_snapshots" / f"{fs_safe_id(str(source.tree))}.git" + snapshot, err = snapshot_tree(source.tree, repo, refresh=not self._integrate_in_flight()) + if snapshot is None: + return None, source, err + base = snapshot worktree_path = workspace / "worktree" branch = f"specialist-{ctx.task.task_id}" wt, err = _setup_worktree(base, worktree_path, branch) if wt is None: - return None, base, err - return wt, base, "" + return None, source, err + return wt, source, "" + + def _integrate_in_flight(self) -> bool: + """Whether an integrate has mutated the framework tree without deciding on it yet.""" + from ..state.shared_state import SharedState + + if self.session_dir is None: + return False + try: + return bool(SharedState.load_or_init(self.session_dir).pending_integrate) + except Exception: + # An unreadable state cannot vouch for the tree. + log.warning( + "specialist worktree: cannot read pending_integrate; not refreshing the snapshot", exc_info=True + ) + return True # Workspace file protocol def _resolve_workspace(self, ctx: RunnerContext) -> Path | None: diff --git a/src/hyperloom/orchestrator/specialists/subprocess_.py b/src/hyperloom/orchestrator/specialists/subprocess_.py index 8b4d490e9f..f187198e4a 100644 --- a/src/hyperloom/orchestrator/specialists/subprocess_.py +++ b/src/hyperloom/orchestrator/specialists/subprocess_.py @@ -647,49 +647,6 @@ def _declared_targets(done_payload: Mapping[str, Any] | None) -> tuple[str, ...] # Worktree management -def _pick_worktree_base( - roots: tuple[str, ...], - *, - preferred: str = "", -) -> Path | None: - """Return the checkout to branch the specialist's worktree off. - - ``preferred`` wins whenever it is a checkout. It names the framework the - session is actually optimising, which ``roots`` cannot express: their order - records only how they were discovered. Selecting by position worked while - exactly one root happened to be a git checkout; when a pod started shipping - aiter as one it sorted first, so WorldPlay specialists were handed an aiter - worktree and the ``hyvideo/`` patches they wrote grounded against nothing. - - Falls back to None when nothing qualifies — the runner then runs the - specialist without an isolated worktree. - - Args: - roots: Candidate root paths to probe for a ``.git`` marker. - preferred: Checkout of the framework under optimisation, if any. Skipped - when it is absent or not a checkout, so a pip-installed framework - costs the specialist nothing. - - Returns: - The chosen checkout root, or ``None`` when none qualify. - """ - - def _is_checkout(path: str) -> Path | None: - p = Path(path) - # ``.git`` may be a file (worktree) or a dir (repo). - return p if p.is_dir() and (p / ".git").exists() else None - - if preferred: - chosen = _is_checkout(preferred) - if chosen is not None: - return chosen - for r in roots: - chosen = _is_checkout(r) - if chosen is not None: - return chosen - return None - - def _setup_worktree( base: Path, worktree_path: Path, @@ -1734,8 +1691,9 @@ def _collect_patches( Args: worktree: Per-task worktree, or None. workspace: Task workspace. - worktree_base: Checkout the worktree was branched off, which is the - apply root of anything harvested from it. + worktree_base: The tree the worktree stands for -- the checkout it + was branched off, or the directory it holds a snapshot of -- + which is the apply root of anything harvested from it. worktree_base_commit: The commit recorded when the worktree was created, so the harvest stays anchored to the pre-round state even if the specialist committed. @@ -1828,7 +1786,6 @@ def _read_done(done_file: Path) -> dict[str, Any] | None: "SpecialistSubprocessConfig", "SpecialistSubprocessDispatcher", "SpecialistSubprocessResult", - "_pick_worktree_base", "_setup_worktree", "resolve_codex_executable", ] diff --git a/src/hyperloom/orchestrator/specialists/tree_snapshot.py b/src/hyperloom/orchestrator/specialists/tree_snapshot.py new file mode 100644 index 0000000000..ebf7199061 --- /dev/null +++ b/src/hyperloom/orchestrator/specialists/tree_snapshot.py @@ -0,0 +1,109 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""A session-owned git history for a framework tree that has none of its own.""" + +from __future__ import annotations + +import subprocess +import threading +from pathlib import Path + +#: Build output and caches: the specialist edits source, and a worktree carrying +#: a framework's shared objects would cost gigabytes on every dispatch. +_EXCLUDES: tuple[str, ...] = ("__pycache__/", "*.pyc", "*.pyo", "*.so", "*.so.*") + +_IDENTITY: tuple[str, ...] = ("-c", "user.name=hyperloom", "-c", "user.email=hyperloom@localhost") + +_GIT_TIMEOUT_SEC = 600.0 + +_locks: dict[str, threading.Lock] = {} +_locks_guard = threading.Lock() + + +def _lock(repo: Path) -> threading.Lock: + """The lock serialising snapshots into ``repo``; specialists dispatch concurrently.""" + with _locks_guard: + return _locks.setdefault(str(repo), threading.Lock()) + + +def _git(repo: Path, tree: Path, *args: str) -> subprocess.CompletedProcess[str]: + """Run git against ``repo`` with ``tree`` as its work tree.""" + return subprocess.run( + [ + "git", + "-c", + f"safe.directory={repo}", + "-c", + f"safe.directory={tree}", + "--git-dir", + str(repo), + "--work-tree", + str(tree), + *args, + ], + capture_output=True, + text=True, + timeout=_GIT_TIMEOUT_SEC, + check=False, + ) + + +def _failure(action: str, cp: subprocess.CompletedProcess[str]) -> str: + """Describe a failed git step.""" + return f"git {action} rc={cp.returncode}: {(cp.stderr or cp.stdout).strip()[:400]}" + + +def snapshot_tree(tree: Path, repo: Path, *, refresh: bool) -> tuple[Path | None, str]: + """Commit ``tree`` as it is now into the bare repository ``repo``. + + The repository lives outside ``tree`` and nothing is written into the tree, + so it keeps reading as the plain directory every other consumer takes it + for. A worktree branched off ``repo`` holds the tree's files at the same + relative paths, so a diff harvested from it applies to ``tree`` itself. + + Args: + tree: The directory to snapshot. + repo: The bare repository holding the history; created on first use. + refresh: Commit ``tree``'s current content. When false, an existing + snapshot is branched off as it stands: the tree is mid-integrate + and would hand the specialist a candidate no decision has kept. + + Returns: + ``(repo, "")`` once ``repo`` has a commit to branch off, else + ``(None, error)``. + """ + with _lock(repo): + try: + fresh = not (repo / "HEAD").is_file() + if fresh and not refresh: + return None, f"{tree} is being integrated into and has no snapshot to branch off yet" + if not refresh: + return repo, "" + if fresh: + repo.parent.mkdir(parents=True, exist_ok=True) + cp = subprocess.run( + ["git", "init", "--bare", "-q", str(repo)], + capture_output=True, + text=True, + timeout=_GIT_TIMEOUT_SEC, + check=False, + ) + if cp.returncode != 0: + return None, _failure("init", cp) + (repo / "info").mkdir(parents=True, exist_ok=True) + (repo / "info" / "exclude").write_text("".join(f"{p}\n" for p in _EXCLUDES), encoding="utf-8") + cp = _git(repo, tree, "add", "-A", "--", ".") + if cp.returncode != 0: + return None, _failure("add", cp) + staged = _git(repo, tree, "diff", "--cached", "--quiet") + if fresh or staged.returncode != 0: + cp = _git(repo, tree, *_IDENTITY, "commit", "-q", "--no-verify", "--allow-empty", "-m", "snapshot") + if cp.returncode != 0: + return None, _failure("commit", cp) + except (OSError, subprocess.SubprocessError) as exc: + return None, f"snapshot of {tree} failed: {exc!r}" + return repo, "" + + +__all__ = ["snapshot_tree"] diff --git a/src/hyperloom/skills/hyperloom-setup/SKILL.md b/src/hyperloom/skills/hyperloom-setup/SKILL.md index 1a202aaa8d..dba7ee4545 100644 --- a/src/hyperloom/skills/hyperloom-setup/SKILL.md +++ b/src/hyperloom/skills/hyperloom-setup/SKILL.md @@ -26,7 +26,7 @@ Whether to generate or run a Docker container is decided later by the example - `baremetal`: run directly in the current development environment, even if that environment is itself a container. It provides ROCm and ROCm torch; setup can - reuse preinstalled SGLang, vLLM, or ATOM, or optionally install SGLang/vLLM. + reuse preinstalled SGLang, vLLM, or ATOM, or optionally install SGLang/vLLM/ATOM. Do not start another Docker container or require a physical host OS. - `docker`: writes `.env` and records the run mode; the example (workload) skill starts the container and runs setup inside it. @@ -163,7 +163,7 @@ value. 7. Only when the user chose `baremetal`, ask whether to install a serving framework (used as the `--install-framework` value in Step 4). Present exactly - these three option labels in this order and do not reorder them by + these four option labels in this order and do not reorder them by recommendation: 1. `none`: use an already-installed vLLM/SGLang/ATOM stack in the selected Python environment. This skips framework installation, not setup's @@ -173,13 +173,23 @@ value. torch/SGLang stack untouched. On ROCm 7.2.x the ROCm wheel brings its own torch; on ROCm 10 vLLM is built from source in a venv that reuses the host ROCm torch. - - Do not mark any option as recommended. Present the three options in the exact + 4. `atom`: install ATOM and its AITER dependency from source (shared with the + host torch). The ATOM commit defaults to the one `rocm/atom-dev:v0.1.7-rc0` + was built from. + - ATOM never shares a Python with SGLang or vLLM: ATOM registers vLLM and + SGLang plugins that both engines load by default, so a later SGLang or + vLLM run there would execute ATOM's platform, model and loader code. Setup + therefore refuses `atom` when SGLang or vLLM imports from the selected + Python (including a ROCm 10 vLLM venv built over it), and refuses `sglang` + or a shared/ROCm 10 `vllm` when ATOM imports there. The ROCm 7.2.x + `vllm (isolated)` venv is self-contained and unaffected. When the user wants + ATOM beside another engine, put ATOM in a separate container. + - Do not mark any option as recommended. Present the four options in the exact order above without a default selection. - - ATOM must already be installed; there is no `--install-framework atom`. - For an ATOM workload, use `none` and explicitly verify/select it with - `--frameworks atom --require-frameworks`, especially when multiple serving - frameworks are installed. Do not change global framework defaults or choose - whichever other framework happens to import first. + - For an ATOM that is already installed, use `none` and explicitly + verify/select it with `--frameworks atom --require-frameworks`, especially + when multiple serving frameworks are installed. Do not change global + framework defaults or choose whichever other framework happens to import first. 8. Only when the user chose `baremetal` **and** `vllm (isolated)` in Step 7, briefly note that on ROCm 7.2.x the installer enforces the vLLM 0.28.0+ @@ -261,7 +271,7 @@ If any required secret is missing or still a placeholder, stop and ask the user ## Step 4: Run Setup Backend In `baremetal` mode, run the backend on the host. The `--install-framework` value -is the framework the user chose in Step 2 (`none` / `vllm` / `sglang`). In +is the framework the user chose in Step 2 (`none` / `vllm` / `sglang` / `atom`). In `docker` mode, skip the backend on the host (see below). ### `baremetal` @@ -335,10 +345,18 @@ export REPO_ROOT="$(pwd -P)" PYTHONPATH="$REPO_ROOT" python3 -m hyperloom.inference_optimizer.setup -- --install-framework sglang --yes ``` +For `atom` (installs AITER, then ATOM at `ATOM_REF` as an editable checkout under +the dependency root, and verifies that `atom` and its server module import): + +```bash +export REPO_ROOT="$(pwd -P)" +PYTHONPATH="$REPO_ROOT" python3 -m hyperloom.inference_optimizer.setup -- --install-framework atom --yes +``` + `none` reuses a preinstalled framework; it does not remove it. If no serving framework is importable, report the missing prerequisite. For ATOM, select an -existing compatible ATOM environment or prepare one separately with approval; -do not bypass the check or install a different framework as a substitute. +existing compatible ATOM environment or install it with `--install-framework atom` +after approval; do not bypass the check or install a different framework as a substitute. `BENCHMARK_BASE_URL` and `HYPERLOOM_SKIP_FRAMEWORK_CHECK` remain available for other existing remote/special workflows, not to bypass checks in this local ATOM example. @@ -374,9 +392,8 @@ the demo skill runs setup inside the container. Read - If `HYPERLOOM_RUN_MODE` is `baremetal` and `FRAMEWORK` is missing or empty, setup did not detect a serving framework (for example, `none` without an importable SGLang/vLLM/ATOM stack). Check the selected interpreter and report - the failure. Offer SGLang/vLLM installation only if that is the user's intended - framework; ATOM users need a preinstalled compatible ATOM stack. Do not invent - a `FRAMEWORK` value or silently switch frameworks. + the failure. Offer SGLang/vLLM/ATOM installation only for the framework the user + intends to run. Do not invent a `FRAMEWORK` value or silently switch frameworks. ## Step 6: Report Result @@ -398,8 +415,9 @@ mode, ask the user whether they want to run a demo optimization now, and if so which option: - `3h` — short, no-kernel run. Best for a first end-to-end check. -- `12h` — medium-length Qwen3-14B-FP8 run; use the ATOM variant when ATOM is - detected or explicitly selected (including a Docker run awaiting detection). +- `12h` — medium-length Qwen3-14B-FP8 run on SGLang, vLLM or ATOM; the demo + follows its ATOM section when ATOM is detected or explicitly selected + (including a Docker run awaiting detection). - `custom advanced` — user-selected model, framework, workload, budget, phase toggles, and advanced CLI flags. @@ -410,9 +428,9 @@ structured UI: which kernel optimization backend the KERNEL_AGENT phase should use. Do not ask this for `3h` (it runs `--no-kernel`, so there is no kernel phase to route) or for `custom advanced` (that skill collects its own flags). -When `FRAMEWORK=atom`, or the user selected the ATOM demo in Docker mode before -container-side detection, ask the same backend question and route to the ATOM -pair instead of the SGLang/vLLM one. If no backend was selected, write nothing +When `FRAMEWORK=atom`, or the user selected ATOM in Docker mode before +container-side detection, ask the same backend question; the same two demo +skills cover ATOM. If no backend was selected, write nothing to `.env`. Preserve an explicit shell or `.env` value; if it is non-empty and does not opt in, report the original value and ask whether to keep it or explicitly switch before launch. Never silently unset or delete it. @@ -428,12 +446,10 @@ kernel backend differs, so the two runs stay directly comparable. The choice selects which demo skill to load and sets `KERNEL_OPT_BACKEND_ORDER`: -- `geak` → load `hyperloom-qwen3-14b-fp8-12h`, or - `hyperloom-qwen3-14b-fp8-12h-atom` when `FRAMEWORK=atom`. Leave +- `geak` → load `hyperloom-qwen3-14b-fp8-12h`. Leave `KERNEL_OPT_BACKEND_ORDER` unset, or write `geak`; anything that does not opt in means GEAK. -- `forge` → load `hyperloom-qwen3-14b-fp8-12h-forge`, or - `hyperloom-qwen3-14b-fp8-12h-atom-forge` when `FRAMEWORK=atom`, and write +- `forge` → load `hyperloom-qwen3-14b-fp8-12h-forge`, and write `KERNEL_OPT_BACKEND_ORDER=forge` to `.env` so a `--resume-from` relaunch keeps the same backend. @@ -469,10 +485,8 @@ The demo skills are installed under each agent's discovery dir (`.agents/skills/ `.claude/skills/`, `.cursor/skills/`); load the matching one by name: - `3h` → `hyperloom-qwen3-8b-3h` -- `12h` + `geak` → `hyperloom-qwen3-14b-fp8-12h` -- `12h` + `forge` → `hyperloom-qwen3-14b-fp8-12h-forge` -- `12h` + `FRAMEWORK=atom` + `geak` → `hyperloom-qwen3-14b-fp8-12h-atom` -- `12h` + `FRAMEWORK=atom` + `forge` → `hyperloom-qwen3-14b-fp8-12h-atom-forge` +- `12h` + `geak` → `hyperloom-qwen3-14b-fp8-12h` (SGLang, vLLM or ATOM) +- `12h` + `forge` → `hyperloom-qwen3-14b-fp8-12h-forge` (SGLang, vLLM or ATOM) - `custom advanced` → `hyperloom-custom-advanced` The demo skill reads the values already in `.env` (LLM keys/base URLs,