diff --git a/README.md b/README.md index e21f3e145b..d6ed9995c2 100755 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ [![Tests](https://github.com/AMD-AGI/Hyperloom/actions/workflows/tests-coverage.yml/badge.svg)](https://github.com/AMD-AGI/Hyperloom/actions/workflows/tests-coverage.yml) [![Lint](https://github.com/AMD-AGI/Hyperloom/actions/workflows/lint.yml/badge.svg)](https://github.com/AMD-AGI/Hyperloom/actions/workflows/lint.yml) -[![Version](https://img.shields.io/badge/version-1.1.1-blue)](https://github.com/AMD-AGI/Hyperloom/blob/main/pyproject.toml) +[![Version](https://img.shields.io/badge/version-1.1.2-blue)](https://github.com/AMD-AGI/Hyperloom/blob/main/pyproject.toml) [![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://github.com/AMD-AGI/Hyperloom/blob/main/LICENSE) [![Python](https://img.shields.io/badge/python-3.10%2B-blue)](https://github.com/AMD-AGI/Hyperloom/blob/main/pyproject.toml) diff --git a/docs/compatibility.rst b/docs/compatibility.rst index 5eaee88761..fc1c317c0c 100644 --- a/docs/compatibility.rst +++ b/docs/compatibility.rst @@ -49,7 +49,7 @@ The following table lists the minimum requirements for running Hyperloom. +---------------------+--------------------------------------------------------+ | Operating System | Ubuntu 24.04 (recommended); Ubuntu 22.04 (legacy) | +---------------------+--------------------------------------------------------+ -| ROCm Version | 7.2.x | +| ROCm Version | 7.2.x, 10.0 | +---------------------+--------------------------------------------------------+ | Python | >= 3.10 | +---------------------+--------------------------------------------------------+ @@ -75,7 +75,7 @@ The following table lists the validated Hyperloom version and component combinat +-------------------+---------------------------+------------------------+------------------------------------+---------------+-------------+-----------------------------+ | Hyperloom version | Component | GPU | ROCm version | Ubuntu | Python | GitHub | +===================+===========================+========================+====================================+===============+=============+=============================+ -| 1.1.1 | `TraceLens 1.0.0`_ | Hardware-agnostic | No dependency | OS-independent| >= 3.6 | |tracelens-github| | +| 1.1.2 | `TraceLens 1.0.0`_ | Hardware-agnostic | No dependency | OS-independent| >= 3.6 | |tracelens-github| | + +---------------------------+------------------------+------------------------------------+---------------+-------------+-----------------------------+ | | `GEAK 4.0.0`_ | MI300X, MI325X, MI355X | 6.4.x, 7.0.x, 7.1.x, 7.2.x, 10.0.0 | 22.04, 24.04 | 3.8, 3.12 | |geak-github| | + +---------------------------+------------------------+------------------------------------+---------------+-------------+-----------------------------+ @@ -160,8 +160,9 @@ mirror, set the registry prefix accordingly. * - ``rocm/atom-dev:v0.1.7-rc0`` - MI355X (verified); MI300X / MI325X untested -The vLLM image entrypoint is ``vllm serve``, so override it (for example -``--entrypoint tail``) when starting a long-running Hyperloom container. +The images default to ``/bin/bash``, which exits at once in a detached +container, so override the entrypoint (for example ``--entrypoint tail``) when +starting a long-running Hyperloom container. The ``rocm/vllm`` image serves an installed wheel, so ``install.sh`` turns its ``/app/vllm`` checkout into the patchable source tree: it pins the tree to vLLM @@ -226,11 +227,11 @@ overridable via env (``SGLANG_REF``, ``SGLANG_ROCM_EXTRA``, ``VLLM_VERSION``, ``VLLM_ROCM_VARIANT``, ``VLLM_INSTALL_METHOD``, ``VLLM_REPO``, ``VLLM_SOURCE_REF``, ``VLLM_ROOT``) for hosts that need a different pinned stack. -The table above is the validated combination, and ROCm 7.2.x under a single -``/opt/rocm`` prefix is the layout to prefer. A host where ROCm arrives as -TheRock's pip wheels instead, split across the ``_rocm_sdk_*`` namespace -packages, is handled rather than validated: the bare-metal installer probes -those packages for library resolution and, before a framework source build, -supplies the devel headers and toolchain root from them, so setup does not fail -on that layout. Only ROCm 7.0.x and 7.2.x have a published ``amd-sglang`` wheel; -any other stack falls back to a source install. See :doc:`/install/install`. +The table above is the validated combination. ROCm 7.2.x is validated under a +single ``/opt/rocm`` prefix. ROCm 10.0 arrives as TheRock's pip wheels, split +across the ``_rocm_sdk_*`` namespace packages, which is the layout the ``rocm10`` +images are built from and the one the ROCm 10 routes are validated on: the +bare-metal installer probes those packages for library resolution and, before a +framework source build, supplies the devel headers and toolchain root from them. +Only ROCm 7.0.x and 7.2.x have a published ``amd-sglang`` wheel; any other stack +falls back to a source install. See :doc:`/install/install`. diff --git a/docs/conf.py b/docs/conf.py index dea12c98ae..18a1b725b2 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -19,7 +19,7 @@ """ # Keep in sync with pyproject.toml [project].version. -version_number = "1.1.1" +version_number = "1.1.2" html_theme = "rocm_docs_theme" html_theme_options = { diff --git a/docs/install/install.md b/docs/install/install.md index 80e6a0fe64..565d941038 100644 --- a/docs/install/install.md +++ b/docs/install/install.md @@ -42,7 +42,7 @@ Before installing Hyperloom, ensure the following requirements are met. From the agent terminal in that workspace, install the published release wheel: ```bash -pip install hyperloom-inference-optimizer==1.1.1 --target . +pip install hyperloom-inference-optimizer==1.1.2 --target . ``` It is normal for the current directory to contain many Python package directories @@ -106,19 +106,20 @@ Requirements: - Ubuntu 24.04 is the recommended host OS for bare-metal setup. vLLM 0.28.0+ requires Ubuntu 24.04 or newer; on Ubuntu 22.04, downgrade vLLM (for example ``VLLM_VERSION=0.27.1``) or use Docker mode instead. -- ROCm runtime and ROCm torch are already installed. ROCm 7.2.x is the - recommended and validated baseline; see - {doc}`Compatibility ` for the combination the release is +- ROCm runtime and ROCm torch are already installed. ROCm 7.2.x and ROCm 10.0 + are the validated stacks; on ROCm 10.0, bare-metal vLLM is built from source. + See {doc}`Compatibility ` for the combination the release is tested against. Other ROCm versions are not validated. - `git` is available for dependency checkouts. - A serving framework is either already installed, or setup might install one. - **Base Python on this GPU host** — the interpreter `install_baremetal.sh` resolves, not a child venv: - Python 3.10+ for SGLang and general operation. - - Exactly Python 3.12 when setup installs vLLM. vLLM ROCm wheels are - built for 3.12 only. vLLM defaults to an isolated framework venv, but that - venv is created from the base interpreter and inherits its version; isolated - mode does not relax the requirement. + - Exactly Python 3.12 when setup installs vLLM from the ROCm 7.2.x wheel, + which is built for 3.12 only; the ROCm 10.0 source build accepts Python + >= 3.10, < 3.15. vLLM defaults to an isolated framework venv, but that venv + is created from the base interpreter and inherits its version; isolated mode + does not relax the requirement. In this scenario, `/hyperloom-setup` runs the packaged setup backend on the host: @@ -146,15 +147,14 @@ and a ROCm-built torch. Two framework-install details are worth knowing: best-effort, trying `libopenmpi3t64` (Ubuntu 24.04's 64-bit `time_t` name) and `libopenmpi3`. It skips silently when the library already resolves, when `apt` is unavailable, or when not running as root. -- **ROCm as pip wheels**: the validated baseline is ROCm 7.2.x under a single - `/opt/rocm` prefix, and that is the layout to prefer. ROCm can instead arrive - as TheRock's wheels, split across the `_rocm_sdk_core`, - `_rocm_sdk_libraries` and `_rocm_sdk_devel` namespace packages. The installer - handles that layout so it does not fail setup on it, but such a host is not a - tested configuration: Phase 1 probes all three packages, including their - `rocm_sysdeps` and `host-math` subdirectories, so the gate does not report - libraries as missing that the loader does resolve at runtime; and before a - source build, which needs hipBLAS/hipSPARSE/thrust headers, it adds +- **ROCm as pip wheels**: ROCm 7.2.x is validated under a single `/opt/rocm` + prefix. ROCm 10.0 arrives as TheRock's wheels, split across the + `_rocm_sdk_core`, `_rocm_sdk_libraries` and `_rocm_sdk_devel` namespace + packages, which is the layout the `rocm10` images are built from and the one + the ROCm 10 routes are validated on. Phase 1 probes all three packages, + including their `rocm_sysdeps` and `host-math` subdirectories, so the gate + does not report libraries as missing that the loader does resolve at runtime; + and before a source build, which needs hipBLAS/hipSPARSE/thrust headers, it adds `rocm-sdk-devel` pinned to the installed `rocm-sdk-core` version, expands it with `rocm-sdk init`, and exports the root `rocm-sdk path --root` reports so the compiler and its include tree come from the same package. All of this is a diff --git a/docs/install/slurm.md b/docs/install/slurm.md index 9f6530c226..fad4dbc901 100644 --- a/docs/install/slurm.md +++ b/docs/install/slurm.md @@ -251,6 +251,7 @@ optimization. Model names must exist in your key's catalog: |---|---|---| | `HL_SHM_SIZE` | `64g` | docker `--shm-size`; raise it for high concurrency. | | `HL_CONTAINER_RUNTIME` | `auto` | Force `docker` or `pyxis`. | +| `HL_ORPHAN_MIN_AGE_S` | `900` | Minimum age in seconds before a container whose `CLAW_SESSION_ID` has no live client is reclaimed as orphaned; younger containers are left alone. | | `HL_GPU_TYPE_OVERRIDE` | — | Override `--gpu-type` (lowercase) when hardware differs from the table row. | | `HL_SHARED_MOUNT` | `/path` | Shared FS bind-mounted into the container. | | `HL_DATA_ROOT` | `/hyperloom-slurm` | Artifact root. | diff --git a/docs/reference/environment-variables.md b/docs/reference/environment-variables.md index d345150008..532df8eb8a 100644 --- a/docs/reference/environment-variables.md +++ b/docs/reference/environment-variables.md @@ -68,6 +68,7 @@ The following variables configure filesystem paths for Hyperloom's runtime depen | `HYPERLOOM`
`_CACHE_`
`DIR` | No | `$REPO_ROOT`
`/.cache` | Writable, repo-local base for auto-cloned open-source deps (TraceLens, Magpie, etc.), cloned per revision as `@`. Not under `$TMPDIR` so a reaper cannot wipe it mid-run. | | `KERNELFORGE`
`_PROJECT_`
`ROOT` | No | `$USER_DATA_PATH/kernelforge`, else `~/.cache/hyperloom/kernelforge` | Writable root for forge's own state and for resource-tree overrides. Holds the learned knowledge base (`knowledge_base//learned/`), the tuning DB, postmortems and `forge_experiments/`. A subtree placed here also **overrides the copy packaged inside `kernelforge`** — a `serving_patches/` or `examples/` directory under this root wins over the shipped one, which is the supported way to try a patch or a task without editing site-packages. Must be writable: it deliberately never resolves to the installed package directory or to the cwd. **This is the replacement for the removed `FORGE_PATH`**, which nothing reads any more — a stale `FORGE_PATH` is still forwarded (the `FORGE_` prefix is on the dotenv allowlist) and then ignored. | | `SKIP_FORGE`
`_PROFILING` | No | Unset (the extra is installed) | Set to `1` to make `install.sh` skip `pip install -e "$REPO_ROOT[forge-profiling]"`. That extra is rocprof-compute's own dependency set (~20 wheels, including the exact `kaleido==0.2.1` / `astunparse==1.6.2` pins ROCm 7.2.x requires); without it forge's profiler degrades to the lightweight PMC path instead of System Speed-of-Light + roofline. Installed by default on purpose — the previous gate made this a silent skip on every pod. | +| `ROCPC_VENV` | No | `/opt/rocprof-compute-venv` | Private venv that rocprof-compute's analyze mode runs in, created by `install.sh` from the tool's own `requirements.txt` so its exact pins cannot displace the serving image's numpy and pandas. `rocpc_profile.py` derives the same path. Without the venv, analyze degrades and profiling still collects. | | `MAGPIE_PATH` | No | Resolved from installed `Magpie` package unless explicitly set | Magpie package root for benchmark wrappers and patch inspection. | | `INFERENCE_`
`OPTIMIZER`
`_MODEL_PATH_ROOTS` | No | Built-in model roots such as `/models` and `/shared_nfs` | `os.pathsep`-separated allowlist for absolute model paths restored from `state.json` during a resume. HuggingFace-style repo IDs remain allowed. Set this when production models live outside the built-in roots. | | `INFERENCE_`
`OPTIMI`
`ZER_SES`
`SION_DIR` | No (multi-node) | Unset | Last-resort session root for multi-node crash-log collection. Point it at one session dir, never at `$USER_DATA_PATH`. | diff --git a/docs/release-notes.md b/docs/release-notes.md index 44d2773111..89f52286e6 100644 --- a/docs/release-notes.md +++ b/docs/release-notes.md @@ -1,13 +1,13 @@ --- myst: html_meta: - "description": "Hyperloom release notes: headline capabilities for version 1.1.1, a patch release that corrects the Recipe knowledge base warm-start path, unifies agent-backend selection, bounds supervisor watchdog restarts, and bumps the bare-metal vLLM default; plus the 1.1.0 feature release." + "description": "Hyperloom release notes: headline capabilities for version 1.1.2, a patch release that validates ROCm 10, makes ENABLEMENT its own phase, retires robustness supervision in favour of explicit resume and recovery, and supports ATOM end to end; plus the 1.1.1 and 1.1.0 releases." "keywords": "Hyperloom, release notes, LLM inference, AMD GPU, ROCm, agentic optimization, TraceLens, GEAK, KernelForge, Primus-Claw, bare metal, kernel optimization" --- # Hyperloom release notes -The current packaged version is 1.1.1 (`pyproject.toml`). For the +The current packaged version is 1.1.2 (`pyproject.toml`). For the per-change history since the initial snapshot, and for a detailed breakdown of all previous Hyperloom pre-release versions, see [Releases](https://github.com/AMD-AGI/Hyperloom/releases); this page @@ -20,6 +20,116 @@ Merged to `main` and not yet carried by a tagged release. Each entry moves into the [release](https://github.com/AMD-AGI/Hyperloom/releases) that ships it. +Nothing yet. + +## Hyperloom 1.1.2 release + +The [1.1.2 release](https://github.com/AMD-AGI/Hyperloom/releases/tag/v1.1.2) +is a patch release on top of 1.1.1. ROCm 10 becomes a validated stack, bring-up +gets a phase of its own, and the automatic supervision layer is retired in +favour of explicit resume and recovery. The session record resumes across the +upgrade: `ensure_schema` migrates a 1.1.1 `coordinator.db` on the way in. The +command line, the environment contract and several report fields do move; see +"Before upgrading from 1.1.1" below. + +### Before upgrading from 1.1.1 + +Two optimizer options and one console script 1.1.1 accepted are removed. The +parser is strict, so a launch or resume command that still carries an option +exits with `unrecognized arguments` before the session starts. + +| Removed | What to do | +|---|---| +| `--conc-sweep-timeout-sec` | Delete it. Each benchmark spawn is bounded by `INFERENCE_OPTIMIZER_BENCHMARK_TIMEOUT_SEC` (default `7800`) and, once a ready marker is seen, `INFERENCE_OPTIMIZER_BENCHMARK_SILENCE_TIMEOUT_SEC` (default `600`); sweep admission uses measured durations and the session deadline. | +| `--recipe-kb-strict-fingerprint` | Delete it. It was read nowhere; partition mode and `ep` are now part of the KB key, so the mismatch it promised to catch cannot occur. | +| `robustness-agent` console script, `tools/robustness_monitor.sh.example` | Nothing restarts a session automatically any more. Resume explicitly with `--resume-from`. | + +Resume now refuses, before any write or dispatch, when the session holds +execution ownership it cannot verify; ownership recorded by an earlier build is +diagnosed rather than assumed dead. After confirming that the task's process +tree, remote workers and Ray actor have stopped, release it with +`inference_optimizer recover-session --session-dir PATH --confirm-stopped TASK_ID --confirmation-reason TEXT`. + +Environment variables fail differently: nothing refuses them, so a box that +still exports one starts normally and behaves as though it were unset. Find +them by reading launch scripts and `.env`. + +| Removed or changed variable | What to do | +|---|---| +| `LLM_GATEWAY_KEY` as the only OpenAI-side credential | Rename it to `OPENAI_API_KEY`; value and endpoint are unchanged. `OPENAI_API_KEY` and the Anthropic-side names are the only credentials read, and `LLM_API_KEY`, `AMD_LLM_API_KEY`, `LLM_GATEWAY_KEY`, `AMD_API_KEY`, `LLM_PROXY_BASE_URL`, `LLM_PROXY_API_KEY` and `GEAK_WORK_DIR` are no longer forwarded to workers. | +| `HYPERLOOM_SUPERVISOR`, `HYPERLOOM_SUPERVISOR_ENFORCE`, `HYPERLOOM_SUPERVISOR_TICK_STALL_SEC`, `ROBUSTNESS_AGENT_ROOT`, `ROBUSTNESS_LLM_RCA_DISABLED` | Delete them. The Supervisor, the Monitor and the Robustness agent are gone. | +| `INFERENCE_OPTIMIZER_DISABLE_LOCAL_INFERENCEX`, `INFERENCE_OPTIMIZER_LOCAL_INFERENCEX_ROOT` | Delete them. Baseline benchmarks run from `$INFERENCEX_PATH` as set; a network-filesystem checkout warns once and is recorded as `network_fs`. | +| `MAGPIE_PATCH_STRICT` | Delete it. It had no effect once the atomic-write patch was dropped. | + +Installation prerequisites move with the stack: + +- TraceLens and its dependencies must be pre-installed in the coordinator's + Python interpreter; analysis no longer runs `pip install -e .` per request. + SGLang >= 0.5.18 profiles through TraceLens `kernel_shape_tool` under + `TRACELENS_ROOT` instead of git-applied roofline patches. +- The recommended SGLang image is + `lmsysorg/sglang-rocm:v0.5.20-rocm10-mi30x|mi35x-20260920`, and the vLLM image + is `rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0`. +- Bare-metal vLLM accepts ROCm 7.2.x (the `0.29.0+rocm723` wheel) or ROCm 10 + (a source build of `VLLM_SOURCE_REF`); any other ROCm stack is rejected, and + `VLLM_INSTALL_METHOD` can only confirm the detected route. +- The Slurm sbatch no longer forwards `ROCR_VISIBLE_DEVICES`, + `HIP_VISIBLE_DEVICES` or `CUDA_VISIBLE_DEVICES` into the container; isolation + comes from the cgroup. + +Report consumers have to follow these field changes; no aliases are kept: + +- SBD V6 `conc_sweep` comparison rows: `baseline_throughput` → `baseline_value` + and `optimized_throughput` → `optimized_value`. V6 failure rows are keyed by + `stage` instead of `phase`. +- `reports/kernel_optimization_summary.json` moves to `schema_version` 2 with a + `lane_totals` block, and `kernel_opt_outcome` can be `unvalidated`. +- A concurrency sweep that produced no comparable pair stops as `sweep_failed` + rather than `sweep_done`. +- `plateau_overrides.explore_empty_streak` counts benched rounds rather than + variants, and the GEMM and fusion E2E KEEP threshold defaults to 1% instead of + 3%. + +### 1.1.2 highlights + +- **ROCm 10 is a validated stack.** Docker runs SGLang 0.5.20 and vLLM on + ROCm 10 user space, bare-metal vLLM builds from source on a ROCm 10 host, and + `rocprof-compute` is installed and found on TheRock's pip-packaged ROCm. A + prebuilt SGLang image whose torch fails under `rocprofv3` is marked + profiling-unavailable instead of hanging the round. + +- **ENABLEMENT is the sixth phase of the loop.** Bring-up leaves + FRAMEWORK_AGENT and gets its own entry and exit, `phase_history` rows and + report section, bounded by the wall clock and `ENABLEMENT_MAX_ATTEMPTS` + rather than a budget share. A KEEP now ships an ordered replay recipe with a + `sufficient` / `insufficient` verdict under `enablement.recipe` in + `session_breakdown.json`. + +- **Automatic supervision is retired; limits and recovery are explicit.** The + Robustness agent, the runtime `recover` action and the Monitor/Supervisor + auto-resume are removed. Each benchmark spawn has a hard deadline plus a + silence limit armed only by a ready marker, resume refuses unverifiable + ownership instead of guessing, and `recover-session --confirm-stopped` + releases a stopped task's residue under the session lock. + +- **ATOM is supported end to end.** Its recovery workers are reaped, its kernel + backend defaults to forge, and detailed profiling adds `--mark-trace` and + ATOM's annotation switches only when the installed build supports them. + +- **KernelForge gains an assembly backend and keeps more of its work.** + `forge-loop --kernel-backend assembly` optimizes compiler-emitted AMDGPU + assembly behind a numerical contract. Warm starts fall back to a fuzzy KB + match across framework versions and GPUs, a lane's patch is merged against + the KEEPs landed before it, and a killed campaign's proven fusions are + salvaged instead of reported as REVERT. + +- **SBD V6 says which axis a session graded on.** `metadata.grading`, per-figure + `perf` and `graded_on` make AgentX and synthetic sessions distinguishable, and + the timeline records the events it used to drop, closing an event cut short by + a recording error as `interrupted`. + +### Other changes + - **Bump validated SGLang stack to 0.5.20 (ROCm 10 docker).** Updates `SGLANG_REF`, `SGLANG_PRETEND_VERSION`, recommended `lmsysorg/sglang-rocm:v0.5.20-rocm10-*` image tags, `models.tsv`, and the @@ -57,6 +167,66 @@ it. hook. `MAGPIE_PATCH_STRICT` no longer exists (setting it had no effect after the atomic patcher was removed). +- **Scheduler stalls found in the ROCm 10 six-leg run are fixed.** KERNEL entry + hooks no longer block the coordinator tick, conc-sweep boot rounds are no + longer counted as measured variants, vLLM roofline reads the real rank trace, + and a trace with zero GPU events stops repeat roofline scheduling with a + recorded reason. Final reports keep GEAK and GEMM-tuning outcomes, and + `kernelforge.gemm_tune run` exits non-zero on a hard AITER tune/serve + alignment mismatch. + +- **One rule picks every agent CLI.** Orchestration goes through the shared + `preferred_agent_backend` ranking, so a host with both sides configured and + only the Codex extra installed now orchestrates on Codex. Preflight and the + kernel agent no longer mirror the credential into other variable names. + +- **One kernel's KB records share one address.** `normalize_operator_name` + splits camelCase, so `KdaPackedDecodeKernel` and `kda_packed_decode_kernel` + meet, and framework versions reduce to their release (`0.24.0+rocm723` and + `v0.24.0` are both `0.24.0`). Task publication refuses a backend outside + `KERNEL_BACKENDS` instead of filing it as flydsl. Records stored under the old + spellings are reached once the KB Store migration has run. + +- **The Explore phase is dissolved into one attempt ledger.** The configuration + arm gives up after five benched rounds without a win rather than five + variants, a grid that kept an earlier variant is no longer read as dry, and a + resumed session extends `tested` / `rejected` / `name_index` instead of + replacing them. Apply-conflict retries no longer end FRAMEWORK_AGENT. + +- **The rewrite pipeline's caller owns its task shape.** A KEEP requires the + task's `compile_command` / `correctness_command` only on the assembly backend; + elsewhere the driver's suite decides. `--no-applyback` skips the framework + apply-back stage and returns its reserve to the search. + +- **Roofline on GPUs without a Hyperloom achievable-peak entry uses AMD vendor + theoretical peaks.** MI355X keeps its per-op PerfModel breakdown on vendor + peaks and MI308X moves to PerfModel; per-kernel bypass rows record the peak's + source. MI300X and MI325X are unchanged. + +- **A benchmark that served its whole protocol is kept despite a non-zero + wrapper exit.** When `completed_requests` reaches `num_prompts`, the round + succeeds with warning `nonzero_rc_after_complete_protocol:`; a round that + served fewer requests still fails. + +- **Profiling follows InferenceX's move of `benchmark_serving.py` under + `infx/bench_serving/`.** Every AgentX roofline had been failing with + `benchmark_serving_ok=False` on current checkouts. + +- **SGLang patches under `sglang/kernels/jit/` no longer trigger an editable + reinstall.** They rebuild as `content_addressed_jit`, which kept the candidate + server importable and the framework version equal to the baseline's; a + framework reinstall is followed by an import check that reverts the patch on + failure. + +- **An agentic AgentX recipe satisfies the server-phase pid contract.** + `aiperf_client.sh` accepts a recipe that already ran the lifecycle and wrote + `inferencex_result.json`, and flat metric keys sit beside the nested schema so + a healthy run no longer reads as 0.00 req/s. + +- **The kernel installer preserves operator-supplied GEAK checkouts.** Only the + installer-managed `GEAK@*` caches are re-cloned; a missing or invalid operator + root fails clearly instead of being overwritten. + ### Added - **An enablement session now ships an ordered replay recipe, and a verdict on @@ -145,7 +315,11 @@ it. killed and even the daemon refuses to remove it (`PID is zombie and can not be killed. Use the --init option ...`); the reclaim keeps a cgroup-level SIGKILL fallback for containers already in that state. `--name - hl--` makes a running container traceable back to its job. + hl--` makes a running container traceable back to its job. The + liveness check walks `/proc` and matches `CLAW_SESSION_ID=` literally + rather than through a truncating, regex-reading `pgrep -f`, which had + reclaimed a live neighbour's container, and a container younger than + `HL_ORPHAN_MIN_AGE_S` (default `900`) is never reclaimed. - **A vLLM profile round had its profiler bounds dropped before launch.** The argv preflight probe sees only `EXTRA_VLLM_ARGS`, while the launcher appends @@ -594,8 +768,7 @@ it. flags.** `--max-hours` carried an argparse default, so a resume passing no flags at all was indistinguishable from one passing the default: a 24 h session was shortened to 2 h and closed as `time_exhausted` before its first - action, and the objective was dropped on the way. This is the path - `robustness_monitor.sh` takes, which auto-resumes with no flags. The flag now + action, and the objective was dropped on the way. The flag now defaults to `None` and resolves to `DEFAULT_MAX_HOURS` only after the archive has had its chance at the persisted budget, so an absent flag restores 8 h while an explicit `--max-hours 2` wins over it. @@ -608,12 +781,6 @@ it. instead of over a steady-state window. The probe goes through the entry point that performs the expansion. -- **The robustness monitor called a session over while it was still running.** - It read the presence of `reports/final.*` as terminal, but the crash path - writes one as a safety net and a resume clears `stop_reason` without removing - it. `state.json` decides now, and the artifacts stand in only when there is - no state to read. - - **The IR-1 stale-process scan failed on an idle machine, and could not see an ATOM server.** It excluded only `os.getpid()`, so the launcher shell — whose argv quotes the whole command — matched the scan's own patterns and failed the diff --git a/examples/README.md b/examples/README.md index e05001aeda..a9d2170fca 100644 --- a/examples/README.md +++ b/examples/README.md @@ -41,7 +41,7 @@ From the agent terminal in that workspace, install the published release wheel into the current directory: ```bash -pip install hyperloom-inference-optimizer==1.1.1 --target . +pip install hyperloom-inference-optimizer==1.1.2 --target . ``` It is normal for the current directory to contain many Python package directories diff --git a/pyproject.toml b/pyproject.toml index 64c21bc6c1..b4c9f73754 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta" [project] name = "hyperloom-inference_optimizer" -version = "1.1.1" +version = "1.1.2" description = "Inference Optimizer — two-role (Orchestration/Critic) autonomous LLM inference optimization runtime for AMD GPU platforms. Kernel optimization is handled by programmatic Python handlers." readme = "README.md" requires-python = ">=3.10" diff --git a/src/hyperloom/inference_optimizer/assets/install_baremetal.sh b/src/hyperloom/inference_optimizer/assets/install_baremetal.sh index 9c39cacba3..69e3ba0999 100755 --- a/src/hyperloom/inference_optimizer/assets/install_baremetal.sh +++ b/src/hyperloom/inference_optimizer/assets/install_baremetal.sh @@ -28,7 +28,7 @@ DOTENV="${REPO_ROOT}/.env" HYPERLOOM_SKILL_PATH="${HYPERLOOM_SKILL_PATH:-${REPO_ROOT}/src/hyperloom/inference_optimizer/SKILL.md}" HYPERLOOM_WHEEL_REPO="${HYPERLOOM_WHEEL_REPO:-AMD-AGI/Hyperloom}" -HYPERLOOM_WHEEL_TAG="${HYPERLOOM_WHEEL_TAG:-v1.1.1}" +HYPERLOOM_WHEEL_TAG="${HYPERLOOM_WHEEL_TAG:-v1.1.2}" ROCM_PROFILER_HOTFIX_TARGET_LIB_DIR="${ROCM_PROFILER_HOTFIX_TARGET_LIB_DIR:-/opt/rocm/lib}" ROCM_PROFILER_HOTFIX_ASSET="${ROCM_PROFILER_HOTFIX_ASSET:-rocm-profiler-hotfix-libs.tar.gz}" # Installed name must outrank the vendor library in ldconfig's ordering. diff --git a/src/hyperloom/inference_optimizer/assets/slurm/models.tsv b/src/hyperloom/inference_optimizer/assets/slurm/models.tsv index 1a68825b44..1f1b578fdb 100644 --- a/src/hyperloom/inference_optimizer/assets/slurm/models.tsv +++ b/src/hyperloom/inference_optimizer/assets/slurm/models.tsv @@ -4,5 +4,5 @@ # ep_size is optional and trails target_gain so existing 13-column rows still # parse; an empty or absent value means expert parallelism stays at 1. deepseek_r1_sglang deepseek-ai/DeepSeek-R1 sglang MI300X lmsysorg/sglang-rocm:v0.5.20-rocm10-mi30x-20260920 8 fp8 1024 1024 64 moe_mla 6 10 1 -deepseek_r1_vllm deepseek-ai/DeepSeek-R1 vllm MI300X vllm/vllm-openai-rocm:v0.29.0 8 fp8 1024 1024 64 moe_mla 6 10 1 -gptoss_vllm openai/gpt-oss-120b vllm MI300X vllm/vllm-openai-rocm:v0.29.0 8 fp4 1024 1024 64 moe_swa 6 10 1 +deepseek_r1_vllm deepseek-ai/DeepSeek-R1 vllm MI300X rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0 8 fp8 1024 1024 64 moe_mla 6 10 1 +gptoss_vllm openai/gpt-oss-120b vllm MI300X rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0 8 fp4 1024 1024 64 moe_swa 6 10 1 diff --git a/src/hyperloom/inference_optimizer/tests/test_optimize_loop_walkthrough.py b/src/hyperloom/inference_optimizer/tests/test_optimize_loop_walkthrough.py index 2305b4dc24..883bb6d861 100644 --- a/src/hyperloom/inference_optimizer/tests/test_optimize_loop_walkthrough.py +++ b/src/hyperloom/inference_optimizer/tests/test_optimize_loop_walkthrough.py @@ -77,10 +77,17 @@ async def test_a_baseline_carries_the_run_into_the_optimisation_phase_with_work( @pytest.mark.asyncio -async def test_both_arms_dry_walks_the_rest_of_the_chain(session_dir: Path): +async def test_both_arms_dry_walks_the_rest_of_the_chain(session_dir: Path, monkeypatch: pytest.MonkeyPatch): """With nothing left to try, the run reaches CLOSE through every phase.""" + from hyperloom.orchestrator.kernel import request_handlers from hyperloom.orchestrator.state.attempt_ledger import record_config_attempt + def _no_geak_runner(tool_name: str) -> Path: + raise FileNotFoundError(tool_name) + + # KERNEL entry runs out of band; a real GEAK subprocess would decide how long the phase is held. + monkeypatch.setattr(request_handlers, "_kernel_agent_tool_path", _no_geak_runner) + coord = _coordinator(session_dir) try: state = coord.shared_state @@ -110,6 +117,7 @@ async def test_both_arms_dry_walks_the_rest_of_the_chain(session_dir: Path): await coord.tick(tick) assert state.phase == ps.PHASE_CLOSE + assert state.geak_result["error_class"] == "runner_not_found" visited = [to_phase for _, to_phase, _ in _chain(state)] assert visited[:2] == [ps.PHASE_PRELUDE, ps.PHASE_FRAMEWORK_AGENT] assert visited[-3:] == [ps.PHASE_KERNEL_AGENT, ps.PHASE_SWEEP, ps.PHASE_CLOSE] diff --git a/src/hyperloom/skills/hyperloom-setup/SKILL.md b/src/hyperloom/skills/hyperloom-setup/SKILL.md index a60fcf97b5..1a9783be3f 100644 --- a/src/hyperloom/skills/hyperloom-setup/SKILL.md +++ b/src/hyperloom/skills/hyperloom-setup/SKILL.md @@ -62,7 +62,7 @@ directory. Tell the user to open the intended dedicated workspace in the agent and install Hyperloom into that current directory: ```bash -pip install hyperloom-inference-optimizer==1.1.1 --target . +pip install hyperloom-inference-optimizer==1.1.2 --target . ``` Then stop and ask the user to rerun `/hyperloom-setup` from that workspace. @@ -164,15 +164,17 @@ value. recommendation: 1. `none`: use an already-installed vLLM/SGLang framework stack on the host. 2. `sglang`: install SGLang ROCm framework components (shared with the host torch). - 3. `vllm (isolated)`: install vLLM into a dedicated venv. vLLM's ROCm wheel - pins its own torch, so it runs in an isolated env and never touches the - host torch/SGLang stack. + 3. `vllm (isolated)`: install vLLM into a dedicated venv, leaving the host + torch/SGLang stack untouched. On ROCm 7.2.x the ROCm wheel brings its own + torch; on ROCm 10 vLLM is built from source in a venv that reuses the host + ROCm torch. - Do not mark any option as recommended. Present the three options in the exact order above without a default selection. 8. Only when the user chose `baremetal` **and** `vllm (isolated)` in Step 7, - briefly note that the installer enforces the vLLM 0.28.0+ glibc floor - (glibc >= 2.39). If setup later fails with that error, explain it in plain + briefly note that on ROCm 7.2.x the installer enforces the vLLM 0.28.0+ + wheel's glibc floor (glibc >= 2.39); the ROCm 10 source build has no such + gate. If setup later fails with that error, explain it in plain language and point the user to Docker mode or a pre-0.28 override — do not implement a second version gate here.