Bring up vLLM on ROCm 10 and fix scheduler stalls found in the 6-leg run

conftest.py

@@ -0,0 +1,24 @@
+# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc.
+# SPDX-License-Identifier: MIT
+
+"""Repository-wide pytest bootstrap."""
+
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+
+def _pin_child_processes_to_this_tree() -> None:
+    """Put this checkout's ``src`` first on the child ``PYTHONPATH``.
+
+    ``pythonpath`` in ``pyproject.toml`` only reaches the pytest process. Tests
+    that shell out inherit the ambient one, so a Hyperloom installed elsewhere
+    on the host silently answers their imports instead of the tree under test.
+    """
+    src = str(Path(__file__).resolve().parent / "src")
+    existing = [entry for entry in os.environ.get("PYTHONPATH", "").split(os.pathsep) if entry]
+    os.environ["PYTHONPATH"] = os.pathsep.join([src, *(entry for entry in existing if entry != src)])
+
+
+_pin_child_processes_to_this_tree()

docs/compatibility.rst

@@ -123,8 +123,10 @@ The following inference frameworks are supported:
      - 10.0 (docker) / 7.2.4 (bare-metal wheel)
      - Default framework. Validated ``docker`` images use ROCm 10.0 (``rocm10`` tags below). On a ROCm 7.2.x bare-metal host the installer still derives ``SGLANG_ROCM_EXTRA=rocm724`` from ``torch.version.hip`` (see below).
    * - vLLM
-     - 7.2.3
-     - Do not mix frameworks within one session
+     - 10.0 (docker) / 7.2.3 (bare-metal wheel)
+     - Validated ``docker`` image is the ROCm 10.0 build below. The bare-metal
+       installer still resolves ``vllm==0.29.0+rocm723`` from wheels.vllm.ai, so
+       that path stays on ROCm 7.2.3. Do not mix frameworks within one session
    * - Atom
      - 7.2.4
      - AMD out-of-tree engine, launched as ``python3 -m atom.entrypoints.openai_server``. Container image only: ``install_baremetal.sh`` verifies ``atom`` but cannot install it, because ``--install-framework`` accepts only ``none``, ``sglang`` and ``vllm``. The kernel phase defaults to the KernelForge backend here (``KERNEL_OPT_BACKEND_ORDER`` is set to ``forge`` when you leave it unset). On a quantized non-vLLM backend GEAK must resolve a live rewrite seam rather than guess one, which forge does not require.
@@ -137,7 +139,7 @@ Container images
 
 Pick the image that matches your environment. Public Docker Hub refs are used
 on your own GPU machine: the official upstream ``lmsysorg/sglang-rocm:<tag>``
-for SGLang, ``vllm/vllm-openai-rocm:<tag>`` for vLLM and
+for SGLang, ``rocm/vllm:<tag>`` for vLLM and
 ``rocm/atom-dev:<tag>`` for Atom. If your deployment uses a private registry
 mirror, set the registry prefix accordingly.
 
@@ -164,7 +166,7 @@ nightly build and moves. Pin the versioned tag so a session stays reproducible.
 
 Browse all available tags at
 `hub.docker.com/r/lmsysorg/sglang-rocm/tags <https://hub.docker.com/r/lmsysorg/sglang-rocm/tags>`_,
-`hub.docker.com/r/vllm/vllm-openai-rocm/tags <https://hub.docker.com/r/vllm/vllm-openai-rocm/tags>`_
+`hub.docker.com/r/rocm/vllm/tags <https://hub.docker.com/r/rocm/vllm/tags>`_
 and
 `hub.docker.com/r/rocm/atom-dev/tags <https://hub.docker.com/r/rocm/atom-dev/tags>`_.
 
@@ -200,9 +202,9 @@ Hyperloom does not install ROCm or torch itself.
      - v0.29.0 (rocm723), isolated venv
      - Installs ``vllm==0.29.0+rocm723`` from the wheels.vllm.ai pip index on Ubuntu 24.04+. vLLM's ROCm wheel pins its own torch, so it installs into a dedicated venv (``--framework-env isolated``, the default for vLLM) and never touches the host torch.
 
-Bare-metal ROCm patch levels differ per framework, and each one matches its
-container image. The vLLM stack installs the ``rocm723`` variant (ROCm
-7.2.3), matching ``vllm/vllm-openai-rocm:v0.29.0``; the SGLang stack
+Bare-metal ROCm patch levels differ per framework. The vLLM stack installs the
+``rocm723`` variant (ROCm 7.2.3), one patch level behind the ROCm 10.0
+``rocm/vllm`` image the ``docker`` route uses; the SGLang stack
 installs from the ROCm 7.2.4 AMD wheel index when a host stays on ROCm 7.2.x;
 the recommended SGLang ``docker`` stack uses the two
 ``lmsysorg/sglang-rocm:v0.5.20-rocm10-*`` images (ROCm 10.0). ``docker`` mode is still

docs/install/install.md

@@ -427,15 +427,15 @@ It is recommended that you use a ROCm image that already ships the serving
 framework, so nothing needs to be installed inside the container beyond
 Hyperloom's runtime deps. The following images are recommended:
 
-- `vllm`: `docker.io/vllm/vllm-openai-rocm:v0.29.0`
+- `vllm`: `docker.io/rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0`
 - `sglang` MI300X: `docker.io/lmsysorg/sglang-rocm:v0.5.20-rocm10-mi30x-20260920`
 - `sglang` MI355X: `docker.io/lmsysorg/sglang-rocm:v0.5.20-rocm10-mi35x-20260920`
 
 Start a long-running container from the repo root, mounting it at the same path
 so `.env`, logs, and session artifacts stay valid:
 
 ```bash
-export HYPERLOOM_IMAGE=docker.io/vllm/vllm-openai-rocm:v0.29.0
+export HYPERLOOM_IMAGE=docker.io/rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0
 export REPO_ROOT="$(pwd -P)"
 docker run -d \
   --name "${HYPERLOOM_CONTAINER_NAME:-hyperloom-local}" \

examples/hyperloom-custom-advanced/SKILL.md

@@ -44,7 +44,7 @@ skip the user-approval step (#1314).
 
 Suggested Docker images:
 
-- `vllm`: `docker.io/vllm/vllm-openai-rocm:v0.29.0`
+- `vllm`: `docker.io/rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0`
 - `sglang` MI300X: `docker.io/lmsysorg/sglang-rocm:v0.5.20-rocm10-mi30x-20260920`
 - `sglang` MI355X: `docker.io/lmsysorg/sglang-rocm:v0.5.20-rocm10-mi35x-20260920`
 

examples/hyperloom-qwen3-14b-fp8-12h-forge/SKILL.md

@@ -78,7 +78,7 @@ skip the user-approval step (#1314).
 
 Suggested Docker images:
 
-- `vllm`: `docker.io/vllm/vllm-openai-rocm:v0.29.0`
+- `vllm`: `docker.io/rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0`
 - `sglang` MI300X: `docker.io/lmsysorg/sglang-rocm:v0.5.20-rocm10-mi30x-20260920`
 - `sglang` MI355X: `docker.io/lmsysorg/sglang-rocm:v0.5.20-rocm10-mi35x-20260920`
 

examples/hyperloom-qwen3-14b-fp8-12h/SKILL.md

@@ -34,7 +34,7 @@ skip the user-approval step (#1314).
 
 Suggested Docker images:
 
-- `vllm`: `docker.io/vllm/vllm-openai-rocm:v0.29.0`
+- `vllm`: `docker.io/rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0`
 - `sglang` MI300X: `docker.io/lmsysorg/sglang-rocm:v0.5.20-rocm10-mi30x-20260920`
 - `sglang` MI355X: `docker.io/lmsysorg/sglang-rocm:v0.5.20-rocm10-mi35x-20260920`
 

examples/hyperloom-qwen3-8b-3h/SKILL.md

@@ -34,7 +34,7 @@ skip the user-approval step (#1314).
 
 Suggested Docker images:
 
-- `vllm`: `docker.io/vllm/vllm-openai-rocm:v0.29.0`
+- `vllm`: `docker.io/rocm/vllm:rocm10.0.0_ubuntu24.04_py3.14_pytorch_2.12.0_vllm_0.27.0`
 - `sglang` MI300X: `docker.io/lmsysorg/sglang-rocm:v0.5.20-rocm10-mi30x-20260920`
 - `sglang` MI355X: `docker.io/lmsysorg/sglang-rocm:v0.5.20-rocm10-mi35x-20260920`
 

src/hyperloom/agents/kernel/tests/test_install_env_credentials.py

@@ -6,6 +6,7 @@
 
 from __future__ import annotations
 
+import os
 import re
 import subprocess
 import tempfile
@@ -16,6 +17,20 @@
 ROOT = Path(__file__).resolve().parents[1]
 INSTALL_SCRIPT = ROOT / "scripts" / "install.sh"
 
+# The env file's whole subject is "was this name already exported at launch", so every name these tests reason about
+# has to start out unset regardless of what the operator's shell carries.
+_CONTROLLED_ENV_NAMES = (
+    "ANTHROPIC_API_KEY",
+    "ANTHROPIC_AUTH_TOKEN",
+    "ANTHROPIC_BASE_URL",
+    "ANTHROPIC_CUSTOM_HEADERS",
+    "CLAUDE_CODE_OAUTH_TOKEN",
+    "OPENAI_API_KEY",
+    "OPENAI_BASE_URL",
+    "OPENAI_CUSTOM_HEADERS",
+    "MAGPIE_PATH",
+)
+
 _INSTALL_KEY = "ak-install-time-key"
 _INSTALL_URL = "https://gateway.install-time.example/v1"
 _INSTALL_TOKEN = "sk-ant-oat01-install-time-token"
@@ -24,6 +39,11 @@
 _ROTATED_TOKEN = "sk-ant-oat01-rotated-token"
 
 
+def _controlled_env() -> dict[str, str]:
+    """A copy of the ambient environment with every name these tests own removed."""
+    return {k: v for k, v in os.environ.items() if k not in _CONTROLLED_ENV_NAMES}
+
+
 def _sourceable_installer(dest_dir: Path) -> Path:
     """Write a copy of install.sh with the top-level ``main \"$@\"`` call removed."""
     text = INSTALL_SCRIPT.read_text(encoding="utf-8")
@@ -63,6 +83,7 @@ def _write_env_file(self, workdir: Path, repo_root: Path) -> Path:
             text=True,
             capture_output=True,
             timeout=60,
+            env=_controlled_env(),
         )
         self.assertEqual(
             proc.returncode,
@@ -86,6 +107,7 @@ def _source_and_report(self, env_file: Path, exported: dict[str, str]) -> tuple[
             text=True,
             capture_output=True,
             timeout=60,
+            env=_controlled_env(),
         )
         self.assertEqual(
             proc.returncode,

src/hyperloom/agents/kernel/tests/test_ray_version_mismatch.py

@@ -53,6 +53,22 @@ def _shutdown(*args, **kwargs):
     return fake
 
 
+def _settle_ray_init_state(timeout: float = 30.0) -> None:
+    """Wait out any abandoned ``ray-init`` runner, then clear the process-wide flag.
+
+    Clearing alone is not isolation: a runner abandoned by an earlier test sets
+    the flag whenever it finally returns, which under load is after the next
+    test has already cleared it. The runner releases ``_INIT_GATE`` as its last
+    action, so holding the gate proves no connect can still set the flag.
+    """
+    held = ray_runtime._INIT_GATE.acquire(timeout=timeout)
+    try:
+        ray_runtime._STALE_CONNECT_POSSIBLE.clear()
+    finally:
+        if held:
+            ray_runtime._INIT_GATE.release()
+
+
 @pytest.fixture(autouse=True)
 def _clear_stale_connect_flag():
     """``_STALE_CONNECT_POSSIBLE`` is process-wide by design, so tests must isolate it.
@@ -61,9 +77,9 @@ def _clear_stale_connect_flag():
     clear a session it never left behind -- the coupling is real in production
     too, where it is exactly the intended behaviour across legs.
     """
-    ray_runtime._STALE_CONNECT_POSSIBLE.clear()
+    _settle_ray_init_state()
     yield
-    ray_runtime._STALE_CONNECT_POSSIBLE.clear()
+    _settle_ray_init_state()
 
 
 def test_is_version_mismatch_detects_banner():

src/hyperloom/common/provenance.py

@@ -7,6 +7,7 @@
 
 import hashlib
 import os
+import gzip
 import re
 import subprocess  # nosec B404 - best-effort, guarded provenance probes only.
 import sys
@@ -29,6 +30,9 @@
     "aiter": ("AITER_COMMIT", "AITER_VERSION", "AITER_REF"),
     "sglang": ("SGLANG_VERSION", "SGL_VERSION"),
     "vllm": ("VLLM_VERSION",),
+    # Probe-only: no env var pins torch, and inventing one nothing writes would be a pin that never fires. It is
+    # recorded because the torch build selects Kineto's GPU backend, which decides whether a trace has any kernel.
+    "torch": (),
 }
 
 # The interpreter preflight resolved for the serving framework. ``--framework-env
@@ -164,7 +168,11 @@ def detect_stack_fingerprint(env: Mapping[str, str], *, probe: bool = True) -> d
 def _framework_site_packages(env: Mapping[str, str], component: str) -> list[str] | None:
     """``site-packages`` dirs of the interpreter preflight resolved, if any."""
     resolved_for = _env_first(env, RESOLVED_FRAMEWORK_ENV)
-    if not resolved_for or resolved_for.strip().lower() != component:
+    resolved_for = resolved_for.strip().lower() if resolved_for else ""
+    # torch rides whichever interpreter serves: the isolated vLLM overlay pins its own, and the orchestrator's answer
+    # would name a build the run never executed on.
+    rides_framework_venv = component == "torch" or (resolved_for == "vllm" and component == "aiter")
+    if component != resolved_for and not rides_framework_venv:
         return None
     python_exe = _env_first(env, RESOLVED_FRAMEWORK_PYTHON_ENV)
     if not python_exe or python_exe == sys.executable:
@@ -185,13 +193,42 @@ def _framework_site_packages(env: Mapping[str, str], component: str) -> list[str
     return hits or None
 
 
+#: Kineto names its GPU backend in the trace's top-level metadata, which PyTorch writes ahead of ``traceEvents``, so
+#: only the head of the file is read rather than the whole (often multi-hundred-MB) event array.
+_KINETO_BACKEND_RE = re.compile(r"\"(rocprofiler-sdk|roctracer)_version\"\s*:\s*\"?([0-9][0-9.]*)\"?")
+_TRACE_HEAD_BYTES = 65536
+
+
+def detect_kineto_backend(trace_path: str | Path) -> str:
+    """The GPU backend Kineto recorded a trace with, e.g. ``rocprofiler-sdk 1.3``, or ``""`` when unreadable.
+
+    The torch build selects this backend and the trace is the only place that states it outright; an unreadable trace
+    yields empty rather than a guess, because this is recorded as provenance.
+    """
+    path = Path(trace_path)
+    try:
+        opener = gzip.open if path.suffix == ".gz" else open
+        with opener(path, "rb") as handle:  # type: ignore[operator]
+            head = handle.read(_TRACE_HEAD_BYTES)
+    except (OSError, EOFError, gzip.BadGzipFile):
+        return ""
+    found = _KINETO_BACKEND_RE.search(head.decode("utf-8", errors="replace"))
+    return f"{found.group(1)} {found.group(2)}" if found else ""
+
+
 def _probe_pkg_version(component: str, venv_path: list[str] | None = None) -> str:
     """Best-effort installed-package version for a stack component."""
     # AITER renamed its distribution from ``aiter`` to ``amd-aiter`` at v0.1.8, so ``aiter`` was simply the wrong
     # name. It is not kept as a fallback: on PyPI that name belongs to an unrelated async-iterator library, and
     # recording its ``0.13.20191203`` as the AITER version would be worse than recording nothing, because it looks
     # like an answer. Installs older than v0.1.8 are covered by ``AITER_REF`` instead, which is exact.
-    dist = {"sglang": "sglang", "vllm": "vllm", "aiter": "amd-aiter"}.get(component)
+    dist = {
+        "rocm": "rocm-sdk-core",
+        "sglang": "sglang",
+        "vllm": "vllm",
+        "aiter": "amd-aiter",
+        "torch": "torch",
+    }.get(component)
     if not dist:
         return ""
     if venv_path is not None: