From d18231d77dbbbd0a4a81e9e04dc029d48ecdc268 Mon Sep 17 00:00:00 2001 From: zhenyulincs Date: Sun, 5 Jul 2026 04:11:35 -0700 Subject: [PATCH 1/2] feat(rlix): lower whole-GPU residual gate default to 7.0; preload dual smoke MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 2026-07-05 audit (4x RTX 5090 cu130, dual smokes EXIT_CODE=0 in both modes): with MILES_TMS_HOOK_MODE=torch, tms pause releases only the 2.76 GiB of init-time weight buffers — Megatron optimizer state (~5.7 GiB, created lazily at the first optimizer step) never offloads, leaving 11.07 GiB whole-GPU residual at cycle 2 (why the default was 13.0). In preload mode everything is pausable: residual is a stable 5.02 GiB. With the paired miles changes the dual smoke runs preload, so the gate default drops 13.0 -> 7.0 (measured floor + margin). - miles_pipeline.py + miles_coordinator.py: default 7.0; gate RuntimeError now names the hook-mode coupling and the rollback knobs (MILES_TMS_HOOK_MODE=torch + MILES_MAX_RESIDUAL_GPU_MEM_GB=13) - run_smoke_dual.sh: MILES_TMS_HOOK_MODE=preload (pre-cu13 Blackwell fallback documented); e2e/inject_fault smokes stay torch-mode, pinned to MILES_MAX_RESIDUAL_GPU_MEM_GB=13 - wiring tests updated to 7.0; docs rows appended (append-only convention) Depends on rlops/miles: the cu13 preload guard relaxation PR AND rlops/miles#5 (demotes the per-engine /server_info accounting assert to diagnostics — its ~9.3 GiB accounting value is not physical residency and would false-fail a 7.0 threshold). E2E: paired trio (this + both miles changes) dual smoke on 4x RTX 5090: EXIT_CODE=0; gate readings 3.45-3.47 GB (cycle 1) / 5.02-5.04 GB (cycle 2) vs threshold 7.00 GB. Tests: 7 passed. Codex APPROVE_WITH_NOTES, 0 blockers. --- docs/m11-implementation-guide.md | 3 +++ rlix/pipeline/miles_coordinator.py | 12 +++++++----- rlix/pipeline/miles_pipeline.py | 7 +++++-- scripts/run_smoke_dual.sh | 16 +++++++++------- scripts/run_smoke_e2e.sh | 4 ++++ scripts/run_smoke_inject_fault.sh | 3 +++ tests/test_miles_residual_threshold_wiring.py | 8 ++++---- 7 files changed, 35 insertions(+), 18 deletions(-) diff --git a/docs/m11-implementation-guide.md b/docs/m11-implementation-guide.md index c4e01e8..6c40e90 100644 --- a/docs/m11-implementation-guide.md +++ b/docs/m11-implementation-guide.md @@ -535,6 +535,7 @@ Four small fixes from the M11 review report. | MED | MED1 | concurrent-resize stress test under `max_concurrency=4` | Open (Codex carryforward) | | NOTE | F8 / F10 | `orchestrator.cleanup_stale_pipelines()` RPC | M11.3 production hardening | | LOW | LOW1 | verify `generate_rollout_fully_async` accepts `rlix_hooks` kw | ~30 min check | +| MED | F2 v2 | whole-GPU used-memory gate landed (`rlops/rlix#17` merged 2026-07-05, default 13.0); follow-up lowers default to 7.0 + preload dual smoke (this PR) — basis: torch-hook residual 11.07 GiB vs preload 5.02 GiB (4x5090 cu130); rollback `MILES_TMS_HOOK_MODE=torch` + `MILES_MAX_RESIDUAL_GPU_MEM_GB=13` _(added 2026-07-05)_ | #17 merged; threshold-7 follow-up in flight | ## §3.6 Tianye's PR #14 — finalize-always-continue + release-on-sync-failure ✅ MERGED 2026-05-18 _(added 2026-05-24)_ @@ -718,6 +719,7 @@ New deferred-work entries surfaced during M11.2 real-overlap work (not in the hi | **F8 / F10 (NOTE)** | Detached coordinator persists after driver crash | M11.3 cleanup RPC scope | Stale actors on operator-killed runs | Still open; needs `orchestrator.cleanup_stale_pipelines()` RPC. | _(added 2026-05-24)_ | | **F9 (NOTE)** | `_split_pools_for_dual` silently ignored extra GPUs on non-2N machines | Static-analysis finding | Silent GPU leak on 5/6/7-GPU machines | **✅ RESOLVED** by miles `1487c3f` — explicit `ValueError` when `num_gpus_per_node != 2*infer_pool_size`; error message points at MILES_DUAL_P* workaround. | _(added 2026-05-24)_ | | **LOW1 (Codex carryforward)** | Verify `generate_rollout_fully_async` actually accepts `rlix_hooks` kw | `inspect.signature` forward silently no-ops if kw is missing | Demand signal path dead-codes silently | Open; ~30 min grep + startup assert. | _(added 2026-05-24)_ | +| **F2 v2 (MED)** | Whole-GPU used-memory gate landed (`rlops/rlix#17` merged 2026-07-05, default 13.0); follow-up threshold 7.0 + preload dual smoke (this PR) | Measured basis: torch-hook residual 11.07 GiB (Megatron optimizer state never offloads) vs preload 5.02 GiB stable on 4x5090 cu130 dual smoke | Tighter gate catches offload regressions; torch-hook deployments trip 7.0 by design | Landed; rollback knobs `MILES_TMS_HOOK_MODE=torch` + `MILES_MAX_RESIDUAL_GPU_MEM_GB=13`. | _(added 2026-07-05)_ | --- @@ -737,6 +739,7 @@ A vast.ai (or equivalent) GPU instance with: | Model checkpoint | Qwen2.5-0.5B at `/root/Qwen2.5-0.5B`, torch_dist at `/root/Qwen2.5-0.5B_torch_dist`, miles bundle at `/root/Qwen2.5-0.5B_miles/` | Any HF Megatron-compatible Qwen2.5-0.5B mirror works. | | Datasets | `/root/dapo-math-17k/dapo-math-17k.jsonl`, `/root/aime-2024/aime-2024.jsonl` | Eval is gated and skipped in smoke; the file just needs to exist. | | Branches | rlix on `zhenyu/miles-mvp-e2e` (or main once merged), miles on `zhenyu/m11-mvp-test` | See Appendix C for HEADs. | +| CUDA — update | 13.0 (cu130 wheels, 4×RTX 5090) | `run_smoke_dual.sh` now defaults to `MILES_TMS_HOOK_MODE=preload` (verified 2026-07-05: preload segfault gone on cu13 wheels; residual gate default 7.0 sized for preload). Pre-cu13 Blackwell stacks: fall back to `MILES_TMS_HOOK_MODE=torch` + `MILES_MAX_RESIDUAL_GPU_MEM_GB=13` (`run_smoke_e2e.sh`/`run_smoke_inject_fault.sh` keep that combination). _(added 2026-07-05)_ | ### 7.2 Smoke scripts diff --git a/rlix/pipeline/miles_coordinator.py b/rlix/pipeline/miles_coordinator.py index 6df001a..13e686b 100644 --- a/rlix/pipeline/miles_coordinator.py +++ b/rlix/pipeline/miles_coordinator.py @@ -439,12 +439,14 @@ def _shrink_workers(self, engine_indices: Set[int]) -> None: # Whole-GPU residual threshold (GiB). Miles shrink_engines logs # SGLang per-process/server_info attribution diagnostics with this # value; the hard whole-GPU gate runs in MilesPipeline after the - # engines report offloaded. Default 13.0 is a temporary smoke-safe - # value based on observed whole-GPU residuals with the known Megatron - # train-offload gap; lower it after that follow-up is fixed and - # re-measured. + # engines report offloaded. Default 7.0 = measured preload-mode + # whole-GPU floor 5.02 GiB (2026-07-05 audit, 4x RTX 5090 cu130 dual + # smoke) + margin; requires MILES_TMS_HOOK_MODE=preload (miles guard + # allows it on cu13+ Blackwell). torch-hook deployments must set + # MILES_MAX_RESIDUAL_GPU_MEM_GB=13. Next step: ~3.0 after the + # suspected tms.disable() weight-sync-window tail (~1.6 GiB) is fixed. residual_threshold_gb = parse_env_positive_float( - "MILES_MAX_RESIDUAL_GPU_MEM_GB", 13.0 + "MILES_MAX_RESIDUAL_GPU_MEM_GB", 7.0 ) shrunk = ray.get( rollout_manager.shrink_engines.remote( diff --git a/rlix/pipeline/miles_pipeline.py b/rlix/pipeline/miles_pipeline.py index be10691..ff93724 100644 --- a/rlix/pipeline/miles_pipeline.py +++ b/rlix/pipeline/miles_pipeline.py @@ -577,7 +577,7 @@ def _wait_for_overlap_engines_offloaded(self, allocated_train_gpus, *, timeout_s # intentionally broader than the SGLang process tree: it catches # non-child/orphan/co-tenant VRAM (Megatron/Miles/vLLM/etc.) on the # GPUs that must be clear before actor_train wakes up. - threshold_gb = parse_env_positive_float("MILES_MAX_RESIDUAL_GPU_MEM_GB", 13.0) + threshold_gb = parse_env_positive_float("MILES_MAX_RESIDUAL_GPU_MEM_GB", 7.0) max_used_gb = self._probe_max_used_gpu_mem_gb(target_gpu_ids) if max_used_gb is None: logger.warning( @@ -600,7 +600,10 @@ def _wait_for_overlap_engines_offloaded(self, allocated_train_gpus, *, timeout_s "check may be caused by non-SGLang co-tenants such as " "Megatron/Miles/vLLM/orphan processes. SGLang per-engine " "process-resident and server_info residual diagnostics are logged " - "engine-side for attribution." + "engine-side for attribution. Note: with MILES_TMS_HOOK_MODE=torch, " + "~11 GiB residual is expected (Megatron optimizer state is not " + "tms-pausable) — either run the preload hook mode (cu13+ stacks) " + "or set MILES_MAX_RESIDUAL_GPU_MEM_GB=13." ) @staticmethod diff --git a/scripts/run_smoke_dual.sh b/scripts/run_smoke_dual.sh index 63794c7..3f5e663 100755 --- a/scripts/run_smoke_dual.sh +++ b/scripts/run_smoke_dual.sh @@ -49,13 +49,15 @@ else fi export NVTE_ALLOW_NONDETERMINISTIC_ALGO=0 export CUBLAS_WORKSPACE_CONFIG=:4096:8 -export MILES_TMS_HOOK_MODE=torch -# B-14 mitigation on 16 GB GPUs: enable torch_memory_saver.pause() so -# Megatron weights actually move off-GPU between train cycles. Original -# `MILES_SKIP_TMS_PAUSE=1` was a workaround for tms.pause segfault on -# CUDA 12.9 + Blackwell + tms 0.0.9. On this RTX 4060 Ti + torch 2.11 -# stack, attempt to use real tms.pause to free GPU memory between -# rollouts. +# preload = full-catchment offload (optimizer state pausable), verified on +# cu130 Blackwell 2026-07-05; matches the 7.0 residual gate default. +# Fallback for pre-cu13 Blackwell stacks: +# export MILES_TMS_HOOK_MODE=torch AND MILES_MAX_RESIDUAL_GPU_MEM_GB=13. +export MILES_TMS_HOOK_MODE=preload +# Real tms.pause stays enabled (MILES_SKIP_TMS_PAUSE unset). The historical +# skip was a workaround for the tms 0.0.9 pause segfault on pre-cu13 +# Blackwell stacks; if you must revive it, also pin +# MILES_MAX_RESIDUAL_GPU_MEM_GB=13 (nothing offloads with the skip). # export MILES_SKIP_TMS_PAUSE=1 export MILES_SKIP_NODE_PG_PIN=1 # NOTE: PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is incompatible with diff --git a/scripts/run_smoke_e2e.sh b/scripts/run_smoke_e2e.sh index 1f8c070..9ad9a1c 100644 --- a/scripts/run_smoke_e2e.sh +++ b/scripts/run_smoke_e2e.sh @@ -26,6 +26,10 @@ export CUBLAS_WORKSPACE_CONFIG=:4096:8 # aggressive offload). export MILES_TMS_HOOK_MODE=torch export MILES_SKIP_TMS_PAUSE=1 +# torch-mode/skip-pause leaves ~11+ GiB residual on the train GPUs; pin the +# residual gate to the torch-mode rollback value (code default is now 7.0, +# sized for preload mode — see run_smoke_dual.sh). +export MILES_MAX_RESIDUAL_GPU_MEM_GB=13 # ROLL ResourceManagerProxy creates a node-PG holding all GPUs which would # pin the MilesPipeline actor and block Phase B's _create_placement_group. # Skip the pin so the coordinator can remove the node-PG mid-run. diff --git a/scripts/run_smoke_inject_fault.sh b/scripts/run_smoke_inject_fault.sh index c9614b7..2f6bff0 100755 --- a/scripts/run_smoke_inject_fault.sh +++ b/scripts/run_smoke_inject_fault.sh @@ -32,6 +32,9 @@ export CUBLAS_WORKSPACE_CONFIG=:4096:8 export MILES_TMS_HOOK_MODE=torch export MILES_SKIP_TMS_PAUSE=1 export MILES_SKIP_NODE_PG_PIN=1 +# torch-mode/skip-pause rollback value for the residual gate (code default 7.0 +# is sized for preload mode — see run_smoke_dual.sh). +export MILES_MAX_RESIDUAL_GPU_MEM_GB=13 # Phase 1 fault injection — raises inside train_group.train at first rollout. export MILES_INJECT_TRAIN_FAULT=1 diff --git a/tests/test_miles_residual_threshold_wiring.py b/tests/test_miles_residual_threshold_wiring.py index bab578d..1391837 100644 --- a/tests/test_miles_residual_threshold_wiring.py +++ b/tests/test_miles_residual_threshold_wiring.py @@ -33,9 +33,9 @@ def test_miles_uses_whole_gpu_residual_threshold() -> None: and isinstance(node.args[0], ast.Constant) and node.args[0].value == "MILES_MAX_RESIDUAL_GPU_MEM_GB" and isinstance(node.args[1], ast.Constant) - and node.args[1].value == 13.0 + and node.args[1].value == 7.0 for node in ast.walk(shrink_fn) - ), "_shrink_workers must parse the residual threshold env var with 13GB default" + ), "_shrink_workers must parse the residual threshold env var with 7GB default" assert any( isinstance(node, ast.Call) @@ -90,9 +90,9 @@ def test_miles_pipeline_enforces_whole_gpu_residual_gate() -> None: and isinstance(node.args[0], ast.Constant) and node.args[0].value == "MILES_MAX_RESIDUAL_GPU_MEM_GB" and isinstance(node.args[1], ast.Constant) - and node.args[1].value == 13.0 + and node.args[1].value == 7.0 for node in ast.walk(wait_fn) - ), "MilesPipeline hard gate must parse MILES_MAX_RESIDUAL_GPU_MEM_GB with 13GB default" + ), "MilesPipeline hard gate must parse MILES_MAX_RESIDUAL_GPU_MEM_GB with 7GB default" assert any(isinstance(node, ast.Raise) for node in ast.walk(wait_fn)), ( "MilesPipeline must raise when whole-GPU residual exceeds the threshold" ) From e4e8a741c45b630f589087028f8fe6877290f677 Mon Sep 17 00:00:00 2001 From: zhenyulincs Date: Sun, 5 Jul 2026 10:22:33 -0700 Subject: [PATCH 2/2] fix(rlix): authorize preload via existing miles escape hatch in dual smoke MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop the planned miles-side guard change entirely: instead of teaching the arch guard about CUDA versions, the dual smoke exports the guard's own MILES_TMS_ALLOW_PRELOAD_ON_BLACKWELL=1 (read by actors via raylet env inheritance — exported before ray start). This exact combination (unmodified miles + hatch + preload) is what passed the original route-A dual E2E (EXIT_CODE=0, residual 5.02 GiB). Codex APPROVE_WITH_NOTES. --- scripts/run_smoke_dual.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/scripts/run_smoke_dual.sh b/scripts/run_smoke_dual.sh index 3f5e663..03dc010 100755 --- a/scripts/run_smoke_dual.sh +++ b/scripts/run_smoke_dual.sh @@ -54,6 +54,11 @@ export CUBLAS_WORKSPACE_CONFIG=:4096:8 # Fallback for pre-cu13 Blackwell stacks: # export MILES_TMS_HOOK_MODE=torch AND MILES_MAX_RESIDUAL_GPU_MEM_GB=13. export MILES_TMS_HOOK_MODE=preload +# miles' arch guard blocks preload on Blackwell (a tms 0.0.9 + CUDA 12.9 +# segfault, since fixed on cu13 images — verified 2026-07-05 on 4x RTX 5090 +# cu130, mock + dual E2E). Authorize preload via the guard's own escape +# hatch; on pre-cu13 Blackwell stacks use the torch+13 fallback above instead. +export MILES_TMS_ALLOW_PRELOAD_ON_BLACKWELL=1 # Real tms.pause stays enabled (MILES_SKIP_TMS_PAUSE unset). The historical # skip was a workaround for the tms 0.0.9 pause segfault on pre-cu13 # Blackwell stacks; if you must revive it, also pin