fix (docker): stop the docker gate from skipping the whole test suite - #1798
Merged
Conversation
Zhichenzzz
requested review from
guapisolo,
yueming-yuan and
yushengsu-thu
as code owners
July 26, 2026 06:32
#1791 made resolve-ci-image depend on docker-build so that a docker PR runs its suites inside the freshly built image. docker-build only runs when docker files change, so on every other PR it is skipped -- and GitHub propagates a skip down the entire needs closure, including through resolve-ci-image's always() guard. stage-a-cpu carries no if: at all, so it was skipped, and stage-b/stage-c gate on stage-a-cpu succeeding, so the whole matrix went with it. The run still reports success, because a skipped job is not a failed one. Since #1791 merged, PRs 1792/1793/1794 each completed "green" with zero test jobs expanded; the last run that actually executed anything was #1790 at 07-24T21:17Z, before the merge. #1791's own CI was green and did run, because it changed docker/Dockerfile -- the regression could not appear in the PR that introduced it. Fix keeps the new behavior and restores the old graph: docker-paths and docker-build now always reach a conclusion instead of skipping, so nothing downstream is poisoned. With nothing to build, docker-build no-ops on ubuntu-latest rather than occupying a GPU runner, and reports built=false; resolve-ci-image reads that output instead of the job result. All seven stage-* jobs are byte-identical to their pre-#1791 definitions. docker-paths also loses its `if: github.event_name == 'pull_request'`, which had the same effect on the nightly cron: no PR context meant a skipped job and a silently empty scheduled run. It now reports changed=false for non-PR events.
Zhichenzzz
force-pushed
the
zhichen/fix-ci-docker-skip-cascade
branch
from
July 26, 2026 06:34
cee1b6f to
b20b4b7
Compare
yueming-yuan
approved these changes
Jul 26, 2026
Zhichenzzz
force-pushed
the
zhichen/fix-ci-docker-skip-cascade
branch
from
July 26, 2026 06:54
9a5f286 to
666fd4c
Compare
The ternary mixed a JSON array with a bare string, and only the string branch is exercised by a PR that touches no docker file. Make both branches arrays so the untested branch cannot fail on a type mismatch.
Zhichenzzz
force-pushed
the
zhichen/fix-ci-docker-skip-cascade
branch
from
July 26, 2026 07:23
666fd4c to
edf9b71
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
#1791 made
resolve-ci-imagedepend ondocker-build, so a docker PR runs its suites inside the freshly built image.docker-buildonly runs when docker files change, so on every other PR it is skipped — and GitHub propagates a skip down the wholeneedsclosure, including throughresolve-ci-image'salways()guard.stage-a-cpuhas noif:at all, so it was skipped too, andstage-b-*/stage-c-*gate onstage-a-cpusucceeding, so the entire matrix went with it.The run still reports success, because a skipped job is not a failed one:
Sub-jobs that actually expanded, per PR's
PR Testrun:docker-build#1791 merged 07-25 01:42 UTC; #1790 was the last run that executed anything.
Two things made this hard to notice. #1791's own CI was green and really ran, because it changed
docker/Dockerfile— the regression could not appear in the PR that introduced it. And labels don't reveal it either:resolve-ci-policyparses them fine (labels=[run-ci-megatron run-ci-lora]), but they are consumed byrun_suite.pyinside the stage jobs, which never start.The fix
Keep #1791's behavior, restore the old graph. The rule is that no job feeding the stage chain may end up "skipped".
docker-paths— always runs. It also losesif: github.event_name == 'pull_request', which had the same effect on the nightly cron: no PR context meant a skipped job and a silently empty scheduled run. It now reportschanged=falsefor non-PR events.docker-build— always reaches a conclusion, underif: always() && !cancelled(). With nothing to build every step is a no-op onubuntu-latest, off the GPU fleet, and it reportsbuilt=false. It deliberately does not requiredocker-pathsto have succeeded: if that job ever fails, the chain must still carry on rather than silently disable the suite again.resolve-ci-image— readsneeds.docker-build.outputs.builtinstead of the job result, and needs onlydocker-build(it never useddocker-paths' output). It keeps noif:guard on purpose: a real build failure should stop the suites rather than run them against a stale image.Fork PRs keep the pre-existing behavior from #1791 — they cannot push to the registry, so they take the no-op path and test against the released image.
Nothing else changed. Verified by hashing the file in three parts against
775c87378^:resolve-ci-policye65866e1…)stage-a-cpu→ end of file (all 7 stage jobs)cd1fa924…)The diff against pre-#1791 also removes zero lines — every line of the old pipeline is still there.
What a non-docker PR gets
Exactly the pre-#1791 path: same image (
radixark/miles:dev), same tag-resolution order, same stage definitions, same fast-fail gating, same label-driven suite selection. The only addition is two shortubuntu-latestjobs (docker-paths+ a no-opdocker-build) ahead of the chain.Test plan
This PR is its own test: it touches no docker path, so
docker-buildtakes the no-op branch — the exact case that was broken.Latest run on this branch: 30 sub-jobs expanded, all green (matching #1790's pre-regression count of 30), with real tests executing —
stage-a-cpu(4 partitions)stage-b-cpustage-b-2-gpu-h200stage-c-*No tests found … cadence=regular— label-gated, and this PR carries norun-ci-*labeldocker-buildbehaved as intended on the no-op path: ran onubuntu-latest, every build step skipped,Nothing to buildsucceeded.Reviewers: please check expanded sub-jobs, not the run conclusion — the failure mode being fixed is a green run with nothing in it.
Not covered here: the
changed=truebranch never executes on this PR, so the real build path — GPU runner,built=true, suites pullingradixark/miles:pr-<N>— will first be exercised by the next PR that touches a docker file.