[ROCm] [Nightly Docker Release] nightly rocm docker

[hongxiayang]

Purpose

nightly rocm docker is requested (#34067).

These changes are generated by agent to create night rocm docker.

Summary of changes: ROCm nightly Docker publish

1. New file: .buildkite/scripts/push-nightly-builds-rocm.sh
   Pulls the ROCm image from ECR (public.ecr.aws/q9t5s3a7/vllm-release-repo:<commit>rocm-base and public.ecr.aws/q9t5s3a7/vllm-release-repo:<commit>-rocm).
   Tags them as vllm/vllm-openai-rocm:base-nightly and vllm/vllm-openai-rocm:base-nightly-<commit>, vllm/vllm-openai-rocm:nightly and vllm/vllm-openai-rocm:nightly-<commit>.
   Pushes all the tags to Docker Hub.
   Local testing: BUILDKITE_COMMIT is required; DRY_RUN=1 runs ECR login + pull + local tagging only (no push).

2. .buildkite/release-pipeline.yaml
   New step “Publish nightly ROCm image to DockerHub”:
   Depends on build-rocm-nightly-image.
   Runs push-nightly-builds-rocm.sh, then cleanup-nightly-builds.sh nightly- vllm/vllm-openai-rocm.
   Uses the same Docker Hub login plugin as the CUDA nightly step.

3. .buildkite/scripts/cleanup-nightly-builds.sh
   New optional argument: [REPO] (default vllm/vllm-openai).
   Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO].
   REPO_API_URL and delete_tag now use REPO, so the script can clean both vllm/vllm-openaiand vllm/vllm-openai-rocm.

4. .buildkite/scripts/annotate-rocm-release.sh
Removed the comment: # TODO: Enable the nightly build for ROCm.
Resulting behavior
When the Release pipeline is run with NIGHTLY=1 (UI, CLI, or schedule), after the ROCm release image is built and pushed to ECR, the new step:
Pushes vllm/vllm-openai-rocm:nightly and vllm/vllm-openai-rocm:nightly-<commit> to Docker Hub.
Prunes old nightly-<commit> tags on vllm/vllm-openai-rocm, keeping the last 14. [This changes is kept because we still have nightly build for wheels to be done.

Implementation details

Currently it is using the rocm/vllm-dev:base as the base image to build Dockerfile.rocm_base. TODO by @tjtanaa to ensure that the base image is also generate from the Release Pipeline.

rocm/vllm-dev:base is a acceptable substitute for now as it is always used to build the vLLM docker image for AMD CI. So, the base docker image is well validated together with the vLLM codebase.

TODO by @tjtanaa to unify the nightly release pipeline with release pipeline

Test Plan

Mock the release pipeline with dry run.

Test Result

https://buildkite.com/vllm/release-pipeline-shadow/builds/3254/steps/canvas

The dry run of the upload shows the

[2026-03-13T15:01:41Z] + docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:c64df63ac1be1a4304716fa0aae7bfac77a416ec-rocm-base vllm/vllm-openai-rocm:base-nightly
[2026-03-13T15:01:41Z] + docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:c64df63ac1be1a4304716fa0aae7bfac77a416ec-rocm-base vllm/vllm-openai-rocm:base-nightly-c64df63ac1be1a4304716fa0aae7bfac77a416ec
[2026-03-13T15:01:41Z] + docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:c64df63ac1be1a4304716fa0aae7bfac77a416ec-rocm vllm/vllm-openai-rocm:nightly
[2026-03-13T15:01:41Z] + docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:c64df63ac1be1a4304716fa0aae7bfac77a416ec-rocm vllm/vllm-openai-rocm:nightly-c64df63ac1be1a4304716fa0aae7bfac77a416ec
[2026-03-13T15:01:41Z] + [[ 1 == \1 ]]
[2026-03-13T15:01:41Z] + echo '[DRY_RUN] Would push vllm/vllm-openai-rocm:base-nightly and vllm/vllm-openai-rocm:base-nightly-c64df63ac1be1a4304716fa0aae7bfac77a416ec'
[2026-03-13T15:01:41Z] [DRY_RUN] Would push vllm/vllm-openai-rocm:base-nightly and vllm/vllm-openai-rocm:base-nightly-c64df63ac1be1a4304716fa0aae7bfac77a416ec
[2026-03-13T15:01:41Z] + echo '[DRY_RUN] Would push vllm/vllm-openai-rocm:nightly and vllm/vllm-openai-rocm:nightly-c64df63ac1be1a4304716fa0aae7bfac77a416ec'
[2026-03-13T15:01:41Z] [DRY_RUN] Would push vllm/vllm-openai-rocm:nightly and vllm/vllm-openai-rocm:nightly-c64df63ac1be1a4304716fa0aae7bfac77a416ec
[2026-03-13T15:01:41Z] + echo '[DRY_RUN] Local tags created. Exiting without push.'
[2026-03-13T15:01:41Z] [DRY_RUN] Local tags created. Exiting without push.
[2026-03-13T15:01:41Z] + exit 0
[2026-03-13T15:01:41Z] + TAG_PREFIX=nightly-

Failure of Publish nightly ROCm image to DockerHub can be ignored because this mock pipeline does not have access to the actual dockerhub.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
• (Optional) Release notes update. If your change is user facing, please update the release notes draft in the Google Doc.

[gemini-code-assist]

Code Review

This pull request introduces a new Buildkite pipeline step to publish nightly ROCm Docker images. The changes include a new script to push images and modifications to the cleanup script to support different repositories. My review focuses on improving the maintainability of these new CI scripts by parameterizing the Docker repository name instead of hardcoding it. I've provided suggestions to use a Buildkite environment variable for the repository name and update the scripts to accept it as an argument. This will make the pipeline more flexible and easier to maintain.

[gemini-code-assist]

To improve maintainability, it's better to define the ROCm nightly repository name as an environment variable and pass it to the scripts. This avoids hardcoding the same string in multiple places and makes it easier to update in the future. This change is required to work with the suggested changes in push-nightly-builds-rocm.sh.

    commands:
      - "bash .buildkite/scripts/push-nightly-builds-rocm.sh \"$ROCM_NIGHTLY_REPO\""
      # Clean up old nightly builds (keep only last 14)
      - "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- \"$ROCM_NIGHTLY_REPO\""
    plugins:
      - docker-login#v3.0.0:
          username: vllmbot
          password-env: DOCKERHUB_TOKEN
    env:
      DOCKER_BUILDKIT: "1"
      DOCKERHUB_USERNAME: "vllmbot"
      ROCM_NIGHTLY_REPO: "vllm/vllm-openai-rocm"

[gemini-code-assist]

For better maintainability and to avoid hardcoding values, the Docker repository name should be passed as an argument to this script instead of being hardcoded. This makes the script more reusable and configurable. This change is similar to how cleanup-nightly-builds.sh was modified in this PR.

This suggestion refactors the script to accept the repository name as the first argument and updates the documentation comments accordingly. You will also need to update the call to this script in .buildkite/release-pipeline.yaml to pass the repository name.

# Push ROCm nightly image from ECR to a specified Docker Hub repository.
# e.g. vllm/vllm-openai-rocm:nightly and vllm/vllm-openai-rocm:nightly-<commit>.
# Run when NIGHTLY=1 after build-rocm-release-image has pushed to ECR.
#
# Local testing (no push to Docker Hub):
#   BUILDKITE_COMMIT=<commit-with-rocm-image-in-ecr> DRY_RUN=1 bash .buildkite/scripts/push-nightly-builds-rocm.sh <repo-name>
# Requires: AWS CLI configured (for ECR public login), Docker. For full run: Docker Hub login.

set -ex

# Use BUILDKITE_COMMIT from env (required; set to a commit that has ROCm image in ECR for local test)
BUILDKITE_COMMIT="${BUILDKITE_COMMIT:?Set BUILDKITE_COMMIT to the commit SHA that has the ROCm image in ECR (e.g. from a previous release pipeline run)}"
DRY_RUN="${DRY_RUN:-0}"
REPO="${1:?Please provide the Docker repository name as the first argument}"

ORIG_TAG="${BUILDKITE_COMMIT}-rocm"
TAG_NAME="nightly"
TAG_NAME_COMMIT="nightly-${BUILDKITE_COMMIT}"

echo "Pushing ROCm image from ECR tag $ORIG_TAG to Docker Hub as ${REPO}:${TAG_NAME} and ${REPO}:${TAG_NAME_COMMIT}"
[[ "$DRY_RUN" == "1" ]] && echo "[DRY_RUN] Skipping push to Docker Hub"

# Login to ECR and pull the image built by build-rocm-release-image
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:"$ORIG_TAG"

# Tag for Docker Hub (nightly and nightly-<commit>)
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:"$ORIG_TAG" "${REPO}:${TAG_NAME}"
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:"$ORIG_TAG" "${REPO}:${TAG_NAME_COMMIT}"

if [[ "$DRY_RUN" == "1" ]]; then
  echo "[DRY_RUN] Would push ${REPO}:${TAG_NAME} and ${REPO}:${TAG_NAME_COMMIT}"
  echo "[DRY_RUN] Local tags created. Exiting without push."
  exit 0
fi

# Push to Docker Hub (docker-login plugin runs before this step in CI)
docker push "${REPO}:${TAG_NAME}"
docker push "${REPO}:${TAG_NAME_COMMIT}"

echo "Pushed ${REPO}:${TAG_NAME} and ${REPO}:${TAG_NAME_COMMIT}"

[tjtanaa]

I will work on this after getting

1. First need to gate the Dockerfile.rocm_base changes, and make sure CI can run and validate the changes of Dockerfile.rocm_base immediately (WIP)
2. Release a running rocm_base docker image

This PR will need the following changes after the two feature above:

1. The docker image has to be built from the Dockerfile.rocm_base in vLLM.
2. Build with sccache enabled to improve building speed.
3. Will need to do trial run on vLLM infra.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @hongxiayang.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[hongxiayang]

Close this PR.
The related PR has been merged: #37283