Skip to content

[ROCm] [Bugfix] [Release] Fix nightly rocm release pipeline #38263

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tjtanaa merged 9 commits into from
Mar 26, 2026
Merged

[ROCm] [Bugfix] [Release] Fix nightly rocm release pipeline #38263

Changes from 8 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
ab0bea0
fix buildkite yaml syntax error
tjtanaa Mar 26, 2026
0720ba8
fix syntax error
tjtanaa Mar 26, 2026
8fd0b72
mock
tjtanaa Mar 26, 2026
401b652
make it ready for review
tjtanaa Mar 26, 2026
01cf75e
make pr ready
tjtanaa Mar 26, 2026
71b6243
revert some other changes
tjtanaa Mar 26, 2026
99f2561
revert some other changes
tjtanaa Mar 26, 2026
5f57c1f
revert some other changes
tjtanaa Mar 26, 2026
4ac2d13
Merge branch 'main' into fix-nightly-rocm
tjtanaa Mar 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 2 additions & 33 deletions .buildkite/release-pipeline.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -402,38 +402,7 @@ steps:
# Save ECR tag for downstream jobs
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"

# Scenario 2: Image cached but wheels missing
elif [ "$${IMAGE_EXISTS}" = "true" ] && [ "$${WHEELS_EXIST}" = "false" ]; then
echo ""
echo "PARTIAL CACHE - Image exists, extracting wheels..."
echo ""

docker pull "$${ECR_CACHE_TAG}"

# Rebuild wheel extraction stage
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag rocm-base-debs:$${BUILDKITE_BUILD_NUMBER} \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--target debs_wheel_release \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--load \
.

# Extract and upload wheels
mkdir -p artifacts/rocm-base-wheels
cid=$(docker create rocm-base-debs:$${BUILDKITE_BUILD_NUMBER})
docker cp $${cid}:/app/debs/. artifacts/rocm-base-wheels/
docker rm $${cid}

.buildkite/scripts/cache-rocm-base-wheels.sh upload

buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"

# Scenario 3: Full rebuild needed
# Scenario 2: Full rebuild needed
Copy link
Copy Markdown
Collaborator Author

@tjtanaa tjtanaa Mar 26, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This case (scenario 2) is not useful as we are not pushing all of the intermediate stages to the ECR:

  • base
  • build_triton
  • build_pytorch
  • build_fa
  • build_aiter
  • build_amdsmi
  • build_mori
  • debs
  • debs_wheel_release

So we are not able to reuse the cache from base image through --cache-from type=registry,ref=public.ecr.aws/q9t5s3a7/vllm-release-repo:1227c9527d573e09-rocm-base

Example command:

DOCKER_BUILDKIT=1 docker buildx build \
  --file docker/Dockerfile.rocm_base \
  --tag rocm-base-debs:$${BUILDKITE_BUILD_NUMBER} \
  --target debs_wheel_release \
  --cache-from type=registry,ref=public.ecr.aws/q9t5s3a7/vllm-release-repo:1227c9527d573e09-rocm-base \
  --build-arg USE_SCCACHE=1 \
  --build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
  --build-arg SCCACHE_REGION_NAME=us-west-2 \
  --build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
  --load \
  .

else
Comment thread
tjtanaa marked this conversation as resolved.
echo ""
echo " CACHE MISS - Building from scratch..."
Expand All @@ -453,7 +422,7 @@ steps:
# Build wheel extraction stage
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag rocm-base-debs:${BUILDKITE_BUILD_NUMBER} \
--tag rocm-base-debs:$${BUILDKITE_BUILD_NUMBER} \
--target debs_wheel_release \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
Expand Down
Loading