Skip to content

fix(release)!: require gated promotion of external channel tags - #13333

Merged
DariuszPorowski merged 3 commits into
mainfrom
dariuszporowski-release-readiness-review
Oct 20, 2026
Merged

DariuszPorowski merged 3 commits into
mainfrom
dariuszporowski-release-readiness-review

Conversation

@DariuszPorowski

@DariuszPorowski DariuszPorowski commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

Summary

Complete the GoReleaser release lifecycle's recovery and publication guarantees:

  • Observe draft releases correctly and reload persisted image/CLI locks on every staging attempt, including failed-job retries whose original preflight reported absent.
  • Reject tagged sources without the complete verification, approval, recheck, and publication graph, including mandatory fail-closed image/CLI and Bicep alias promotion.
  • Stage Deployment Engine, dashboard, and both Bicep type artifacts under full-version references; bind their source revisions and digests into verification; promote external channel aliases only during serialized finalization.
  • Pin the release Bicep image to the staged full-version extensions, while retaining edge behavior.
  • Update the release runbook, and remove the completed implementation plan and review-note directory while retaining the design. GoReleaser OSS v2.18.3 is already pinned on main by chore(tools): bump versions #13338.

Coordinated companion changes:

Rollout order: merge and deploy the publisher first, then this PR and the dashboard PR in either order, and the Deployment Engine sender last; each step sends inputs that only the previous one accepts. The current Deployment Engine sender rejects dotted RC tags, so #684 is required before the first vX.Y.Z-rc.N release. Create no Radius or Deployment Engine release tag until all four are deployed.

Reason for change

The release-readiness review found that read-only preflight credentials could hide an existing draft and bypass lock downloads, the source compatibility check accepted the historical pre-gate pipeline, and external publishers still moved mutable channel references before verification. These gaps could rebuild staged outputs or expose an unverified patch through existing channel consumers.

Breaking automation change: direct Deployment Engine mirror requests targeting stable X.Y aliases are rejected. The approved finalizer owns channel promotion; mirror callers use a full-version destination plus a separate source tag. Existing Bicep channels without comparable source/version annotations require a separately reviewed migration, not blind overwrite or edits to old immutable artifacts. A new stable channel needs no legacy bootstrap.

N/A

How to test

Use a Linux checkout with Bash, Node.js, and the Go version from go.mod. The following exercises local fixtures, not live release publication:

make install-jq install-yq install-oras install-helm
make test-release-controller test-release-cutover test-release-publication \
  test-release-oci-artifacts test-release-parity-manifest \
  test-monitor-remote-workflow test-release-sboms

Expected: matching source/digest locks are reused; missing gates, disabled or failure-tolerant promotion steps, conflicting immutable artifacts, optional/skipped checks, and premature channel writes are rejected. The external-image tests preserve a dashboard manifest's digest and leave its independently owned latest tag unchanged. The Bicep promotion tests reject unverified/RC manifests and stop immediately on publisher failure.

Verify the actual release image build receives the immutable extension tag without building or publishing an image:

make -n build-bicep-linux-amd64 \
  REL_CHANNEL=0.61 REL_VERSION=0.61.2 BICEP_EXTENSION_TAG=0.61.2

Expected: the generate-bicepconfig.sh command receives 0.61.2, not 0.61. The cutover fixtures also verify that edge still resolves extensions through latest.

Validation performed:

Command Result
make test-release-controller test-release-cutover test-release-publication test-release-oci-artifacts test-release-parity-manifest Passed through WSL in a native Git fixture of the final head, with pinned ORAS.
make test-monitor-remote-workflow test-release-sboms Passed through WSL before the final contract fix; those suites are unaffected by it.
bash .github/scripts/verify-release-build-contract_test.sh Passed all 39 real-Git cases, including stable-alias promotion removal, if: false, and continue-on-error rejections.
bash .github/scripts/release-cutover_test.sh Passed all 13 tests.
node --test .github/scripts/verify-release-manifest_test.mjs .github/scripts/promote-bicep-types_test.mjs Passed all 8 tests.
goreleaser check Passed with pinned OSS v2.18.3 in a clean Git fixture using this configuration.
markdownlint-cli2 --config .github/linters/.markdownlint-cli2.yaml on the runbook and design note Passed, 0 errors.
cspell --config .github/linters/.cspell.yml on the runbook and design note Passed.
shellcheck on changed shell scripts Passed with the pinned version.
git diff --check Passed.

Changed YAML/JavaScript passed the repository Prettier configuration. Actionlint passed with only its unsupported queue key excluded; that key is supported by GitHub and already used by the release workflows.

No live release, registry publication, cloud deployment, or cross-version installation rehearsal was performed. Those remain the operator rollout gate described in the runbook; a green local fixture suite is not production sign-off.

File change summary

File Summary of change
.github/scripts/promote-bicep-types.mjs Build source/digest-bound promotion requests from a verified stable manifest and await both publishers.
.github/scripts/promote-bicep-types_test.mjs Cover valid requests, rejected manifests, repeat reconciliation, and immediate failure propagation.
.github/scripts/release-assets_test.mjs Verify optional lock downloads recover persisted artifacts without mutation.
.github/scripts/release-controller_test.sh Assert immutable DE conflicts stop dispatch and the mirror preserves separate source/destination tags.
.github/scripts/release-cutover_test.sh Cover unconditional lock recovery, AWS staging, gated Bicep promotion, and full-version image configuration.
.github/scripts/release-oci-artifacts.sh Verify external source locks and promote their channel tags with digest-preserving ORAS operations.
.github/scripts/release-oci-artifacts_test.sh Cover external provenance, digest preservation, and unchanged external edge aliases.
.github/scripts/release-parity-manifest.sh Observe staged Bicep artifacts through immutable full-version references.
.github/scripts/release-parity-manifest_test.sh Assert staged type references do not use mutable channels.
.github/scripts/validate-release-controller-plan.sh Separate full-version DE destination tags from upstream ACR source tags.
.github/scripts/validate-release-controller-plan_test.sh Verify that split for RC, final, and patch plans.
.github/scripts/verify-release-build-contract.sh Enforce the complete publication graph, draft-only staging, mandatory rechecks, ordering, and fail-closed alias promotion.
.github/scripts/verify-release-build-contract_test.sh Reject missing dependencies, bypassable conditions, skipped checks, disabled promotion, and incomplete pipelines.
.github/scripts/verify-release-manifest.mjs Bind all image locks and immutable Bicep artifacts to the approved source revisions.
.github/scripts/verify-release-manifest_test.mjs Cover distinct external sources, provenance conflicts, and immutable references.
.github/scripts/verify-release-publication.sh Include verified external image digests and source identities in the complete lock.
.github/workflows/__build-bicep-types.yaml Explicitly stage and await the plan's frozen AWS type source with a bounded monitor.
.github/workflows/__build-helm-chart.yaml Consume the staged full-version dashboard image rather than its channel alias.
.github/workflows/__release-controller.yaml Pass DE source/destination/provenance fields and reject immutable conflicts before dispatch.
.github/workflows/build-release.yaml Fix draft visibility and retries; finalize all external aliases after verification; pin Bicep image extensions.
.github/workflows/publish-de-image.yaml Preserve separate source tags and provenance while rejecting direct stable channel writes.
build/build.mk Add a release-scoped Bicep extension tag override with unchanged developer defaults.
build/test.mk Include the Bicep promotion tests in the existing publication target.
docs/contributing/contributing-releases/README.md Document corrected recovery, cross-repository rollout order, manual settings, and remaining live verification requirements.
eng/design-notes/tools/2026-03-goreleaser-release-lifecycle.md Remove a fragment that referenced the deleted implementation plan.
eng/design-notes/tools/2026-03-goreleaser-release-lifecycle-implementation-plan.md Remove the completed implementation plan as requested.
eng/design-notes/tools/2026-09-goreleaser-stack-review/ (20 Markdown files) Remove the completed index, follow-up plan, and 18 per-phase review notes as requested.

Recover draft locks on every staging attempt and enforce the complete
verification and approval graph at the tagged source. Stage source-bound
external artifacts under full versions and promote their channel aliases
only after verification, preserving external edge tags and image digests.

Update GoReleaser, regression coverage and the operator runbook, and remove
the completed implementation plan and review notes.

BREAKING CHANGE: Direct Deployment Engine mirror requests for stable X.Y
tags are rejected. Deploy the coordinated publisher and sender changes
before enabling this release pipeline. Existing unannotated Bicep channel
aliases require a reviewed migration.

Signed-off-by: Dariusz Porowski <3431813+DariuszPorowski@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@DariuszPorowski
DariuszPorowski requested review from a team as code owners October 17, 2026 20:38
Copilot AI balanced review requested due to automatic review settings October 17, 2026 20:38
@github-actions github-actions Bot added the pr:waiting-for-review A reviewer owns the next action; required approval is pending or unverifiable label Oct 17, 2026
@github-actions

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@DariuszPorowski DariuszPorowski added the pr:important High-impact changes needing special attention during release and in release notes label Oct 17, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-repository rollout requires human validation, and the unresolved alias-promotion contract gap needs correction.

1 open issue
What changed in this PR

Strengthens Radius release recovery and defers external channel promotion until verification and approval complete.

Changes:

  • Recover persisted locks and enforce the publication dependency graph.
  • Stage external artifacts under full-version references and promote verified digests.
  • Expand regression coverage, update GoReleaser and the runbook, and remove completed planning notes.
File Description
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​README.md Removes review index.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-18-release-final-cleanup.md Removes cleanup review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-17-helm-immutable-image-tags.md Removes Helm review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-16-release-publication-gate.md Removes publication-gate review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-15-release-controller.md Removes controller review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-14-goreleaser-sboms.md Removes SBOM review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-13-goreleaser-tag-cutover.md Removes cutover review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-12-prepare-release-backports.md Removes preparation review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-11-dotted-rc-identifiers.md Removes RC identifier review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-10-release-identifier-correlation.md Removes dispatch review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-09-tag-reconciliation.md Removes reconciliation review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-08-title-check-required.md Removes required-title-check review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-07-goreleaser-shadow.md Removes shadow-release review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-06-edge-tags.md Removes edge-tag review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-05-split-build-workflows.md Removes workflow-split review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-04-git-cliff-changelog.md Removes changelog review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-03-conventional-commit-title.md Removes advisory-title-check review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-02-goreleaser-snapshot.md Removes snapshot review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​pr-01-parity-manifest.md Removes parity review notes.
eng/​design-notes/​tools/​2026-09-goreleaser-stack-review/​follow-up-plan.md Removes follow-up plan.
eng/​design-notes/​tools/​2026-03-goreleaser-release-lifecycle-implementation-plan.md Removes implementation plan.
docs/​contributing/​contributing-releases/​README.md Documents rollout, gates, and recovery.
build/​tools.yaml Pins GoReleaser v2.18.3.
build/​tools.generated.mk Updates generated tool metadata.
build/​test.mk Includes Bicep promotion tests.
build/​build.mk Allows immutable Bicep extension tags.
.github/​workflows/​publish-de-image.yaml Separates source/destination tags and rejects channel writes.
.github/​workflows/​build-release.yaml Recovers locks and gates external promotion.
.github/​workflows/​__release-controller.yaml Passes provenance and rejects immutable conflicts.
.github/​workflows/​__build-helm-chart.yaml Consumes full-version dashboard images.
.github/​workflows/​__build-bicep-types.yaml Stages and awaits planned AWS types.
.github/​scripts/​verify-release-publication.sh Includes external images in verification locks.
.github/​scripts/​verify-release-manifest.mjs Checks external sources and immutable references.
.github/​scripts/​verify-release-manifest_test.mjs Covers provenance and reference conflicts.
.github/​scripts/​verify-release-build-contract.sh Validates publication gates and ordering.
.github/​scripts/​verify-release-build-contract_test.sh Covers missing and weakened gates.
.github/​scripts/​validate-release-controller-plan.sh Separates Deployment Engine source/destination tags.
.github/​scripts/​validate-release-controller-plan_test.sh Covers tag selection across release types.
.github/​scripts/​release-parity-manifest.sh Observes immutable staged Bicep references.
.github/​scripts/​release-parity-manifest_test.sh Checks staged reference selection.
.github/​scripts/​release-oci-artifacts.sh Verifies and promotes external image locks.
.github/​scripts/​release-oci-artifacts_test.sh Covers provenance and preserved edge aliases.
.github/​scripts/​release-cutover_test.sh Covers recovery and immutable Bicep configuration.
.github/​scripts/​release-controller_test.sh Covers immutable mirror destinations.
.github/​scripts/​release-assets_test.mjs Covers persisted-lock recovery.
.github/​scripts/​promote-bicep-types.mjs Dispatches digest-bound Bicep promotions.
.github/​scripts/​promote-bicep-types_test.mjs Covers promotion validation and failure propagation.

Review effort: Balanced

Comment thread .github/scripts/verify-release-build-contract.sh Outdated
@codecov

codecov Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.27%. Comparing base (c496ab2) to head (d09dfe0).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #13333      +/-   ##
==========================================
- Coverage   60.27%   60.27%   -0.01%     
==========================================
  Files         748      748              
  Lines       44255    44255              
==========================================
- Hits        26676    26675       -1     
- Misses      17579    17580       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Unit Tests

    2 files  ±0    422 suites  ±0   17m 53s ⏱️ -2s
7 200 tests ±0  7 198 ✅ ±0  2 💤 ±0  0 ❌ ±0 
8 383 runs  ±0  8 381 ✅ ±0  2 💤 ±0  0 ❌ ±0 

Results for commit d09dfe0. ± Comparison against base commit c496ab2.

♻️ This comment has been updated with latest results.

@github-actions github-actions Bot added pr:waiting-for-author The author needs to address human review feedback or human-endorsed inline feedback and removed pr:waiting-for-review A reviewer owns the next action; required approval is pending or unverifiable labels Oct 19, 2026
DariuszPorowski and others added 2 commits October 19, 2026 13:27
The release source contract only checked that the stable alias step existed
and was ordered. A disabled or failure-tolerant step could still publish the
GitHub Release without promoting image and CLI aliases.

Require the same promotion condition and fail-closed handling as the Bicep
promotion step, and add rejection fixtures for removal, disabling, and
continue-on-error.

Also correct the cross-repository rollout order in the release runbook and
remove a design-note fragment that referenced the deleted implementation plan.

Signed-off-by: Dariusz Porowski <3431813+DariuszPorowski@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Dariusz Porowski <3431813+DariuszPorowski@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added pr:waiting-for-review A reviewer owns the next action; required approval is pending or unverifiable and removed pr:waiting-for-author The author needs to address human review feedback or human-endorsed inline feedback labels Oct 19, 2026
@radius-functional-tests

radius-functional-tests Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Radius functional test overview

🔍 Go to test action run

Click here to see the test run details
Name Value
Repository radius-project/radius
Commit ref d09dfe0
Unique ID func3fcdc92271
Image tag test-func3fcdc92271-39319625687-1
  • Dapr: 1.14.4
  • Azure KeyVault CSI driver: 1.4.2
  • Azure Workload identity webhook: 1.3.0
  • Bicep recipe location ghcr.io/radius-project/dev/test/testrecipes/test-bicep-recipes/<name>:test-func3fcdc92271-39319625687-1
  • Terraform recipe location http://tf-module-server.radius-test-tf-module-server.svc.cluster.local/<name>.zip (in cluster)
  • applications-rp test image location: ghcr.io/radius-project/dev/applications-rp:test-func3fcdc92271-39319625687-1
  • dynamic-rp test image location: ghcr.io/radius-project/dev/dynamic-rp:test-func3fcdc92271-39319625687-1
  • controller test image location: ghcr.io/radius-project/dev/controller:test-func3fcdc92271-39319625687-1
  • ucp test image location: ghcr.io/radius-project/dev/ucpd:test-func3fcdc92271-39319625687-1
  • deployment-engine test image location: ghcr.io/radius-project/deployment-engine:latest

Test Status

⌛ Building Radius and pushing container images for functional tests...
✅ Container images build succeeded
⌛ Publishing Bicep Recipes for functional tests...
✅ Recipe publishing succeeded
⌛ Starting corerp-cloud functional tests...
⌛ Starting ucp-cloud functional tests...
✅ ucp-cloud functional tests succeeded
✅ corerp-cloud functional tests succeeded

@brooke-hamilton brooke-hamilton left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed d09dfe0; no actionable findings. Focused offline Node tests and shell syntax checks passed. Shell regression tests were blocked by missing jq/yq; live publication, registry permissions, and the coordinated release cycle were not exercised.

@github-actions github-actions Bot added pr:ready-for-queue Review, checks and mergeability permit adding this PR to the queue pr:review-approved Required reviews are approved; checks may still be pending and removed pr:waiting-for-review A reviewer owns the next action; required approval is pending or unverifiable labels Oct 20, 2026
@DariuszPorowski
DariuszPorowski added this pull request to the merge queue Oct 20, 2026
@github-actions github-actions Bot removed the pr:ready-for-queue Review, checks and mergeability permit adding this PR to the queue label Oct 20, 2026
Merged via the queue into main with commit 645b198 Oct 20, 2026
97 checks passed
@DariuszPorowski
DariuszPorowski deleted the dariuszporowski-release-readiness-review branch October 20, 2026 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr:important High-impact changes needing special attention during release and in release notes pr:review-approved Required reviews are approved; checks may still be pending

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants