Repository navigation
Flaky workflows: transient external dependency fetches need consistent bounded retries #12768
Description
Bug information
Steps to reproduce
Run scheduled workflows that fetch external dependencies or artifacts during setup or build. Recent examples include:
- Scheduled functional test (noncloud) failed - Run ID: 32486168852 #12766 — Dapr Helm chart download returned HTTP 503.
- Scheduled functional test failed - Run ID: 32407524062 #12747 —
proxy.golang.orgmodule ZIP downloads failed with HTTP/2INTERNAL_ERROR. - Scheduled functional test (noncloud) failed - Run ID: 31658032379 #12665 — Contour Helm chart download ended with EOF.
- Purge old images failed - Run ID: 29832301684 #12499 — GitHub Packages returned
{"message":"Server Error"}while listing package versions. - Flaky test: install-bicep fails with curl (56) Connection died downloading bicep-linux-x64 during tool install #12661 — Bicep-specific tracking issue with occurrences Scheduled functional test failed - Run ID: 31632402210 #12656, Scheduled functional test (noncloud) failed - Run ID: 31632368589 #12658, and Scheduled functional test (noncloud) failed - Run ID: 32291345300 #12725.
Observed behavior
A transient external HTTP or transport failure can fail an entire workflow before the relevant product tests execute. Retry ownership and behavior are inconsistent across Go module downloads, Helm and Dapr chart downloads, Bicep release assets, and GitHub Packages operations. Some fetches are one-shot; some delegate retry behavior to another tool; and some use fixed retry policies that can be exhausted.
These failures span unrelated upstream services and do not indicate a single upstream outage. This issue does not cover Radius/UCP readiness failures or internal API HTTP 503 responses.
Desired behavior
Read-only external fetch boundaries should use a consistent, finite retry policy for transient failures, including retryable HTTP statuses and transport errors. The policy should use bounded exponential backoff with jitter, honor Retry-After where available, preserve the final exit code and diagnostics, and fail promptly for deterministic errors such as authentication failures, 404 responses, checksum failures, invalid arguments, cancellations, and test failures.
Retries should target the narrow fetch operation rather than an entire mutating installation command or workflow job. Downloads should use temporary files, integrity verification where available, and atomic replacement.
Workaround
Rerun the failed job or workflow. This often succeeds but repeats broader setup and mutation work and does not provide consistent protection at the failing operation.
System information
rad Version
N/A; this affects repository workflow, build, and dependency-installation reliability.
Operating system
GitHub-hosted Ubuntu runners used by scheduled Radius workflows.
Additional context
Initial implementation candidates include a shared, locally testable Bash retry/download helper for workflow and installer operations, a bounded go mod download prefetch, and targeted Go retry around rad Helm chart pulls. GitHub Actions does not provide native per-step retry, so failed-job reruns should remain a final fallback rather than the primary mechanism.
Activity
radius-triage commentedon Aug 21, 2026
👋 @brooke-hamilton Thanks for filing this bug report.
A project maintainer will review this report and get back to you soon. If you'd like immediate help troubleshooting, please visit our Discord server.
For more information on our triage process please visit our triage overview
radius-triage commentedon Aug 24, 2026
👍 We've reviewed this issue and have agreed to add it to our backlog. Please subscribe to this issue for notifications, we'll provide updates when we pick it up.
We also welcome community contributions! If you would like to pick this item up sooner and submit a pull request, please visit our contribution guidelines and assign this to yourself by commenting "/assign" on this issue.
For more information on our triage process please visit our triage overview
nicolejms commentedon Oct 13, 2026
Another occurrence: non-cloud samples job failed creating the local registry because \docker run registry:2\ could not pull from Docker Hub (auth.docker.io token request cancelled, exit 125). Run: https://github.com/radius-project/radius/actions/runs/38192715059 (main, 2026-10-12). This needs the same bounded retry (or a mirrored/pre-pulled image).
ci: retry registry image pull in create-local-registry
docs: describe local registry image pull retry
ci: preserve registry pull failures and bound retries
ci: fix literal variable assertion in registry tests
ci: retry registry image pull in create-local-registry (radius-projec…
nicolejms commentedon Oct 16, 2026
Current assessment
#12768 is partially fixed, but it is not complete and should remain open.
Implemented coverage
go mod downloadin the repository workflows that compile Go.radHelm chart pull failures. It does not retry install or upgrade.registry:2image pull used by the local-registry action.unknown blobimage-push failure.These changes address the Go proxy, Bicep, Helm chart, and local-registry examples in the issue. There is not enough post-merge history to calculate a meaningful residual flake rate for the fixes merged on 2026-10-14.
Outstanding areas
.github/workflows/purge-old-images.yamlstill callssnok/container-retention-policyv3.1.0 directly. Inspection of the pinned action shows concurrency and rate-limit middleware, but no retry middleware around GitHub Packages or GHCR requests. This leaves the original{"message":"Server Error"}class from Purge old images failed - Run ID: 29832301684 #12499 uncovered. The last 20 scheduled purge runs, from 2026-10-06 through 2026-10-15, passed, but that is not evidence that the failure path is fixed because none of those runs exercised a transient failure.curl --retry-all-errorsto 14 installers, including Dapr and Helm. The scripts explicitly state that this also retries permanent HTTP 404 responses. They do not use the selective shared time budget added only to the Bicep installer by fix(build): bound retries for Bicep downloads #13291. This conflicts with the issue requirement to fail promptly for deterministic errors and to use a consistent bounded policy.registry.terraform.iodiscovery timeout after Terraform's own four attempts.pkg/recipes/terraform/execute.gostill callstf.Init(ctx)directly in apply and destroy paths and returns the failure. The measured occurrence was 1 / 7 scheduled workflow runs inspected from 2026-09-20 through 2026-09-26, using workflow runs as a proxy because some runs did not execute the affected tests. A provider mirror/cache or a narrowly classified initialization retry still needs investigation; retrying apply, destroy, or the whole test would be unsafe.wget .../deploy/install.sh | bashin.github/workflows/long-running-azure.yamland directhelm pullcalls in release verification scripts. These are not all tied to a confirmed flake, but they show that the repository has not yet reached the issue's stated goal of consistent retry ownership at external read boundaries.Recommendation
Keep #12768 open. Prioritize a repository-owned wrapper or upstream change for GitHub Packages reads, extend the Bicep selective policy to the other tool installers, and track the Terraform registry work separately while linking it here. Then audit scheduled-workflow download sites and either adopt the common policy or document why the invoked tool already provides an equivalent bounded policy.
This is a test/workflow reliability defect caused by non-hermetic external dependencies and inconsistent retry handling. No product invariant failure was identified in this audit. No code change or pull request was made.
fix(build): retry image pushes after unknown blob (radius-project#13288)
fix(cli): retry transient Helm chart downloads (radius-project#13286)
fix: standardize bounded external fetch retries (radius-project#13322)