Flaky test: Run upgrade-noncloud functional tests: An unknown error was returned while testing Radius API status 503

Area for Improvement

The upgrade noncloud functional tests are seeing intermittent failures during execution

Observed behavior

    rptest.go:177: Loaded workspace: kind-kind (Kubernetes (context=kind-kind))
    rptest.go:180: 
        	Error Trace:	/home/runner/work/radius/radius/test/rp/rptest.go:180
        	            				/home/runner/work/radius/radius/test/functional-portable/upgrade/upgrade_test.go:233
        	            				/home/runner/work/radius/radius/test/functional-portable/upgrade/upgrade_test.go:234
        	            				/opt/hostedtoolcache/go/1.26.1/x64/src/runtime/asm_amd64.s:1771
        	Error:      	Received unexpected error:
        	            	An unknown error was returned while testing Radius API status:
        	            	Status Code: 503
        	            	Response Headers:
        	            	  Audit-Id: 73f9b0ea-defe-42e3-850a-8093e4711e80
        	            	  Cache-Control: no-cache, private
        	            	  Content-Length: 20
        	            	  Content-Type: text/plain; charset=utf-8
        	            	  Date: Fri, 08 May 2026 13:45:02 GMT
        	            	  X-Content-Type-Options: nosniff
        	            	  X-Kubernetes-Pf-Flowschema-Uid: 211e83f3-102c-4be3-b085-b3168d0ac12a
        	            	  X-Kubernetes-Pf-Prioritylevel-Uid: e153f0a4-3684-4848-b180-47599071d202
        	            	Response Body:
        	            	service unavailable
        	            	
        	Test:       	Test_PreflightContainer/Disabled
        	Messages:   	failed to create ApplicationsManagementClient
    upgrade_test.go:138: 
        	Error Trace:	/home/runner/work/radius/radius/test/functional-portable/upgrade/upgrade_test.go:225
        	            				/home/runner/work/radius/radius/test/functional-portable/upgrade/upgrade_test.go:138
        	Error:      	Condition never satisfied
        	Test:       	Test_PreflightContainer/Disabled
        	Messages:   	Control plane did not become available within timeout

Desired behavior

We should investigate why the scheduled tests are failing for this job

Proposed Fix

No response

rad Version

latest

Operating system

No response

Additional context

No response

@sk593

👋 @sk593 Thanks for filing this issue.

A project maintainer will review this issue and get back to you soon.

We also welcome community contributions! If you would like to pick this item up sooner and submit a pull request, please visit our contribution guidelines and assign this to yourself by commenting "/assign" on this issue.

For more information on our triage process please visit our triage overview

A 503 error is usually a platform issue or transient issue, but in this case, it happens too frequently to be so. We should take a closer look.

👍 We've reviewed this issue and have agreed to add it to our backlog. Please subscribe to this issue for notifications, we'll provide updates when we pick it up.

We also welcome community contributions! If you would like to pick this item up sooner and submit a pull request, please visit our contribution guidelines and assign this to yourself by commenting "/assign" on this issue.

For more information on our triage process please visit our triage overview

Recurring data + proposed fixes — `Test_PreflightContainer` timing/readiness flake

Adding fresh evidence and concrete fix proposals to this existing flake. Same test and same failure family as the original report (control-plane readiness / helm --wait timeout).

Recent evidence (June 2026)

functional-test-noncloud.yaml on main: 4 of the last 8 runs failed; at least 2 on the upgrade-noncloud leg (runs 28102985422, 28062964410).
Both subtests flake depending on which one loses the timing race:
- main run 28102985422: Test_PreflightContainer/**Enabled** failed (143.4s)
- PR run 28137799045: Test_PreflightContainer/**Disabled** failed (189.5s)
The /Disabled failure at ~189s ≈ just over the 3m helm --wait timeout, which points at install/upgrade readiness rather than assertion logic — consistent with the originally-reported 503 "Control plane did not become available within timeout".

Root-cause factors

In test/functional-portable/upgrade/upgrade_test.go:

helmTimeout = "3m" is too tight. A cold kind cluster installing the full chart (ucp, applications-rp, dynamic-rp, controller, DE, contour) under --wait regularly approaches or exceeds 3m on shared CI runners.
Aggregated API 503 race. After helm upgrade, the UCP aggregated APIService briefly returns 503 while pods roll; waitForControlPlane can construct the management client during that window.
Fixed time.Sleep(5 * time.Second) negative check in testPreflightDisabled is a race — it asserts "no preflight Job" after an arbitrary 5s rather than synchronizing on helm upgrade --wait completion.
apiServiceDeregistrationWait = 3s fixed sleep in cleanupAndWait is likewise arbitrary versus polling actual APIService deregistration.

Proposed fixes

Bump helmTimeout to 5–6m (largest single lever).
In waitForControlPlane, poll the UCP APIService Available=True condition (and treat 503 as retryable) before building the management client.
Replace the 5s sleep in /Disabled with a deterministic wait on helm upgrade --wait completion, then a single Job Get (the upgrade completing is the correct synchronization point).
Replace cleanupAndWait's fixed 3s deregistration sleep with polling the APIService list until the Radius-owned entry is gone.
Longer term: per Restructure the test matrix around correct test responsibilities (functional vs integration/E2E) #12163, reclassify this as a serial integration/E2E test rather than a functional-matrix leg.

Fix flaky upgrade test: increase timeouts and replace fixed sleeps wi…

Still reproducible after #12245 (August 20, 2026)

This flake is still active on main after the June mitigations in #12245 were merged.

Fresh evidence

main run 32407484384, SHA 2ba622f8dc8f55b86f7bd9fdf53fdacfa5e4045e: Test_PreflightContainer/Disabled failed after about 5 minutes with HTTP 503 service unavailable and Control plane did not become available within timeout.
PR #12746 run 32421685064, attempt 1, whose merge base is that exact main SHA: same test, HTTP 503, and readiness timeout.
The same PR run, attempt 2: rerunning only the failed job reproduced the same failure again.

This is unrelated to #12746: that PR changes only workflow templates, their structural shell test, build wiring for that test, and documentation. More importantly, main itself fails at the PR's exact merge base with the same signature.

Recent scheduled history also remains meaningfully flaky: 4 of the latest 10 main runs of Functional Tests (with Non-Cloud Resources) failed as of this check.

What this tells us about the merged mitigation

#12245 is present in the failing revision (98f5623de9f660301880ea95c775210a92afcb2b is an ancestor). The current test already has:

helmTimeout = "5m"
controlPlaneTimeout = 4 * time.Minute
retry handling in waitForControlPlane
polling for APIService deregistration

The failure therefore survives the previously proposed timeout increases and fixed-sleep replacements. It is not just the old 3-minute Helm timeout.

The next investigation should capture why the aggregated UCP APIService remains unavailable for the full readiness window: APIService conditions, backing Service/EndpointSlices, and UCP pod status/events/logs at timeout. Without those diagnostics, another timeout increase would only mask the underlying unavailable backend and make the flaky leg slower.

I searched open issues and PRs before adding this update. There is no open PR referencing this issue or changing helmTimeout, waitForControlPlane, or apiServiceDeregistrationWait. #12163 remains open as a broader proposal to reclassify this as serial integration/E2E coverage, but it does not currently provide a fix for the 503.

fix(chart): make the readiness hook upgrade-compatible

test(upgrade): rely on Helm readiness and install once

fix(chart): derive readiness image from UCP

test(upgrade): rely on Helm readiness and install once

fix(chart): honor readiness image override precedence

test(upgrade): rely on Helm readiness and install once

fix(chart): wait for Radius aggregated API readiness (radius-project#…

test(upgrade): trust Helm readiness and install once (radius-project#…

fix(chart): wait for Radius aggregated API readiness (#12785)

Closing following an issue audit against Radius main at ec97901 and the current referenced implementations.

The August comment correctly rules out closing on #12245's timeout/sleep mitigations alone. Later merged #12785 addresses the actual boundary: UCP must be listening before entering Service endpoints, and a mandatory Helm hook verifies the aggregated API through kube-apiserver before installation/upgrade succeeds. Its live validation reproduced the original 503 without the hook and passed the unchanged upgrade test with it. Merged #12790 then removes the fragile parallel readiness check and installs once before sequential enabled/disabled scenarios. Both fixes are present in the baseline, providing stronger exact evidence than the earlier partial mitigation.

Evidence:

Flaky test: Run upgrade-noncloud functional tests: An unknown error was returned while testing Radius API status 503 #11841 (comment) — post-Fix flaky upgrade test: replace fixed sleeps with polling and increase timeouts #12245 recurrence, explicitly considered.
fix(chart): wait for Radius aggregated API readiness #12785 — merged 2026-08-24; exact reproduction, hook-removal comparison, live kind and CI validation.
test(upgrade): trust Helm readiness and install once #12790 — merged 2026-08-24; single-install test lifecycle and successful live enabled/disabled validation.

radius/deploy/Chart/templates/ucp/deployment.yaml

Lines 84 to 104 in ec97901

    
           # The kube-apiserver aggregation layer forwards api.ucp.dev requests to this 
        
           # pod through the ucp Service. Without a readiness gate the pod is added to 
        
           # the Service endpoints as soon as the container starts, so the aggregator can 
        
           # route to a backend that is not listening yet and return 503 to callers. 
        
           # 
        
           # The probe targets the Kubernetes discovery document, which is the same 
        
           # endpoint the aggregator polls to decide APIService availability and is 
        
           # served without authentication. UCP only begins listening after its modules 
        
           # and default planes are configured, so a successful response here means the 
        
           # Radius API is genuinely serving. This also makes `helm --wait` and 
        
           # `rad install kubernetes` return only once the API answers, rather than as 
        
           # soon as the process starts. 
        
           readinessProbe: 
        
             httpGet: 
        
               path: /apis/api.ucp.dev/v1alpha3 
        
               port: ucp 
        
               scheme: HTTPS 
        
             initialDelaySeconds: 5 
        
             periodSeconds: 5 
        
             timeoutSeconds: 5 
        
             failureThreshold: 3

— HTTPS discovery readiness probe.

radius/deploy/Chart/templates/control-plane-readiness/job.yaml

Lines 28 to 67 in ec97901

    
           apiVersion: batch/v1 
        
           kind: Job 
        
           metadata: 
        
             name: radius-api-readiness 
        
             namespace: "{{ .Release.Namespace }}" 
        
             labels: 
        
               control-plane: radius-api-readiness 
        
               app.kubernetes.io/name: radius-api-readiness 
        
               app.kubernetes.io/part-of: radius 
        
             annotations: 
        
               "helm.sh/hook": post-install,post-upgrade 
        
               "helm.sh/hook-weight": "0" 
        
               "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded 
        
               "helm.sh/hook-output-log-policy": hook-failed 
        
           spec: 
        
             backoffLimit: 0 
        
             activeDeadlineSeconds: {{ add $timeoutSeconds 30 }} 
        
             ttlSecondsAfterFinished: {{ $ttlSecondsAfterFinished }} 
        
             template: 
        
               metadata: 
        
                 labels: 
        
                   control-plane: radius-api-readiness 
        
                   app.kubernetes.io/name: radius-api-readiness 
        
                   app.kubernetes.io/part-of: radius 
        
               spec: 
        
                 restartPolicy: Never 
        
                 serviceAccountName: radius-api-readiness 
        
                 {{- if .Values.global.imagePullSecrets }} 
        
                 imagePullSecrets: 
        
                   {{- toYaml .Values.global.imagePullSecrets | nindent 8 }} 
        
                 {{- end }} 
        
                 containers: 
        
                   - name: radius-api-readiness 
        
                     image: "{{ include "radius.image" (dict "image" $readinessImage "tag" $readinessTag "global" .Values.global) }}" 
        
                     imagePullPolicy: Always 
        
                     args: 
        
                       - wait-for-control-plane 
        
                       - "--timeout={{ $timeoutSeconds }}s" 
        
                       - "--poll-interval={{ $pollIntervalSeconds }}s" 
        
                       - "--request-timeout={{ $requestTimeoutSeconds }}s"

— mandatory post-install/post-upgrade hook.

radius/test/functional-portable/upgrade/upgrade_test.go

Lines 88 to 124 in ec97901

    
           // Test_PreflightContainer exercises the chart's pre-upgrade Helm hook. 
        
           // 
        
           // The hook only runs on helm upgrade, so installing Radius is setup cost rather than the 
        
           // subject of the test. The parent installs once with the hook enabled and uninstalls 
        
           // once; the chart's post-install hook guarantees the aggregated API is serving before 
        
           // the test client is constructed. The subtests then run in sequence against that single 
        
           // release, upgrading first with the hook enabled and then again with it disabled. They 
        
           // share a Helm release name and a Kubernetes namespace and so must not run in parallel. 
        
           func Test_PreflightContainer(t *testing.T) { 
        
           	ctx := t.Context() 
        
           	image, tag := getPreUpgradeImage() 
        
           	k8sClient, err := newKubernetesClient() 
        
           	require.NoError(t, err, "Failed to create Kubernetes client") 
        
           	cleanupAndWait(t, ctx, k8sClient) 
        
           	t.Log("Installing Radius with preflight enabled and custom configuration") 
        
           	require.NoError(t, helmInstall(ctx, image, tag, preflightEnabledValues()), "Failed to install Radius") 
        
           	t.Cleanup(func() { 
        
           		cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), cleanupCommandTimeout) 
        
           		defer cancel() 
        
           		helmUninstall(t, cleanupCtx) 
        
           	}) 
        
           	// The chart's post-install hook verifies the aggregated API through 
        
           	// kube-apiserver before Helm returns, so no test-specific readiness wait is 
        
           	// needed here. 
        
           	release := preflightRelease{options: rp.NewRPTestOptions(t), image: image, tag: tag} 
        
           	// The subtests are given the parent's context rather than their own so the shared 
        
           	// release outlives any single subtest. 
        
           	if !t.Run("Enabled", func(t *testing.T) { testPreflightEnabled(t, ctx, release) }) { 
        
           		return 
        
           	} 
        
           	t.Run("Disabled", func(t *testing.T) { testPreflightDisabled(t, ctx, release) })

— client construction follows verified Helm readiness.

Navigation Menu

Repository navigation

Flaky test: Run upgrade-noncloud functional tests: An unknown error was returned while testing Radius API status 503 #11841

Area for Improvement

Observed behavior

Desired behavior

Proposed Fix

rad Version

Operating system

Additional context

Recurring data + proposed fixes — `Test_PreflightContainer` timing/readiness flake

Recent evidence (June 2026)

Root-cause factors

Proposed fixes

Still reproducible after #12245 (August 20, 2026)

Fresh evidence

What this tells us about the merged mitigation

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	# The kube-apiserver aggregation layer forwards api.ucp.dev requests to this
	# pod through the ucp Service. Without a readiness gate the pod is added to
	# the Service endpoints as soon as the container starts, so the aggregator can
	# route to a backend that is not listening yet and return 503 to callers.
	#
	# The probe targets the Kubernetes discovery document, which is the same
	# endpoint the aggregator polls to decide APIService availability and is
	# served without authentication. UCP only begins listening after its modules
	# and default planes are configured, so a successful response here means the
	# Radius API is genuinely serving. This also makes `helm --wait` and
	# `rad install kubernetes` return only once the API answers, rather than as
	# soon as the process starts.
	readinessProbe:
	httpGet:
	path: /apis/api.ucp.dev/v1alpha3
	port: ucp
	scheme: HTTPS
	initialDelaySeconds: 5
	periodSeconds: 5
	timeoutSeconds: 5
	failureThreshold: 3

	apiVersion: batch/v1
	kind: Job
	metadata:
	name: radius-api-readiness
	namespace: "{{ .Release.Namespace }}"
	labels:
	control-plane: radius-api-readiness
	app.kubernetes.io/name: radius-api-readiness
	app.kubernetes.io/part-of: radius
	annotations:
	"helm.sh/hook": post-install,post-upgrade
	"helm.sh/hook-weight": "0"
	"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
	"helm.sh/hook-output-log-policy": hook-failed
	spec:
	backoffLimit: 0
	activeDeadlineSeconds: {{ add $timeoutSeconds 30 }}
	ttlSecondsAfterFinished: {{ $ttlSecondsAfterFinished }}
	template:
	metadata:
	labels:
	control-plane: radius-api-readiness
	app.kubernetes.io/name: radius-api-readiness
	app.kubernetes.io/part-of: radius
	spec:
	restartPolicy: Never
	serviceAccountName: radius-api-readiness
	{{- if .Values.global.imagePullSecrets }}
	imagePullSecrets:
	{{- toYaml .Values.global.imagePullSecrets \| nindent 8 }}
	{{- end }}
	containers:
	- name: radius-api-readiness
	image: "{{ include "radius.image" (dict "image" $readinessImage "tag" $readinessTag "global" .Values.global) }}"
	imagePullPolicy: Always
	args:
	- wait-for-control-plane
	- "--timeout={{ $timeoutSeconds }}s"
	- "--poll-interval={{ $pollIntervalSeconds }}s"
	- "--request-timeout={{ $requestTimeoutSeconds }}s"

	// Test_PreflightContainer exercises the chart's pre-upgrade Helm hook.
	//
	// The hook only runs on helm upgrade, so installing Radius is setup cost rather than the
	// subject of the test. The parent installs once with the hook enabled and uninstalls
	// once; the chart's post-install hook guarantees the aggregated API is serving before
	// the test client is constructed. The subtests then run in sequence against that single
	// release, upgrading first with the hook enabled and then again with it disabled. They
	// share a Helm release name and a Kubernetes namespace and so must not run in parallel.
	func Test_PreflightContainer(t *testing.T) {
	ctx := t.Context()
	image, tag := getPreUpgradeImage()

	k8sClient, err := newKubernetesClient()
	require.NoError(t, err, "Failed to create Kubernetes client")

	cleanupAndWait(t, ctx, k8sClient)

	t.Log("Installing Radius with preflight enabled and custom configuration")
	require.NoError(t, helmInstall(ctx, image, tag, preflightEnabledValues()), "Failed to install Radius")

	t.Cleanup(func() {
	cleanupCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), cleanupCommandTimeout)
	defer cancel()
	helmUninstall(t, cleanupCtx)
	})

	// The chart's post-install hook verifies the aggregated API through
	// kube-apiserver before Helm returns, so no test-specific readiness wait is
	// needed here.
	release := preflightRelease{options: rp.NewRPTestOptions(t), image: image, tag: tag}

	// The subtests are given the parent's context rather than their own so the shared
	// release outlives any single subtest.
	if !t.Run("Enabled", func(t *testing.T) { testPreflightEnabled(t, ctx, release) }) {
	return
	}
	t.Run("Disabled", func(t *testing.T) { testPreflightDisabled(t, ctx, release) })

Flaky test: Run upgrade-noncloud functional tests: An unknown error was returned while testing Radius API status 503 #11841

Description

Area for Improvement

Observed behavior

Desired behavior

Proposed Fix

rad Version

Operating system

Additional context

Activity

radius-triage commented on May 8, 2026

willdavsmith commented on May 29, 2026

radius-triage commented on Jun 15, 2026

sylvainsf commented on Jun 25, 2026

Recurring data + proposed fixes — Test_PreflightContainer timing/readiness flake

Recent evidence (June 2026)

Root-cause factors

Proposed fixes

nellshamrell commented on Aug 20, 2026

Still reproducible after #12245 (August 20, 2026)

Fresh evidence

What this tells us about the merged mitigation

12 remaining items

nicolejms commented on Sep 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Recurring data + proposed fixes — `Test_PreflightContainer` timing/readiness flake