Recurring long-running test failure: Test_RadiusCore_AzureMySql_PortalLink fails on delete (not fixed by #12653) #12726

Summary

The scheduled long-running (Azure) functional test Test_RadiusCore_AzureMySql_PortalLink fails on rad application delete. This was surfaced by the auto-filed issue #12648 (run 31587251857).

PR #12653 (merged 2026-08-12) was a candidate fix for this delete failure — it fixed a nil-pointer panic when unfolding recipe delete errors and added observability so a backend async panic surfaces on the operation status. That fix merged but did not resolve the failure: the test continues to fail on the delete path in every long-running run since.

Both failure signatures below are the same test failing on the same delete path (rad application delete → the Radius.Data/mySqlDatabases recipe delete async op ends Failed / code: "Internal"). Tracking them together here.

Error 1 — exceeded max retry count (run 31587251857, from #12648)

The environment deletes fine, then the app delete reports a failed async operation:

[rad] Radius.Core/environments/azure-mysql-portallink-env-longrc9d8d deleted
[rad] Failed to delete resources for application 'azure-mysql-portallink-app':
      GET .../Radius.Data/locations/global/operationStatuses/d1063951-b058-4bbb-a2c1-974388b2a85c
[rad] ERROR CODE: Internal
      "status": "Failed",
        "code": "Internal",
        "message": "exceeded max retry count to process async operation message: 3"

The delete async operation is silently requeued until DequeueCount exceeds MaxOperationRetryCount, then ends Failed with the generic "exceeded max retry count to process async operation message: 3". This is exactly the symptom PR #12653 targeted (an underlying nil-pointer panic in the recipe delete path).

Error 2 — 404 NotFound on environment during recipe delete (run 32241208014, from #12713)

After #12653 the panic is gone, but the same delete still fails — the async op ends Failed / code: "Internal", now with the message reporting a 404 Not Found when the recipe delete path GETs the (already-deleted) environment:

[rad] Failed to delete resources for application 'azure-mysql-portallink-app':
      GET .../Radius.Data/locations/global/operationStatuses/bee3fe5f-...
      "status": "Failed",
        "code": "Internal",
        "message": "GET .../Radius.Core/environments/azure-mysql-portallink-env-longr8a6f0
                    RESPONSE 404: 404 Not Found
                    ERROR CODE: NotFound
                    { \"error\": { \"code\": \"NotFound\",
                      \"message\": \"the resource with id '.../Radius.Core/environments/azure-mysql-portallink-env-longr8a6f0' was not found\" } }"

The environment is already gone at that point, so the recipe delete's environment GET returns 404, which surfaces as code: "Internal" instead of being treated as already-deleted.

Evidence: still failing after #12653

Auto-filed long-running failures after #12653 merged (Aug 12), all failing on the same test's delete:

Suggested mitigation: skip the test until root cause

Since this fails on every scheduled long-running run and blocks the pipeline, temporarily skip Test_RadiusCore_AzureMySql_PortalLink with a t.Skip that links back to this issue, following the existing repo precedent (e.g. Test_ACI in test/functional-portable/corerp/cloud/resources/aci_test.go and the t.Skip("https://github.com/radius-project/radius/issues/...") pattern in recipe_bicep_test.go).

File: test/functional-portable/corerp/cloud/resources/radiuscore_azure_mysql_portallink_test.go

func Test_RadiusCore_AzureMySql_PortalLink(t *testing.T) {
	t.Skip("Temporarily disabled: rad application delete fails intermittently on the Radius.Data/mySqlDatabases recipe delete path. Not fixed by #12653. See #<this-issue>.")
	// ...
}

Remove the skip once the delete-path root cause is fixed.

Root cause investigation TODO

• Error 1: root-cause why the Radius.Data/mySqlDatabases recipe delete async operation fails and exhausts its retries (exceeded max retry count to process async operation message: 3).
• Error 2: why the recipe delete path GETs Radius.Core/environments/... after it is already deleted, and why the resulting 404 surfaces as code: "Internal" instead of being treated as already-deleted.
• Confirm delete ordering between the application/environment and the Radius.Data/mySqlDatabases recipe-backed resource.
• Re-enable the test (remove t.Skip) once fixed.

Links

• Original auto-filed issue: Scheduled long running test did not complete successfully - Run ID: 31587251857 #12648
• Candidate fix that merged but did not resolve this: Fix nil-pointer panic when unfolding recipe delete errors #12653
• Error 1 run: https://github.com/radius-project/radius/actions/runs/31587251857
• Error 2 run: https://github.com/radius-project/radius/actions/runs/32241208014