Skip to content

Rotate edge addresses after control stream failures - #1785

Open
akasakariko wants to merge 1 commit into
cloudflare:masterfrom
akasakariko:fix/1771-control-stream-address-rotation
Open

akasakariko wants to merge 1 commit into
cloudflare:masterfrom
akasakariko:fix/1771-control-stream-address-rotation

Conversation

@akasakariko

Copy link
Copy Markdown

Summary

  • Rotate to another edge address after a control stream failure instead of repeatedly reconnecting to the same edge
  • Preserve permanent registration failures, duplicate connection handling, and the existing dial failure budget for protocol fallback
  • Add regression coverage for repeated failures after registration, wrapped errors, retryable and permanent registration failures, and per-connection dial budgets

Why

The address fallback policy does not recognize ordinary control stream failures, so retries keep the address already assigned to that connection
Successful registration also resets the retry backoff, allowing repeated control stream failures to keep a connector pinned indefinitely
Rotate on the control stream failure itself rather than introducing another counter that successful registration could reset

Reproduction

  • Confirmed on upstream revision 77a79bf before changing production code
  • Use a deterministic fault-injection test with real QUIC serving code, a mock transport, and a control handler that reports registration success before failing
  • Exercise the same address policy and pool operations used by the edge tunnel server against a two-address pool
  • Before the fix, all 12 attempts reuse one address and the regression fails
  • After the fix, all 12 attempts rotate between the two addresses and the same regression passes
  • This reproduces the retry and address-selection defect, not the original production edge fault or the health-metric symptoms

Run the focused regression from the supervisor directory

cd supervisor
go test -mod=readonly -run '^TestControlStreamFailureRotatesEdgeAddress$' -count=1 -v

Verification

  • Full make test lint passes on macOS ARM64 with the repository pinned Go toolchain and race detection enabled
  • All 49 test packages pass and lint reports zero issues
  • Targeted supervisor, connection, and edge discovery race tests pass
  • Address-selection regressions pass across 50 repeated runs
  • Final regression fails with unchanged upstream source overlaid and passes with the fix

Scope

This change addresses automatic edge recovery only, HA connection metrics and readiness semantics remain unchanged

Fixes #1771

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

🐛 cloudflared retries a failing edge address indefinitely instead of rotating; connector reports 4/4 HA connections while serving no traffic

1 participant