rta: keep reconnecting until the Conn is closed - #42
Open
HashimTheArab wants to merge 3 commits into
Open
Conversation
A dropped RTA WebSocket got four redials about twenty seconds apart and then the Conn closed itself with "max reconnect attempt reached" as the cause. A service outage lasts longer than that, so every long-lived Conn ended the outage dead: each later Subscribe returned the cached cause and nothing short of building a new Conn recovered. The same budget applied to a socket that kept dropping mid-handshake. Reconnect now retries with capped exponential backoff (1s doubling to 60s, 50% jitter) until the dial lands or the Conn is closed, and an interrupted resubscribe round backs off and goes again instead of counting toward a limit. Subscriptions stay owned by the reconnect the whole time; if the Conn closes underneath it they get the close cause. Tests drive an outage through a server that refuses upgrades and a handshake the server keeps cutting, and cover Close during an outage. (cherry picked from commit 7ab4616)
…d Close If Close ran while a resubscribe handshake was in flight, its deactivation loop saw an empty map: the reconnect had taken the subscriptions, and the successful handshake then tracked an active subscription back onto the closed Conn with no HandleError. The reconnect now finishes that deactivation when it sees the Conn closed. Also: only the first failed redial logs at Error, later ones at Warn, so a long outage is not an Error stream; and tests shorten the backoff through a package variable set before Dial instead of writing to a live dialer. (cherry picked from commit 6ebe74f)
A dial that completed just as Close swept c.conn could be stored after the sweep, leaving an open socket behind a closed Conn. The reconnect now publishes under connMu with a ctx check, so either Close sees the socket or the reconnect sees the cancelled ctx and closes it itself.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
When the RTA WebSocket drops,
rta.Connredials four times with 1/2/4/8-second backoff and then closes itself for good withmax reconnect attempt (4) reachedas the cause. That is about twenty seconds of patience. An Xbox Live outage lasts far longer, so every long-lived connection comes out of one dead: each laterSubscribereturns the cached cause, and only building a newConnrecovers. A socket that keeps dropping mid-handshake hits the same wall after four resubscribe rounds. We hit exactly this during the Xbox Live incident on 2026-09-02: session broadcasters that had been running for days looped on the cached error every two minutes until they were restarted.What changed
dialer.reconnectretries until the dial lands or the context is done, with capped exponential backoff: 1s doubling to a 60s cap, plus up to 50% jitter. The first failure logs at Error, later ones at Warn, so a long outage is not an Error stream.runReconnectno longer closes theConnon dial failure or after repeated interrupted resubscribes. An interrupted round backs off on the same schedule and tries again. The subscriptions stay owned by the reconnect throughout; if theConnis closed underneath it, they are deactivated with the close cause so nothing is left looking active on a dead connection. That includes a handshake that lands afterClosehas already run its own deactivation loop.Unchanged: a reconnect with no live subscriptions still closes the socket normally,
Closestill ends everything, andSubscribecallers still bound their own wait with their context.Tests
go test -race ./rta/andgo test ./...pass. New tests drive a server that refuses upgrades for ten attempts before accepting, a server that cuts the handshake eight times before letting it through,Closeduring an outage, andClosewhile a resubscribe handshake is mid-flight. The old "closes after persistent interrupted resubscribe" test is replaced by the outlasting version.🤖 Generated with Claude Code
https://claude.ai/code/session_01GMfvFUjXaF2P5m2gUmAJaT