Skip to content

run_live does not retry 1008 "session is already connected to an existing client" when resuming with a session handle #8258

Description

@ThamaluM

Describe the bug

When a Live connection drops and ADK reconnects using the session resumption
handle (Vertex AI, SessionResumptionConfig(transparent=True)), the server
sometimes rejects the new connection with close code 1008:

google.genai.errors.APIError: 1008 None. BidiGenerateContent session is already connected to an existing client

ADK reconnects immediately, before Vertex has released the previous socket.
_live_llm_flow.run_live_flow treats only 1000, 1006, 1011 as recoverable, so
the 1008 is re-raised and the whole run_live stream fails. In a phone-call
agent, that ends the call.

The condition is transient: the server releases the old session within about a
second. Other Gemini Live clients report the same error and fix it with backoff
plus a retry. adk-go has a matching change for 1008 in isResumable
(google/adk-go#1993).

To reproduce

  1. Vertex AI backend (GOOGLE_GENAI_USE_VERTEXAI=1), a native-audio Live model,
    RunConfig(session_resumption=types.SessionResumptionConfig(transparent=True), context_window_compression=...).
  2. Run runner.run_live(...) long enough for the connection to drop or rotate
    (GoAway / network drop) while a resumption handle is held.
  3. Intermittently, ADK's reconnect attempt fails at llm.connect() with 1008 and
    run_live raises instead of retrying.

Traceback (trimmed)

File ".../google/adk/live/_live_llm_flow.py", line 595, in run_live_flow
  async with llm.connect(llm_request) as llm_connection:
File ".../google/adk/models/google_llm.py", line 665, in connect
  async with self._live_api_client.aio.live.connect(
File ".../google/genai/live.py", line 1198, in connect
  raw_response = await ws.recv(decode=False)
websockets.exceptions.ConnectionClosedError: received 1008 (policy violation)
  BidiGenerateContent session is already connected to an existing client
...
google.genai.errors.APIError: 1008 None. BidiGenerateContent session is already
  connected to an existing client

The error is raised inside live.connect() while it waits for the setup
response, so this is the reconnect attempt. A first connection has no handle
and cannot collide.

Where

src/google/adk/live/_live_llm_flow.py (2.14.1; same logic on main):

except errors.APIError as e:
  ...
  # Error code 1000, 1006 and 1011 indicates a recoverable connection drop.
  if e.code in [1000, 1006, 1011]:
    if invocation_context.live_session_resumption_handle and (not client_closed):
      ...
      continue

There is also no delay between reconnect attempts (while True → continue).

Expected behavior

When a resumption handle is held, a 1008 "already connected" on reconnect is
retried with a short backoff (e.g. 0.5 s, 1 s, 2 s), within
DEFAULT_MAX_RECONNECT_ATTEMPTS, instead of failing the stream.

Suggested fix

  • Add 1008 to the recoverable codes only for the reconnect path (a handle is
    set), ideally matched on the "already connected" reason so other policy
    violations still fail fast.
  • Add exponential backoff before each reconnect attempt (asyncio.sleep), for all
    recoverable codes. Reconnecting instantly is what loses the race.
  • Make sure the previous connection is fully closed before opening the new one.

Related (smaller)

  • A connect-time TimeoutError("timed out during opening handshake") or OSError
    on a reconnect with a handle also escapes as "An unexpected error occurred in
    live flow". With a handle held, it could get the same bounded retry.
  • main already keeps the last non-empty resumption handle when an update arrives
    with an empty new_handle. In 2.14.x, an empty update overwrites the handle, so
    a drop during generation or a tool call cannot be resumed. Could that fix ship in
    the next release?

Environment

  • google-adk 2.14.0 (live flow identical in 2.14.1)
  • google-genai 2.34.0
  • websockets 15.0.1
  • Python 3.11, Linux (Docker), uvloop
  • Backend: Vertex AI Live API, transparent=True session resumption

Workaround

We catch APIError 1008, and connect-time TimeoutError/OSError raised from
the websocket, around runner.run_live ourselves. We then open a fresh
run_live on the same ADK session after a short delay, which reseeds the
history. That works, but it loses server-side resumption and duplicates logic
ADK already has.

Activity

  1. MaiAditya commented on Nov 19, 2026

    @MaiAditya

    Thanks for the detailed report and the adk-go pointer. If you're not planning to send the PR yourself, I'd like to take this, following your suggested fix:

    • In run_live_flow, treat a 1008 whose message contains "already connected to an existing client" as recoverable only on the reconnect path (a resumption handle is held and the client hasn't closed), so other 1008 policy violations still fail fast.
    • Add a bounded exponential backoff (e.g. 0.5 s, 1 s, 2 s, ...) before each reconnect attempt for all recoverable codes, still capped by DEFAULT_MAX_RECONNECT_ATTEMPTS.
    • Unit tests in the existing live-flow tests: a 1008 "already connected" on reconnect is retried and then succeeds, a 1008 with a different reason is re-raised, and the backoff delays are applied.

    I'd leave the connect-time TimeoutError/OSError case from "Related" for a follow-up unless you'd prefer it in the same change.

  2. ThamaluM commented on Nov 19, 2026

    @ThamaluM
    Author

    Thanks, please go ahead. Two requests, since this is used for phone calls where any added delay is dead air:

    1. No delay on the first reconnect attempt. Keep the first resume after a drop/GoAway immediate, and back off only on retries after a failed attempt (fail → 0.5 s → 1 s → 2 s). Most resumes succeed first time.
    2. Reset the backoff after a successful connect, like attempt is reset today, so each rotation on a long call starts from zero delay.

    Small note on matching: the reason is only in the message (APIError: 1008 None. BidiGenerateContent session is already connected to an existing client), so match on str(e) / e.message.

    The TimeoutError/OSError follow-up is fine.

  3. surajksharma07 commented on Nov 20, 2026

    @surajksharma07
    Collaborator

    @ThamaluM Reproduced it with a mocked connection on 2.15.0 and main: 1008 still isn’t treated as recoverable, and there’s no delay between reconnects. The empty-handle fix you mentioned did make it into 2.15.0.

    Until #8260 lands, a workaround is to wrap Gemini.connect() so it retries only the “already connected to an existing client” 1008 when a handle is set. Skip the delay on the first try, then back off 0.5s/1s/2s. Because the retry happens inside connect, the server-side resumption should be kept instead of replaying history.

    One thing before anyone tries it: the snippet link in the draft, <link/gist to fix_8258.py>, is still a placeholder. Please paste the real gist URL or inline the code, otherwise there’s nothing for ThamaluM to run.

    Could you test it on a few real calls with GoAway/drops and let us know if it holds up? @MaiAditya, if it works for them, please check #8260 against the same cases (zero first delay, backoff reset after a successful connect) before it goes for review.

  4. MaiAditya commented on Nov 20, 2026

    @MaiAditya

    Thanks for reproducing it. I checked #8260 against both cases:

    • No delay on the first reconnect: _reconnect_delay(1) returns 0, so the first resume after a drop or GoAway is immediate. Retries after a failed attempt wait 0.5 s, 1 s, 2 s, then 4 s (the cap). test_reconnect_delay_is_immediate_then_doubles_up_to_the_cap pins this.
    • Backoff reset after a successful connect: the attempt count, and with it the backoff, resets once the new connection delivers its first event, so each rotation on a long call starts again from zero delay. I reset on the first event rather than on the connect itself so that a connection that is accepted and then dropped straight away (as the 1008 loop does) keeps backing off instead of retrying immediately forever. I've pushed test_run_live_backoff_restarts_after_a_connection_delivers_an_event to cover this.

    The CLA check is the remaining blocker on my side; I'm sorting that out now.

  5. surajksharma07 commented on Nov 20, 2026

    @surajksharma07
    Collaborator

    Thanks @MaiAditya, went through 6d6c11f and traced the loop. The first resume is immediate. Failed retries wait 0.5/1/2/4 s and it raises after the 5th. The old connection is already exited before the wait. Resumption updates come through as events, so the counter resets on every GoAway rotation. A 1008 after a GoAway still lands in the APIError branch, so it gets the backoff even though the RESUME continue skips the max-attempts check (fine as the follow-up you noted). CLA is green now, so it’s just waiting on workflow approval and review.

    @ThamaluM sorry, the link in my last comment was a placeholder. Here’s the snippet. Use model=ResilientGemini(model="") on the agent:

    import asyncio, contextlib
    from google.adk.models.google_llm import Gemini
    from google.genai import errors
    
    class ResilientGemini(Gemini):
      @contextlib.asynccontextmanager
      async def connect(self, llm_request):
        cfg = llm_request.live_connect_config
        resuming = bool(cfg and cfg.session_resumption and cfg.session_resumption.handle)
        async with contextlib.AsyncExitStack() as stack:
          for delay in (0.5, 1.0, 2.0, None):
            try:
              conn = await stack.enter_async_context(super().connect(llm_request))
              break
            except errors.APIError as e:
              if not (resuming and delay and e.code == 1008
                      and "already connected to an existing client" in str(e)):
                raise
              await asyncio.sleep(delay)
          yield conn
    

    It only retries around the connect/setup step, so a 1008 during receive still goes through ADK’s normal handling. Let us know how it holds up on real calls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions