Skip to content

fix(http): release global rate limit lock when a request is cancelled - #1585

Open
zachnieto wants to merge 2 commits into
DisnakeDev:masterfrom
zachnieto:fix/global-ratelimit-deadlock
Open

fix(http): release global rate limit lock when a request is cancelled#1585
zachnieto wants to merge 2 commits into
DisnakeDev:masterfrom
zachnieto:fix/global-ratelimit-deadlock

Conversation

@zachnieto

@zachnieto zachnieto commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Summary

A request that is cancelled while sleeping off a global rate limit permanently deadlocks the entire HTTP client.

In HTTPClient.request:

if is_global:
    _log.warning("Global rate limit has been hit. Retrying in %.2f seconds.", retry_after)
    self._global_over.clear()

await asyncio.sleep(retry_after)
_log.debug("Done sleeping for the rate limit. Retrying...")

# release the global lock now that the
# global rate limit has passed
if is_global:
    self._global_over.set()
    _log.debug("Global rate limit is now over.")

The set() is not protected. If the task is cancelled during asyncio.sleep(retry_after), CancelledError propagates out and the event is never re-set.

From then on, every request in the process blocks forever near the top of request:

if not self._global_over.is_set():
    # wait until the global lock is complete
    await self._global_over.wait()

There is no recovery path. The client has to be restarted.

Why this is easy to hit

The gateway is a separate websocket and is unaffected, so the bot stays connected, keeps receiving events, and keeps passing any health check that does not exercise the HTTP API. It simply stops being able to send anything. That makes it look like an intermittent "some commands stopped responding" problem rather than a deadlock.

Cancellation inside that window happens in ordinary usage:

  • a send wrapped in asyncio.wait_for that times out
  • tasks cancelled during shutdown
  • any user code calling Task.cancel()
  • library-level cancellation of pending work

It needs a global 429 to coincide with a cancellation, so it is rare, but it is unrecoverable when it does happen. We hit it in production on a bot across a few thousand guilds: online, healthy, gateway fine, and not a single outbound request completing until a restart.

Fix

Move the release into a finally, so the global lock is dropped whether the sleep finishes or the request is cancelled. Behaviour is otherwise unchanged: on the normal path the event is still set right after the sleep, before the retry.

Tests

Added test_global_rate_limit_released_on_cancellation to tests/test_http.py. It drives a real HTTPClient.request against a stub session that returns a global 429, cancels the request while it is sleeping, and asserts _global_over ends up set.

Verified it fails on master and passes with this change:

# without the fix
>       assert http._global_over.is_set()
E       assert False
1 failed

# with the fix
1 passed

Notes

discord.py has the same issue in the equivalent code path, in case it is worth flagging upstream.

Changelog fragment added as changelog/1585.bugfix.rst.

Checklist

  • If code changes were made, then they have been tested
    • I have updated the documentation to reflect the changes
    • I have formatted the code properly by running uv run nox -s lint
    • I have type-checked the code by running uv run nox -s pyright
  • This PR fixes an issue
  • This PR adds something new (e.g. new method or parameters)
  • This PR is a breaking change (e.g. methods or parameters removed/renamed)
  • This PR is not a code change (e.g. documentation, README, ...)

zachnieto and others added 2 commits July 29, 2026 10:06
On a global 429 the request clears `_global_over`, sleeps for
`retry_after`, then sets it again. The set was not protected, so a
request cancelled while sleeping never re-set the event.

Once that happens every subsequent request blocks forever on
`await self._global_over.wait()` near the top of `request`, and the
client cannot recover without a restart. The gateway is unaffected, so
the bot stays connected and looks healthy while no HTTP request ever
completes again.

Cancellation during that window is easy to hit in normal use, for
example a send wrapped in `asyncio.wait_for`, a task cancelled on
shutdown, or any user code calling `Task.cancel()`.

Move the set into a `finally` so the lock is always released.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@read-the-docs-community

Copy link
Copy Markdown

Documentation build overview

📚 disnake | 🛠️ Build #33817715 | 📁 Comparing 06e98ba against latest (c85e0b4)

  🔍 Preview build  

48 files changed · ± 48 modified

± Modified

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant