Skip to content

logging: classify bare rate-limit messages as transient - #710

Open
ben-dz wants to merge 1 commit into
mainfrom
bdz/dberror-connection-rate-limit
Open

logging: classify bare rate-limit messages as transient#710
ben-dz wants to merge 1 commit into
mainfrom
bdz/dberror-connection-rate-limit

Conversation

@ben-dz

@ben-dz ben-dz commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary of Changes

  • Broaden dberror's rate-limit pattern from "rate limited" to "rate limit", so messages like rpcpool's HTTP 429 body — Connection rate limits exceeded, surfaced through the jsonrpc client's error dump where neither "rate limited" nor "status 429" substring-match — classify as ErrorTypeRateLimit/transient. The broader pattern also covers "rate limiting".
  • Why it matters: transient classification selects the lenient escalation threshold (10 consecutive failures) instead of the strict one (3), so a throttled-but-progressing loop logs WARN rather than paging on-call. Observed during the 2026-07-23 testnet permission-events drain (indexer: drain permission-events accounts in durable chunks to break testnet poison loop #706): a mid-drain 429 from the DZ ledger RPC classified as Unknown and would have escalated as if actionable.

Diff Breakdown

Category Files Lines (+/-) Net
Core logic 1 +3 / -1 +2
Tests 1 +6 / -0 +6

One pattern broadened, pinned by table-test cases.

Testing Verification

  • New table cases (verified failing first) assert ClassifyErrorTypeRateLimit and IsTransient → true for the exact prod error text (jsonrpc dump format with Connection rate limits exceeded) and a "rate limiting" variant; existing cases confirm "rate limited" and "status 429" still classify unchanged.
  • utils/... suite passes with -race.

The DZ ledger RPC (rpcpool) surfaces HTTP 429 as a jsonrpc error dump
whose message is "Connection rate limits exceeded" - neither "rate
limited" nor "status 429" substring-matches it, so it classified as
Unknown/non-transient and escalated at the strict threshold (3) instead
of the transient one (10). Observed during the 2026-07-23 testnet
permission-events drain, where a mid-drain 429 logged as if actionable.

Broaden the pattern to "rate limit", covering "rate limited",
"rate limits exceeded", and "rate limiting": an error mentioning
rate limits is a throttle.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant