Skip to content

logging: classify bare rate-limit messages as transient #710

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ben-dz wants to merge 1 commit into
base: main
Choose a base branch
from
Open

logging: classify bare rate-limit messages as transient #710

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion utils/pkg/dberror/dberror.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,9 @@ func Classify(err error) ErrorType {
// 429 or a gRPC service returning ResourceExhausted, like InfluxDB).
// These are transient and self-healing — worth retrying, not worth paging on.
rateLimitPatterns := []string{
"rate limited",
// Broad on purpose: covers "rate limited", "rate limits exceeded" (rpcpool's
// 429 body), and "rate limiting" — an error mentioning rate limits is a throttle.
"rate limit",
"too many requests",
"status 429",
"request too large",
Expand Down
6 changes: 6 additions & 0 deletions utils/pkg/dberror/dberror_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,12 @@ func TestClassifyAndIsTransient(t *testing.T) {
// "request too large" prefix — both must classify as transient rate limits.
{"influxdb resources exhausted plural", errors.New("Resources exhausted: Heap exhausted"), dberror.ErrorTypeRateLimit, true},
{"influxdb request too large", errors.New("request too large: Error running series plans for namespace 'x': Resources exhausted: Heap exhausted"), dberror.ErrorTypeRateLimit, true},
// rpcpool 429 as surfaced by the jsonrpc client's dump formatting: neither
// "rate limited" nor "status 429" substring-matches it, but it is plainly a
// transient throttle (observed from the DZ ledger RPC during the 2026-07-23
// permission-events drain).
{"rpcpool connection rate limit", errors.New("get transaction: (*jsonrpc.RPCError)(0xc003100330)({\n Code: (int) 429,\n Message: (string) (len=31) \"Connection rate limits exceeded\",\n Data: (interface {}) <nil>\n})"), dberror.ErrorTypeRateLimit, true},
{"rate limiting variant", errors.New("upstream is rate limiting requests"), dberror.ErrorTypeRateLimit, true},

// The actual prod CH-connection-drop errors we saw — must be transient.
{"ch read packet reset", errors.New("query processing: failed to read packet from 52.4.220.199:9440 (conn_id=492): read: connection reset by peer"), dberror.ErrorTypeConnectivity, true},
Expand Down
Loading