feat: restored retrying of rate-limited Leantime requests - #327
Conversation
fetchFromLeantime() calls getContent() with default error handling, so a 429 throws out of updateAsJob() before the next page is queued — one rate-limited response ended the whole pagination chain. With five staggered nightly syncs and a 15-minute incremental cron against the same API, that is the likeliest halt in production. RetryableHttpClient with a 429 strategy was added for exactly this in b64773d and deleted in b27ba16 with the old Jira stack, leaving docker-compose.server.override.yml still commenting that the sync is rate limited by the Leantime API. This restores it as app.leantime.http_client, injected only into LeantimeApiService so the monitoring ping keeps the plain client, and tunable via APP_HTTP_CLIENT_RETRY_DELAY_MS and APP_HTTP_CLIENT_MAX_RETRIES. The retried status codes are a flat list rather than GenericRetryStrategy's defaults: those express transport errors and 500/504/507/510 as [code => idempotent methods], which excludes POST, and the Leantime data API uses POST even for reads. Also sets timeout and max_duration on the default HTTP client. Symfony caps neither, so a Leantime instance that accepted a connection and then stalled held the worker indefinitely — messenger's --time-limit is only checked between messages. max_duration is 300s to clear the unpaginated /deleted endpoint used by sync-deleted --interval=P1W.
turegjorup
left a comment
turegjorup
left a comment
There was a problem hiding this comment.
I think this is the wrong direction for retries. I think the message handlers should differentiate between exceptions and only throw UnrecoverableMessageHandlingException when it actually makes sense. so NOT for HTTP errors. Then configure retry on the message queue.
Retry on the http client makes sense when running synchronous. When running async in job queues I would let retry on the http client be disabled (which is default: https://symfony.com/doc/current/reference/configuration/framework.html#enabled-9). And let the message handling control retry.
| # has to clear the /deleted endpoint, which the Leantime plugin does not paginate, so | ||
| # `sync-deleted --interval=P1W` pulls a week of history in one response. | ||
| timeout: 30 | ||
| max_duration: 300 |
There was a problem hiding this comment.
30 is high for timeout. I would go for 5
300 for max_duration also high. If the deletedendpoint performs this badly it should be paginated. As an alternative configure two http clients. One with normal, thight defaults, one with longer timeouts for the deleted endpoints.
Make scoped clients, so you can configure them individually, don't alter and depend on defaults.
Use the rate limiter: https://symfony.com/doc/current/http_client.html#limit-the-number-of-requests and configure it to match Leantime, maybe with a little headroom
| $client: "@http_client" | ||
| $strategy: '@Symfony\Component\HttpClient\Retry\RetryStrategyInterface' | ||
| $maxRetries: "%env(int:APP_HTTP_CLIENT_MAX_RETRIES)%" | ||
| $logger: "@logger" |
There was a problem hiding this comment.
Set shorter timeout here, use the rate_limiter option: https://symfony.com/doc/current/http_client.html#limit-the-number-of-requests
| $statusCodes: [0, 423, 425, 429, 500, 502, 503, 504, 507, 510] | ||
| $delayMs: "%env(int:APP_HTTP_CLIENT_RETRY_DELAY_MS)%" | ||
| $multiplier: 2.0 | ||
| # RetryableHttpClient prefers the response's Retry-After header over this backoff, so the |
There was a problem hiding this comment.
Have you validated that Leantime (or our leantime plugin) even sends a Retry-After header?
2 participants
Link to ticket
https://leantime.itkdev.dk/?tab=ticketdetails#/tickets/showTicket/8000
Description
RetryableHttpClientwith a 429 retry strategy was added inb64773db("1595: Added retryable http client to handle rate limiting") and lost inb27ba16ewhen the Jira stack was removed, leavingdocker-compose.server.override.ymlstill commenting that the sync is rate limited by the Leantime API. A 429 surfaces insideupdateAsJob()before the next page is queued, so a single one ended the whole pagination chain.LeantimeApiServicenow gets a retrying client viaapp.leantime.http_client, tunable withAPP_HTTP_CLIENT_RETRY_DELAY_MSandAPP_HTTP_CLIENT_MAX_RETRIES. The retried status codes are a flat list, becauseGenericRetryStrategy's defaults restrict transport errors and 5xx to idempotent methods — which excludes the POSTs the Leantime data API uses even for reads.timeout: 30andmax_duration: 300toframework.http_client.default_options. Symfony caps neither by default, so a Leantime instance that accepted a connection and then stalled held the messenger worker indefinitely —--time-limitis only checked between messages, never during one.max_durationhas to clear the unpaginated/deletedendpoint, which returns a week of history in one response forsync-deleted --interval=P1W.Checklist