From 72860b3c2937ca56440dd4c918b7a08a9742799b Mon Sep 17 00:00:00 2001 From: Troels Ugilt Jensen <6103205+tuj@users.noreply.github.com> Date: Tue, 18 Aug 2026 13:07:24 +0000 Subject: [PATCH 1/3] docs: describe and diagram the Leantime sync Adds docs/leantime-sync.md with a flowchart, a sequence diagram of one paged run, the scheduled jobs and the command options, plus a hand-drawn SVG of the same pipeline. Rewrites the README Synchronization section, which pointed at a QueueSyncCommand and an app:queue-sync command that no longer exist, claimed Symfony Scheduler queues the jobs, and named the wrong data provider interface. --- CHANGELOG.md | 3 + README.md | 38 +++++--- docs/images/leantime-sync.svg | 163 ++++++++++++++++++++++++++++++++++ docs/leantime-sync.md | 161 +++++++++++++++++++++++++++++++++ 4 files changed, 352 insertions(+), 13 deletions(-) create mode 100644 docs/images/leantime-sync.svg create mode 100644 docs/leantime-sync.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 5c6b8547f..e8a5316b9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +* Documented how Economics synchronizes from the Leantime data-api plugin in + `docs/leantime-sync.md`, with a diagram in `docs/images/leantime-sync.svg`, + and rewrote the outdated `Synchronization` section in `README.md`. * [PR-324](https://github.com/itk-dev/economics/pull/324) Added game center with snake * [PR-303](https://github.com/itk-dev/economics/pull/303) diff --git a/README.md b/README.md index bfeab0032..f66e35f3c 100644 --- a/README.md +++ b/README.md @@ -52,38 +52,50 @@ accounts. ## Synchronization -Economics depends on data fra external systems. The integrations with external systems are called Data Providers. +Economics depends on data from external systems. The integrations with external systems are called Data Providers. -Each Data Provider integration should implement `App\Interface\DataProviderServiceInterface`. +Each Data Provider integration should implement `App\Interface\DataProviderInterface`. The only +implementation today is `App\Service\LeantimeApiService`, which pulls from the +[data-api plugin](https://github.com/itk-dev/data-api) running in Leantime. -The data synchronization is handled by symfony messenger. This is handled differently in development and production. +Synchronization is a paged pull driven by Symfony Messenger: a command dispatches one message per +entity type, each message fetches a page of at most 100 rows and re-dispatches itself for the next +page, and every fetched row becomes an upsert message of its own. -### Production +See [docs/leantime-sync.md](docs/leantime-sync.md) for the full picture, the scheduled jobs and the +command options. -Supervisor is added to `docker-compose.server.override.yaml` to make sure the job queue is running. +### Production -Symfony scheduler is used for creating a new job each hour at minute 5. See `App\Command\QueueSyncCommand`. +Cron jobs on the server run the sync commands — see the `cron` section of +`.woodpecker/prod_itk_economics.yml`. Supervisor is added in `docker-compose.server.override.yml` to +keep a worker consuming the `async` transport. ### Develop In development the job queue should be run manually. -```sh -docker compose exec phpfpm bin/console messenger:consume async -vv --failure-limit 1 +```shell +task messenger ``` ### Queuing jobs -Jobs can be queued manually with App\Command\QueueSyncCommand +Jobs can be queued manually. -```sh -docker compose exec phpfpm bin/console app:queue-sync +```shell +# Everything modified within the last hour. +task phpfpm -- bin/console app:data-providers:sync-modified + +# Entities deleted within the last hour. +task phpfpm -- bin/console app:data-providers:sync-deleted + +# A full sync of a single entity type, as async jobs, ignoring modified timestamps. +task phpfpm -- bin/console app:data-providers:sync -j -p -d ``` Jobs can also be queued in the admin interface in the bottom left corner. -In production jobs are queued automatically each hour. - ## Development Getting started: diff --git a/docs/images/leantime-sync.svg b/docs/images/leantime-sync.svg new file mode 100644 index 000000000..73112b388 --- /dev/null +++ b/docs/images/leantime-sync.svg @@ -0,0 +1,163 @@ + + + + + + + + + + + + + How Economics syncs from Leantime + Cron-driven, paged pull through Symfony Messenger. Leantime never pushes — Economics asks, one page of 100 rows at a time. + + + + CRON · PRODUCTION HOST + + + ECONOMICS + + + LEANTIME + DATA-API PLUGIN + + + + */15 · sync-modified + changed within PT1H, async + + + */25 + 02:50 · sync-deleted + deletions, handled inline + + + 02:00–02:40 · full sync + one entity type per job, + -j (async) -d (write every row) + + + + Sync commands + app:data-providers:sync… + + + LeantimeApiService + per enabled DataProvider + (base url + x-api-key) + dispatch LeantimeUpdateMessage + or LeantimeDeleteMessage, + start 0 · limit 100 + + + + Symfony Messenger + async transport + consumed by supervisor: + messenger:consume async + 1 worker, time-limit 900 + + Without -j the same + messages run inline + through sync:// + + A handler that throws is + not retried — the message + goes to the failed transport + + + + LeantimeUpdateHandler + updateAsJob() + sends start · limit · + modifiedAfter · projectIds + reads results + resultsCount + + + LeantimeDeleteHandler + deleteAsJob() + sends the four types + reads the deleted ids + + + + POST /APIData/API/{type} + projects · milestones · tickets · + timesheets · workers + start is an id cursor; modifiedAfter + filters on itk_data_api_modified + + + POST /APIData/API/deleted + reads itk_projects_deleted, + itk_tickets_deleted and + itk_timesheets_deleted + + + + + Leantime DB + zp_projects · zp_tickets · zp_timesheets · zp_user + modified column and delete tables + kept current by database triggers + + + + Upsert*Handler + one message per row: + project · version · issue · + worklog · worker + EntityRemoved…Handler + + + DataProviderService + creates or updates the entity, + or marks it removed + skips the row when its modified + date is unchanged (-d disables) + + + + Economics DB + Project · Version · Issue + Worklog · Worker + + + + + + + + + + + + + + + + + + + + + + + + + + Upsert*Message — one per row + next page when resultsCount == limit, + start = last returned id + 1 + EntityRemovedFromDataProviderMessage + results + deleted ids + + + + message dispatched or request sent + + data coming back, and what it queues next + Milestones become Version, tickets become Issue, timesheets become Worklog, users become Worker. Details: docs/leantime-sync.md + diff --git a/docs/leantime-sync.md b/docs/leantime-sync.md new file mode 100644 index 000000000..293ac5a3d --- /dev/null +++ b/docs/leantime-sync.md @@ -0,0 +1,161 @@ +# Synchronization from Leantime + +Economics does not talk to Leantime directly. Leantime runs the +[data-api plugin](https://github.com/itk-dev/data-api), which exposes read-only endpoints under +`/APIData/API/`, and Economics pulls from those endpoints on a schedule. Nothing is pushed from +Leantime; every sync starts as a cron job on the Economics host. + +The pull is paged, incremental and queue driven: a command dispatches one message per entity type, +each message fetches one page of at most 100 rows, dispatches one upsert message per row, and +re-dispatches itself for the next page until a short page ends the run. + +![How Economics syncs from Leantime](images/leantime-sync.svg) + +## The pipeline + +```mermaid +flowchart TB + subgraph cron["Cron on the Economics host"] + direction LR + C1["*/15
sync-modified"] + C2["*/25 + 02:50
sync-deleted"] + C3["02:00-02:40
sync -j -<type> -d"] + end + + subgraph economics["Economics"] + direction TB + CMD["Sync commands"] + SVC["LeantimeApiService
one message per enabled DataProvider"] + UAJ["LeantimeUpdateHandler
updateAsJob()"] + DAJ["LeantimeDeleteHandler
deleteAsJob()"] + UPS["Upsert*Handler"] + RMV["EntityRemovedFromDataProviderHandler"] + DPS["DataProviderService
skips row if sourceModifiedDate is unchanged"] + end + + Q(["Messenger 'async' transport
consumed by supervisor:
messenger:consume async"]) + + subgraph leantime["Leantime + data-api plugin"] + direction TB + API["Controllers/API.php
400 on bad parameters"] + APIS["Services/APIData"] + REPO["Repositories/ApiDataRepository"] + end + + LTDB[("Leantime DB
zp_projects, zp_tickets, zp_timesheets, zp_user
itk_data_api_modified kept current by triggers
itk_*_deleted filled by triggers")] + ECDB[("Economics DB
Project, Version, Issue, Worklog, Worker")] + + C1 --> CMD + C2 --> CMD + C3 --> CMD + CMD --> SVC + SVC -- "LeantimeUpdateMessage
start 0, limit 100" --> Q + SVC -- "LeantimeDeleteMessage" --> Q + Q --> UAJ + Q --> DAJ + UAJ -- "POST /APIData/API/{projects,milestones,tickets,timesheets,workers}
x-api-key, start, limit, modifiedAfter, projectIds" --> API + DAJ -- "POST /APIData/API/deleted
x-api-key, types, deleted" --> API + API --> APIS --> REPO --> LTDB + LTDB -. "results, resultsCount" .-> UAJ + LTDB -. "deleted ids" .-> DAJ + UAJ -- "Upsert*Message, one per row" --> Q + UAJ -- "next page if resultsCount == limit
start = last id + 1" --> Q + DAJ -- "EntityRemovedFromDataProviderMessage" --> Q + Q --> UPS --> DPS + Q --> RMV --> DPS + DPS --> ECDB +``` + +Everything above assumes async handling (`-j`). Without it the same messages are stamped for the +`sync://` transport and run inline in the cron process instead of going through the queue — which is +what `app:data-providers:sync-deleted` does. + +## One update run, page by page + +```mermaid +sequenceDiagram + autonumber + participant Cron + participant Service as LeantimeApiService + participant Queue as async transport + participant Worker as LeantimeUpdateHandler + participant API as data-api plugin + participant DPS as DataProviderService + + Cron->>Service: updateAll(async, modifiedAfter) + loop per entity type, per enabled DataProvider + Service->>Queue: LeantimeUpdateMessage(start=0, limit=100) + end + Queue->>Worker: LeantimeUpdateMessage + Worker->>API: POST /APIData/API/tickets
{start, limit, modifiedAfter, projectIds} + API-->>Worker: {parameters, resultsCount, results} + loop per row + Worker->>Queue: UpsertIssueMessage(DataProviderIssueData) + end + alt resultsCount == limit + Worker->>Queue: LeantimeUpdateMessage(start = last id + 1) + else short page + Note over Worker: run ends + end + Queue->>DPS: UpsertIssueMessage + alt sourceModifiedDate unchanged and check enabled + Note over DPS: row skipped, nothing written + else + DPS->>DPS: create or update entity, flush + end +``` + +## Scheduled jobs + +Installed by the release playbook, see `.woodpecker/prod_itk_economics.yml`. + +| Job | Schedule | Command | +| --- | --- | --- | +| `sync-modified` | every 15 min | `app:data-providers:sync-modified` (window `PT1H`, async) | +| `sync-deleted` | every 25 min | `app:data-providers:sync-deleted` (window `PT1H`, handled inline) | +| `sync-deleted-week` | 02:50 | `app:data-providers:sync-deleted --interval=P1W` | +| `full-sync-projects` | 02:00 | `app:data-providers:sync -j -p -d` | +| `full-sync-workers` | 02:10 | `app:data-providers:sync -j -r -d` | +| `full-sync-versions` | 02:20 | `app:data-providers:sync -j -s -d` | +| `full-sync-issues` | 02:30 | `app:data-providers:sync -j -i -d` | +| `full-sync-worklogs` | 02:40 | `app:data-providers:sync -j -w -d` | + +The full syncs are staggered ten minutes apart because one worker consumes the queue and a full run +of one entity type takes a while. + +## Command options + +`app:data-providers:sync` selects what to sync and how: + +* `-j`, `--job`: dispatch to the `async` transport instead of handling everything inline. +* `-a`, `--all`, or one of `-p` projects, `-s` versions, `-i` issues, `-w` worklogs, `-r` workers. +* `--modified`: only fetch rows changed since the given date, passed on as `modifiedAfter`. +* `-d`, `--disable-modified-at-check`: write every fetched row even when its modified timestamp is + unchanged. The nightly full syncs use this to repair rows that drifted out of sync. + +`sync-modified` and `sync-deleted` take `--interval` (a `DateInterval` string, default `PT1H`) and +derive `modifiedAfter` / `deleted` from it. + +## Details worth knowing + +* **Entity mapping.** Leantime milestones become Economics `Version`, tickets become `Issue`, + timesheets become `Worklog`, users become `Worker`. +* **Project scoping.** Milestones, tickets and timesheets are requested only for the project ids + Economics already knows and includes (`ProjectRepository::getProjectTrackerIdsByDataProviders()`). + Projects and workers are fetched unscoped, so a new project has to be synced before its content + can follow. +* **Paging.** `start` is an id cursor, not an offset: the next page starts at the last returned id + plus one. A page shorter than the limit ends the run. +* **Incrementality.** `modifiedAfter` filters on `itk_data_api_modified`, a column the plugin adds to + the Leantime tables and keeps current with triggers, because Leantime does not update its own + `modified` column on every write path. +* **Deletions.** Leantime rows are gone by the time Economics asks, so the plugin records them in + `itk_projects_deleted`, `itk_tickets_deleted` and `itk_timesheets_deleted` via triggers, and the + `deleted` endpoint reads those tables. +* **First sync after a plugin install returns everything**, because installing stamps every existing + row with the install time. +* **Failures.** A handler that throws wraps the error in `UnrecoverableMessageHandlingException`, so + the message is not retried; it lands in the `failed` transport and the reason is logged. +* **Authentication.** Each `DataProvider` row holds the Leantime base url and the API key sent as + `x-api-key`. Only providers with `class = App\Service\LeantimeApiService` and `enabled = true` are + synced. From 7981ab2f538e937ec5d29c3c0cc91e82534f552e Mon Sep 17 00:00:00 2001 From: Troels Ugilt Jensen <6103205+tuj@users.noreply.github.com> Date: Thu, 20 Aug 2026 13:37:44 +0000 Subject: [PATCH 2/3] docs: split the sync diagrams and corrected the delete path --- docs/images/leantime-sync.svg | 163 ---------------------------------- docs/leantime-sync.md | 143 +++++++++++++++++------------ 2 files changed, 86 insertions(+), 220 deletions(-) delete mode 100644 docs/images/leantime-sync.svg diff --git a/docs/images/leantime-sync.svg b/docs/images/leantime-sync.svg deleted file mode 100644 index 73112b388..000000000 --- a/docs/images/leantime-sync.svg +++ /dev/null @@ -1,163 +0,0 @@ - - - - - - - - - - - - - How Economics syncs from Leantime - Cron-driven, paged pull through Symfony Messenger. Leantime never pushes — Economics asks, one page of 100 rows at a time. - - - - CRON · PRODUCTION HOST - - - ECONOMICS - - - LEANTIME + DATA-API PLUGIN - - - - */15 · sync-modified - changed within PT1H, async - - - */25 + 02:50 · sync-deleted - deletions, handled inline - - - 02:00–02:40 · full sync - one entity type per job, - -j (async) -d (write every row) - - - - Sync commands - app:data-providers:sync… - - - LeantimeApiService - per enabled DataProvider - (base url + x-api-key) - dispatch LeantimeUpdateMessage - or LeantimeDeleteMessage, - start 0 · limit 100 - - - - Symfony Messenger - async transport - consumed by supervisor: - messenger:consume async - 1 worker, time-limit 900 - - Without -j the same - messages run inline - through sync:// - - A handler that throws is - not retried — the message - goes to the failed transport - - - - LeantimeUpdateHandler - updateAsJob() - sends start · limit · - modifiedAfter · projectIds - reads results + resultsCount - - - LeantimeDeleteHandler - deleteAsJob() - sends the four types - reads the deleted ids - - - - POST /APIData/API/{type} - projects · milestones · tickets · - timesheets · workers - start is an id cursor; modifiedAfter - filters on itk_data_api_modified - - - POST /APIData/API/deleted - reads itk_projects_deleted, - itk_tickets_deleted and - itk_timesheets_deleted - - - - - Leantime DB - zp_projects · zp_tickets · zp_timesheets · zp_user - modified column and delete tables - kept current by database triggers - - - - Upsert*Handler - one message per row: - project · version · issue · - worklog · worker - EntityRemoved…Handler - - - DataProviderService - creates or updates the entity, - or marks it removed - skips the row when its modified - date is unchanged (-d disables) - - - - Economics DB - Project · Version · Issue - Worklog · Worker - - - - - - - - - - - - - - - - - - - - - - - - - - Upsert*Message — one per row - next page when resultsCount == limit, - start = last returned id + 1 - EntityRemovedFromDataProviderMessage - results - deleted ids - - - - message dispatched or request sent - - data coming back, and what it queues next - Milestones become Version, tickets become Issue, timesheets become Worklog, users become Worker. Details: docs/leantime-sync.md - diff --git a/docs/leantime-sync.md b/docs/leantime-sync.md index 293ac5a3d..3ec2c4e54 100644 --- a/docs/leantime-sync.md +++ b/docs/leantime-sync.md @@ -9,66 +9,66 @@ The pull is paged, incremental and queue driven: a command dispatches one messag each message fetches one page of at most 100 rows, dispatches one upsert message per row, and re-dispatches itself for the next page until a short page ends the run. -![How Economics syncs from Leantime](images/leantime-sync.svg) +The three flowcharts below split that structure into the parts worth looking at separately, and the +sequence diagram after them follows a single update run end to end. Throughout, a solid arrow is a +message dispatched or a request sent, and a dashed arrow is a response. -## The pipeline +## Fetching a page ```mermaid flowchart TB - subgraph cron["Cron on the Economics host"] - direction LR - C1["*/15
sync-modified"] - C2["*/25 + 02:50
sync-deleted"] - C3["02:00-02:40
sync -j -<type> -d"] - end - - subgraph economics["Economics"] - direction TB - CMD["Sync commands"] - SVC["LeantimeApiService
one message per enabled DataProvider"] - UAJ["LeantimeUpdateHandler
updateAsJob()"] - DAJ["LeantimeDeleteHandler
deleteAsJob()"] - UPS["Upsert*Handler"] - RMV["EntityRemovedFromDataProviderHandler"] - DPS["DataProviderService
skips row if sourceModifiedDate is unchanged"] - end + CRON["Cron on the Economics host
*/15 · */25 · 02:00–02:40"] + CMD["Sync commands
app:data-providers:sync…"] + SVC["LeantimeApiService
one message per entity type,
per enabled DataProvider"] + Q(["Messenger async transport
messenger:consume async"]) + H["LeantimeUpdateHandler · LeantimeDeleteHandler
updateAsJob() · deleteAsJob()"] + API["data-api plugin in Leantime"] + + CRON --> CMD --> SVC + SVC -- "LeantimeUpdateMessage / LeantimeDeleteMessage
start 0, limit 100" --> Q + Q --> H + H -- "POST /APIData/API/{type}
x-api-key, start, limit,
modifiedAfter or deletedAfter" --> API + API -. "results, resultsCount" .-> H + H -- "next page while resultsCount == limit" --> Q +``` - Q(["Messenger 'async' transport
consumed by supervisor:
messenger:consume async"]) +Everything above assumes async handling (`-j`). Without it the same messages are stamped for the +`sync://` transport and run inline in the cron process instead of going through the queue — which is +what `app:data-providers:sync-deleted` does, deliberately. Read **Delete ordering** below before +changing that. - subgraph leantime["Leantime + data-api plugin"] - direction TB - API["Controllers/API.php
400 on bad parameters"] - APIS["Services/APIData"] - REPO["Repositories/ApiDataRepository"] - end +## Turning a page into rows - LTDB[("Leantime DB
zp_projects, zp_tickets, zp_timesheets, zp_user
itk_data_api_modified kept current by triggers
itk_*_deleted filled by triggers")] - ECDB[("Economics DB
Project, Version, Issue, Worklog, Worker")] - - C1 --> CMD - C2 --> CMD - C3 --> CMD - CMD --> SVC - SVC -- "LeantimeUpdateMessage
start 0, limit 100" --> Q - SVC -- "LeantimeDeleteMessage" --> Q - Q --> UAJ - Q --> DAJ - UAJ -- "POST /APIData/API/{projects,milestones,tickets,timesheets,workers}
x-api-key, start, limit, modifiedAfter, projectIds" --> API - DAJ -- "POST /APIData/API/deleted
x-api-key, types, deleted" --> API - API --> APIS --> REPO --> LTDB - LTDB -. "results, resultsCount" .-> UAJ - LTDB -. "deleted ids" .-> DAJ - UAJ -- "Upsert*Message, one per row" --> Q - UAJ -- "next page if resultsCount == limit
start = last id + 1" --> Q - DAJ -- "EntityRemovedFromDataProviderMessage" --> Q +```mermaid +flowchart TB + H["LeantimeUpdateHandler · LeantimeDeleteHandler"] + Q(["Messenger async transport"]) + UPS["Upsert*Handler
project · version · issue · worklog · worker"] + RMV["EntityRemovedFromDataProviderHandler"] + DPS["DataProviderService
creates or updates the entity,
hard-deletes it, or marks sourceDeletedDate"] + ECDB[("Economics DB
Project · Version · Issue · Worklog · Worker")] + + H -- "Upsert*Message, one per row" --> Q + H -- "EntityRemovedFromDataProviderMessage,
one per deletion" --> Q Q --> UPS --> DPS Q --> RMV --> DPS DPS --> ECDB ``` -Everything above assumes async handling (`-j`). Without it the same messages are stamped for the -`sync://` transport and run inline in the cron process instead of going through the queue — which is -what `app:data-providers:sync-deleted` does. +## The plugin side + +```mermaid +flowchart TB + H["LeantimeUpdateHandler · LeantimeDeleteHandler"] + API["Controllers/API.php
validates, 400 on a bad parameter"] + APIS["Services/APIData"] + REPO["Repositories/ApiDataRepository"] + LTDB[("Leantime DB
zp_projects · zp_tickets · zp_timesheets · zp_user
itk_data_api_modified and the itk_*_deleted tables
kept current by triggers")] + + H -- "POST /APIData/API/{type}" --> API + API --> APIS --> REPO --> LTDB + API -. "results, resultsCount" .-> H +``` ## One update run, page by page @@ -93,7 +93,9 @@ sequenceDiagram Worker->>Queue: UpsertIssueMessage(DataProviderIssueData) end alt resultsCount == limit - Worker->>Queue: LeantimeUpdateMessage(start = last id + 1) + Worker->>Queue: LeantimeUpdateMessage(start = highest id + 1) + else resultsCount == limit, no usable id + Note over Worker: run stops, error logged else short page Note over Worker: run ends end @@ -120,8 +122,10 @@ Installed by the release playbook, see `.woodpecker/prod_itk_economics.yml`. | `full-sync-issues` | 02:30 | `app:data-providers:sync -j -i -d` | | `full-sync-worklogs` | 02:40 | `app:data-providers:sync -j -w -d` | -The full syncs are staggered ten minutes apart because one worker consumes the queue and a full run -of one entity type takes a while. +A single supervisor worker consumes the `async` transport — `APP_SUPERVISOR_WORKERS=1` and +`messenger:consume --time-limit=900` in `docker-compose.server.override.yml`. That is why the full +syncs are staggered ten minutes apart: a full run of one entity type takes a while, and nothing else +drains the queue while it does. ## Command options @@ -134,7 +138,7 @@ of one entity type takes a while. unchanged. The nightly full syncs use this to repair rows that drifted out of sync. `sync-modified` and `sync-deleted` take `--interval` (a `DateInterval` string, default `PT1H`) and -derive `modifiedAfter` / `deleted` from it. +derive `modifiedAfter` / `deletedAfter` from it. ## Details worth knowing @@ -144,18 +148,43 @@ derive `modifiedAfter` / `deleted` from it. Economics already knows and includes (`ProjectRepository::getProjectTrackerIdsByDataProviders()`). Projects and workers are fetched unscoped, so a new project has to be synced before its content can follow. -* **Paging.** `start` is an id cursor, not an offset: the next page starts at the last returned id - plus one. A page shorter than the limit ends the run. +* **Paging.** On the entity endpoints `start` is an id cursor, not an offset: the next page starts at + the highest usable id on the page plus one. The delete endpoint pages on `deletionId` instead — the + deletion's own row id, not the deleted entity's. Deletions are ordered by when they happened while + the entity ids on a page are in no order at all, so paging on them would skip deletions. A page + shorter than the limit ends the run either way. * **Incrementality.** `modifiedAfter` filters on `itk_data_api_modified`, a column the plugin adds to the Leantime tables and keeps current with triggers, because Leantime does not update its own `modified` column on every write path. * **Deletions.** Leantime rows are gone by the time Economics asks, so the plugin records them in `itk_projects_deleted`, `itk_tickets_deleted` and `itk_timesheets_deleted` via triggers, and the - `deleted` endpoint reads those tables. + `deleted` endpoint reads those tables. It serves one type per request: `deleteAsJob()` sends `type`, + `start`, `limit` and `deletedAfter`, and pages the way the entity endpoints do. The cursor advances + past a deletion that names no entity, because a skipped row still occupies a page position; a full + page with no usable `deletionId` stops the run with a logged error rather than re-queueing itself, + which would re-read the same page until the queue starves. +* **Delete ordering.** The delete types run timesheets → tickets → milestones → projects, children + before the parents they hang off. What holds that order is the inline `sync://` transport rather + than the dispatch order: `deleteAll()` passes `asyncJobQueue` false, so every page of one type is + handled before the next type is dispatched. On the `async` transport the four types interleave, and + a project can be reached while its timesheets are still a page behind. That matters because a + parent which cannot be hard-deleted is only marked with `sourceDeletedDate`, and nothing revisits + the mark — so `sync-deleted` running inline is load-bearing, not an oversight to fix with `-j`. +* **What blocks a removal.** `DataProviderService` hard-deletes an entity only when nothing points at + it. A project is kept if it still has invoices, issues, worklogs, versions, project billings or + service agreements; an issue is kept if it still has worklogs. Each of those points back with a + non-nullable, non-cascading foreign key, so removing anyway would be a database error rather than a + soft delete. Versions are always removable. * **First sync after a plugin install returns everything**, because installing stamps every existing row with the install time. -* **Failures.** A handler that throws wraps the error in `UnrecoverableMessageHandlingException`, so - the message is not retried; it lands in the `failed` transport and the reason is logged. +* **Failures.** A handler that fails with 408, 423, 425 or 429 rethrows, so the `async` transport + retries the message: three attempts spaced 10s, 30s and 90s, the last landing 130s after the first + failure. Any other 4xx describes the request itself, which no retry can change, so it becomes an + `UnrecoverableMessageHandlingException`, lands in the `failed` transport and is logged. Requests to + Leantime are capped at `timeout: 5` and `max_duration: 30` by `app.leantime.http_client`, so no + single page can hold the worker indefinitely. +* **Retries are an `async` transport feature only.** `sync://` has no retry strategy, so the inline + `sync-deleted` run gets none: a failure there propagates out to the command. * **Authentication.** Each `DataProvider` row holds the Leantime base url and the API key sent as `x-api-key`. Only providers with `class = App\Service\LeantimeApiService` and `enabled = true` are synced. From 6a8f15791d70ac05fceb00bd5e71c8ae90340df4 Mon Sep 17 00:00:00 2001 From: Troels Ugilt Jensen <6103205+tuj@users.noreply.github.com> Date: Thu, 20 Aug 2026 16:48:49 +0000 Subject: [PATCH 3/3] docs: noted that milestones and tickets share a delete table --- docs/leantime-sync.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/leantime-sync.md b/docs/leantime-sync.md index 3ec2c4e54..d806040e0 100644 --- a/docs/leantime-sync.md +++ b/docs/leantime-sync.md @@ -158,7 +158,9 @@ derive `modifiedAfter` / `deletedAfter` from it. `modified` column on every write path. * **Deletions.** Leantime rows are gone by the time Economics asks, so the plugin records them in `itk_projects_deleted`, `itk_tickets_deleted` and `itk_timesheets_deleted` via triggers, and the - `deleted` endpoint reads those tables. It serves one type per request: `deleteAsJob()` sends `type`, + `deleted` endpoint reads those tables. Three tables cover the four types because Leantime keeps + milestones and tickets in the same `zp_tickets` table, so both their deletions land in + `itk_tickets_deleted`. The endpoint serves one type per request: `deleteAsJob()` sends `type`, `start`, `limit` and `deletedAfter`, and pages the way the entity endpoints do. The cursor advances past a deletion that names no entity, because a skipped row still occupies a page position; a full page with no usable `deletionId` stops the run with a logged error rather than re-queueing itself,