Skip to content

feat: paginated the Leantime delete sync - #334

Merged
tuj merged 4 commits into
developfrom
feature/leantime-deleted-pagination
Aug 20, 2026
Merged

feat: paginated the Leantime delete sync#334
tuj merged 4 commits into
developfrom
feature/leantime-deleted-pagination

Conversation

@tuj

@tuj tuj commented Aug 19, 2026

Copy link
Copy Markdown
Contributor

Link to ticket

https://leantime.itkdev.dk/#/tickets/showTicket/8000

Description

The plugin's /deleted endpoint is now paginated and serves one type per request
(data-api#21), so this is the consumer
side of that change. It has to land together with the plugin: typestype and the flat
results array are both breaking, and the delete sync is broken in between either way.

Until now deleteAsJob() asked for all four types at once and read back an object keyed by
type, which meant a single response carrying the entire deletion history — the thing the
300s max_duration in config/packages/framework.yaml was sized for. That value stays as
it is, since it also covers the entity endpoints, but nothing depends on it being that
generous any more.

The shape now mirrors updateAsJob(), which is the point:

  • delete() queues one LeantimeDeleteMessage per type instead of one per data provider.
    The order is kept in a DELETED_TYPES constant — timesheets, tickets, milestones,
    projects — so children are removed before the parents they hang off.
  • LeantimeDeleteMessage gains type, start and limit, matching
    LeantimeUpdateMessage.
  • deleteAsJob(string $type, int $startId, int $limit, int $dataProviderId, …) fetches one
    page, dispatches an EntityRemovedFromDataProviderMessage per row, and queues the next
    page while resultsCount === limit.

The cursor is the endpoint's new deletionId, not the deleted entity's id. Deletions
are ordered by when they happened, so the entity ids on a page are in no particular order —
paging on them would skip deletions. Two consequences, both deliberate and both tested:

  • The cursor is advanced before the "no id, skip it" guard. A deletion that names no
    entity still occupies a place in the page, so it has to be paged past or the next request
    refetches the same page.
  • A full page with no usable deletionId anywhere stops with an error rather than
    re-queueing itself. This is the same guard, and the same reasoning, as
    #326 added to updateAsJob(): an
    invisible infinite loop starves every other sync of the single worker, so stopping loudly
    beats looping.

The narrow catches around the row loop from
#325 are untouched — a row-level failure
is still logged and skipped, while a dead database or an unreachable Leantime still halts
the run.

SyncDeletedCommand, deleteAll() and delete()'s public signatures are unchanged, so
app:data-providers:sync-deleted --interval=P1W is called exactly as before.

Screenshot of the result

Not applicable — no user interface is affected.

Checklist

  • My code is covered by test cases.
  • My code passes our test (all our tests).
  • My code passes our static analysis suite.
  • My code passes our continuous integration process.

task test — 414 tests, 19216 assertions, green. task coding-standards:check and
markdownlint clean.

New and updated coverage:

  • Five cases added to Unit\Service\LeantimeApiServiceTest, alongside the updateAsJob()
    ones from fix: stop the Leantime pagination cursor restarting the sync #326: the next page starts after the highest deletionId; the cursor follows
    deletionId and not the entity id, with the two deliberately in opposite orders; a
    deletion with no entity id still moves the cursor; a full page with no usable
    deletionId stops and logs; a partial page queues nothing.
  • Integration\Service\LeantimeApiServiceTest::testDeleted now feeds one flat per-type
    payload per request and asserts all four request bodies, including that the timestamp
    goes out under deleted — the key the plugin actually reads. Every existing assertion is
    kept: the invoiced project and issue survive, sourceDeletedDate is stamped, and the
    rows behind the entry with an unparsable date still get processed.
  • Unit\MessageHandler\LeantimeDeleteHandlerTest updated for the new argument list.

As with updateAsJob(), the integration test does not exercise the requeue: its next-page
message would be handled inline by the container's real service and hit the network. That
path is covered by the unit tests instead.

/deleted now serves one type per request with start/limit (data-api#21), so the
whole deletion history no longer has to arrive in a single response — which is
what the 300s max_duration was sized for.

delete() queues a message per type and deleteAsJob() pages through them the way
updateAsJob() already does. The cursor is the endpoint's new deletionId, not the
deleted entity's id, since deletions are ordered by when they happened. It
advances past a deletion that names no entity — a skipped row still has to be
paged past — and a full page with no usable deletionId stops with an error
rather than re-queueing itself.
@tuj tuj self-assigned this Aug 19, 2026
tuj added 2 commits August 19, 2026 08:11
deleteAsJob() gained a match over the type and a second $data->resultsCount
read, both of which the baseline already ignores once for updateAsJob(), and its
$dataProviderId moved from the first parameter to the fourth.
data-api#21 renamed the parameter to match `modifiedAfter` on the entity
endpoints. The old `deleted` now answers 400 rather than being ignored, so the
key cannot go missing unnoticed the way it did before.
@tuj
tuj requested a review from turegjorup August 19, 2026 08:28

@turegjorup turegjorup left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know enough about the sync logic to know about the above comment. It it is a blocker or a trivial issue?

Comment thread src/Service/LeantimeApiService.php Outdated
public const TICKETS = 'tickets';
public const TIMESHEETS = 'timesheets';
public const WORKERS = 'workers';
/** The types the deleted endpoint tracks, children before the parents they hang off. */

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can not be guaranteed when running async. Depending on the entity model consider using doctrines orphan removal to also delete children when parents are deleted.

Or scope deletes by projects so that all deletes within a project happens as one message/job.

@tuj tuj Aug 19, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right that the queue gives no ordering guarantee, and the comment was wrong to imply the
constant provides one. What actually holds the order here is that this path is not async:
SyncDeletedCommand calls deleteAll(false, …), so both the delete messages and the
EntityRemovedFromDataProviderMessages get TransportNamesStamp('sync') and every handler runs
inline — including the next-page dispatch, so a type's pages all finish before the next type
starts. I've rewritten the comment to say that, and to record that fanning the four types out up
front only works under inline handling: switching this path to async would mean chaining the types
instead.

On orphan removal — it would take billing data with it. projectRemovedFromDataProvider() and
issueRemovedFromDataProvider() refuse to hard-delete while invoice-bound children exist and set
sourceDeletedDate instead, and worklogRemovedFromDataProvider() protects any worklog attached
to an invoiceEntry. A cascade from the parent would delete exactly the rows those checks exist to
keep.

On scoping by project — /deleted pages by type over deletionId and has no project dimension, so
grouping by project would mean holding the whole deletion history in memory, which is what the
pagination is here to avoid.

Digging into this did turn up a real bug, unrelated to pagination:
projectRemovedFromDataProvider() checks invoices, issues and worklogs but not versions, and
version.project_id has no ON DELETE, so a project whose milestone deletion was missed passes the
removable check and then hits an FK violation. Fixing that separately.

@tuj
tuj merged commit fc779d0 into develop Aug 20, 2026
6 checks passed
@tuj
tuj deleted the feature/leantime-deleted-pagination branch August 20, 2026 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants