Skip to content

perf(ama): drain durable operations inline, relax cron cadence - #299

Merged
CaliCastle merged 3 commits into
devfrom
perf/ama-inline-operations-drain
Aug 23, 2026
Merged

perf(ama): drain durable operations inline, relax cron cadence#299
CaliCastle merged 3 commits into
devfrom
perf/ama-inline-operations-drain

Conversation

@CaliCastle

@CaliCastle CaliCastle commented Aug 23, 2026

Copy link
Copy Markdown
Owner

Why

Neon compute usage has been ~183 CU-hrs/month — the endpoint never autosuspends. The */5 * * * * AMA work cron issues an unconditional hold-release UPDATE plus a claimDue query on every run, resetting Neon's 5-minute autosuspend timer exactly at the threshold, 288 times a day. Page visits are not a factor (public routes are fully static under Cache Components and never touch the database).

What changed

Inline drain on mutation. New kickAmaOperations() in lib/ama/booking/server.ts runs runner.run() via waitUntil in the background of the current invocation. It is called after a successful response in the five routes that enqueue durable work:

  • Stripe webhook — confirmation email and Finalizing Booking recovery start the moment payment lands (previously up to 5 minutes later)
  • Guest cancel / reschedule — refunds, artifact updates, and emails go out immediately
  • Admin booking action / operation retry — admin "retry" executes instantly

Each of these routes now sets maxDuration = 60 so the runner's 45-second time budget always fits.

Sparser crons. AMA work drops from */5 to */30; media reconcile drops from */15 to hourly on the hour, aligned so wakeups consolidate. The cron is now purely the clock fallback for reminders, retry backoff, and expired Slot Hold bookkeeping.

Reviewer notes

  • No double-execution risk: the operations runner claims work under leases (claimDue + lease token), so a concurrent inline drain and scheduled sweep cannot run the same operation twice. An interrupted drain's lease expires and the next run reclaims it.
  • Correctness never depended on sweep cadence: availability filters expired-but-unreleased holds by their own expiry, and hold→booking conversion transactionally requires expiresAt > now. releaseExpiredHolds is bookkeeping only.
  • Accepted trade-off: the 24h/1h session reminders and failed-operation retry backoff can now land up to 30 minutes later than scheduled. Everything guest-facing gets faster.
  • Expected Neon impact: 48 short wakeups/day instead of always-on — roughly ~30 CU-hrs/month (~85% reduction).
  • Cron schedule changes take effect on the next production deploy.

Verification

  • npx tsc --noEmit clean
  • pnpm test:ama — 668 tests + 9 migration tests pass
  • pnpm test:unit — 1245 tests pass

Greptile Summary

The PR starts AMA durable-operation drains from successful enqueueing mutations and reduces cron frequency to allow the database to autosuspend.

  • Adds background, lease-safe draining after booking creation, guest management mutations, and relevant admin actions.
  • Repeats full operation batches within a bounded inline budget.
  • Avoids draining for Stripe webhook outcomes that enqueue no work.
  • Changes AMA work cadence to every 30 minutes and media reconciliation to hourly.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
lib/ama/booking/server.ts Adds a bounded inline drain that repeatedly runs full operation batches under the existing lease-based runner.
lib/ama/booking/http.ts Exposes processed webhook outcomes to route wiring without changing signature verification or durable booking semantics.
app/api/ama/stripe/webhook/route.ts Starts the inline drain only when webhook processing creates a booking and enqueues durable work.
app/api/ama/manage/[token]/cancel/route.ts Starts durable-operation processing after successful guest cancellation responses.
app/api/ama/manage/[token]/reschedule/route.ts Starts durable-operation processing after successful guest rescheduling responses.
app/api/admin/ama/bookings/[bookingId]/route.ts Starts durable-operation processing after successful owner booking actions.
app/api/admin/ama/operations/[operationId]/route.ts Starts durable-operation processing after successful operation actions such as retry.
vercel.json Reduces AMA and media cron frequency to consolidate database wakeups.

Sequence Diagram

sequenceDiagram
  participant Mutation as AMA mutation/webhook
  participant Queue as Durable operations
  participant Drain as Inline runner
  participant Provider as Email/Calendar/Stripe
  participant Cron as 30-minute cron
  Mutation->>Queue: Enqueue durable work
  Mutation->>Drain: kickAmaOperations via waitUntil
  loop While batch is full and budget remains
    Drain->>Queue: Claim due work under leases
    Drain->>Provider: Execute side effects
    Drain->>Queue: Complete or schedule retry
  end
  Cron->>Queue: Fallback sweep
Loading

Reviews (3): Last reviewed commit: "fix(ama): give the inline drain room to ..." | Re-trigger Greptile

Context used (3)

The */5 AMA work cron kept the Neon endpoint awake around the clock:
each run issues an unconditional hold-release UPDATE plus a claimDue
query, resetting the 5-minute autosuspend timer every cycle (~183
CU-hrs/month of always-on compute).

Mutating endpoints now kick a background drain of the durable
operations queue via waitUntil in the same invocation that enqueued
the work, so booking emails, Finalizing Booking recovery, refunds,
and admin-triggered retries start immediately instead of waiting for
the next sweep. The lease-based runner already tolerates concurrent
drains, so the kick and the scheduled sweep cannot double-execute.

With request-triggered work handled inline, the cron becomes a pure
clock fallback (reminders, retry backoff, expired Slot Hold
bookkeeping) and drops to */30, with media reconcile hourly and
aligned so wakeups consolidate. Reminders and retries may now land up
to 30 minutes later; everything guest-facing gets faster, and the
database can suspend between wakeups.
@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown

Preview deployment ready

Environment Deployment Commit
Preview Visit Preview 26ce340

Branch: perf/ama-inline-operations-drain · View deployment logs

Comment thread app/api/ama/stripe/webhook/route.ts Outdated
Comment thread lib/ama/booking/server.ts
Address review feedback on the inline operations drain:

- The Stripe webhook handler now surfaces the processed outcome via an
  onOutcome callback, and the route kicks the drain only for
  booking_created. Duplicate, ignored, orphaned, booking-exists, and
  hold-release deliveries return 200 without enqueueing work, so they
  no longer start a runner pass.
- A single runner pass caps at its batch size, so older due work could
  crowd out the operation the triggering mutation just enqueued. The
  kick now re-runs the drain while full batches come back, bounded by
  a 30-second inline budget; leftovers stay with the scheduled sweep.
Comment thread lib/ama/booking/server.ts
A single slow first pass (ten due operations with provider calls) could
exhaust the 30-second outer budget before the drain reached the
operation the triggering mutation enqueued, deferring it to the
scheduled sweep.

Raise the mutating routes to the platform-default 300s maxDuration
(the explicit 60 had actually lowered it) and extend the drain budget
to 240s, so 240s of passes plus one worst-case 45s pass still finish
inside the ceiling. Passes are claim-ordered by nextAttemptAt, so each
one moves the queue strictly toward the newest operation.
@CaliCastle
CaliCastle merged commit 9837a25 into dev Aug 23, 2026
5 checks passed
@CaliCastle
CaliCastle deleted the perf/ama-inline-operations-drain branch August 23, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant