CS-11125: per-realm advisory lock on data-plane write paths

[lukemelia]

Summary

• Threads the per-realm withWriteLock (now on DBAdapter, via Refactor: move per-realm advisory lock into DBAdapter.withWriteLock #4839) through the data-plane mutations in packages/runtime-common/realm.ts, so two replicas behind a load balancer can no longer race on the same realm URL.
• Pre-CS-11125 the lock from CS-10891 only wrapped realm-lifecycle handlers (create / publish / unpublish / delete). POST / PATCH / DELETE <realm>/<path> and POST <realm>/_atomic — the routes real client traffic actually hits — did not take it. Single-replica deploys were fine because the Node event loop was the implicit serializer; the moment a second replica appears, last-writer-wins on EFS and interleaved boxel_index rows.

Stacks on #4839.

Linear: CS-11125.

What's in

• write / writeMany / delete / deleteAll wrap their inner work in dbAdapter.withWriteLock(this.url, ...). Inner methods are renamed _batchWriteUnlocked / _deleteUnlocked / _deleteAllUnlocked so handlers that need the lock around a wider critical section reuse the inner without re-entering the lock (a second pg_advisory_xact_lock on the same key would block on a different pinned pool connection).
• handleAtomicOperations takes the lock at the handler boundary so the add / update precheck is in the same critical section as the writeMany — fixes the /_atomic TOCTOU spelled out in the ticket.
• patchCardInstance takes the lock around the indexEntry read + merge + write, so two concurrent PATCHes from different replicas can't both read the same original and clobber each other's merge.

Lock granularity stays per-realm — concurrent writes to different realms remain fully parallel. SQLite (host) is a passthrough, so this is a no-op there.

Regression test

packages/realm-server/tests/data-plane-write-lock-test.ts:

• Two concurrent PATCHes against the same card with non-overlapping attribute changes — both must persist. Without the lock the second writer computes its merge from pre-first state and drops the first writer's field.
• Two concurrent /_atomic add ops on the same href — must yield 201 + 409, not 201 + 201. Without the lock both pass the precheck and both write.

Compatibility

• Single-replica deploys: the lock is a no-op without contention — pure overhead is one pg_advisory_xact_lock round-trip per data-plane mutation.
• Lands before CS-10899 (atomicity follow-up) and the cache-invalidation tickets that assume writes are already serialized.

Test plan

• realm-advisory-locks-test (7/7) + realm-cleanup-transaction-test (3/3) pass after the rebase on Refactor: move per-realm advisory lock into DBAdapter.withWriteLock #4839.
• pnpm exec tsc -p packages/runtime-common --noEmit and the realm-server / postgres / host projects — zero new type errors vs. main.
• pnpm test:wait-for-servers against a running stack to drive the new data-plane-write-lock-test end-to-end. (Requires Synapse + base realm-server up — not run locally, deferred to CI.)
• Eyeballed patchCardInstance / handleAtomicOperations re-indentation diffs (heavy because everything is wrapped in a closure) — please review the structural changes carefully.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR extends the per-realm advisory write lock (added to DBAdapter.withWriteLock in #4839) to the data-plane mutation paths in runtime-common so that concurrent writers on different realm-server replicas can’t race on the same realm URL (preventing lost updates and TOCTOU issues around PATCH and /_atomic).

Changes:

• Wrap Realm.write, writeMany, delete, and deleteAll in dbAdapter.withWriteLock(this.url, …) and introduce unlocked inner helpers (e.g. _batchWriteUnlocked) for callers that already hold the lock.
• Take the lock at the handler boundary for /_atomic and PATCH so reads + validation + writes are serialized in one critical section.
• Add a new realm-server regression test module and register it in the test index.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
packages/runtime-common/realm.ts	Threads withWriteLock through public mutation entry points and wraps /_atomic + PATCH critical sections; renames inner write/delete helpers to avoid lock re-entrancy.
packages/realm-server/tests/index.ts	Registers the new data-plane lock regression test module.
packages/realm-server/tests/data-plane-write-lock-test.ts	Adds regression tests for concurrent PATCH lost-update and /_atomic add TOCTOU serialization.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

Host Test Results

    1 files  ±    0      1 suites  ±0   3h 47m 48s ⏱️ + 1h 51m 1s
2 658 tests ±    0  2 642 ✅  -     1  15 💤 ± 0  0 ❌ ±0  1 🔥 +1 
5 354 runs  +2 677  5 322 ✅ +2 660  30 💤 +15  1 ❌ +1  1 🔥 +1 

Results for commit 65b0596. ± Comparison against earlier commit e8f6a07.

For more details on these errors, see this check.

Realm Server Test Results

    1 files  ±0      1 suites  ±0   9m 36s ⏱️ +25s
1 376 tests ±0  1 376 ✅ ±0  0 💤 ±0  0 ❌ ±0 
1 457 runs  ±0  1 457 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 65b0596. ± Comparison against earlier commit e8f6a07.

[lukemelia]

Addressed both Copilot review comments and the Lint failure:

• Lint (prettier on realm.ts:4111) — collapsed the buildCardJsonEtag(...) call back onto a single line. Unrelated to advisory locks, blocking merge.
• r3243846068 — Realm.clearLocalCaches() deletion was an unintentional rebase casualty. Restored at the original CS-11043 site in commit e8f6a07.
• r3243846106 — added ?waitForIndex=true to the two /_atomic POSTs in data-plane-write-lock-test.ts. /_atomic returns on durability, not on index completion, so the immediate GET could race the indexer. The PATCH test in the same file uses writeMany which defaults to waitForIndex: true, so only /_atomic needed the explicit opt-in.

Follow-up CS-11156 (cross-replica cache invalidation via NOTIFY broadcast) is open as draft PR #4842, stacked on this branch.