Skip to content

Fix state events missing from MSC4222 state_after when the since token falls inside a persist batch - #20171

Draft
barodeur wants to merge 7 commits into
element-hq:developfrom
barodeur:fix-state-after-batches
Draft

Fix state events missing from MSC4222 state_after when the since token falls inside a persist batch#20171
barodeur wants to merge 7 commits into
element-hq:developfrom
barodeur:fix-state-after-batches

Conversation

@barodeur

@barodeur barodeur commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

This PR fixes the issue described as comment here: #18793 (comment)

In Element Call, this shows up as ghost participants: someone who left the call keeps being displayed until a later state change refreshes the room.

The bug is not specific to Element Call: any state event can be affected, RTC membership just changes often enough to make it visible.

What happens

Alice has a client syncing against a homeserver where events are persisted by one worker (the event persister) and /sync is served by another (the sync worker). Her client is parked in a long-poll: GET /sync?since=s99&timeout=30000.

Bob joins a call at the same moment Carol sends a message. Carol's message reaches the persister first; Bob's m.call.member arrives while that write is still in flight, so the per-room persist queue groups them into one transaction:

events (each gets its own stream ordering):
    stream_ordering 100:  m.room.message   Carol
    stream_ordering 101:  m.call.member    Bob        (state)

current_state_delta_stream (how state_after finds state changes):
    stream_id 100 ────►  (m.call.member, @bob) -> $bob_join_call
          ▲
          └─ stamped with the batch MINIMUM (100), not the event's own 101
              (see `_update_current_state_txn`)

The transaction commits: both events and the delta row are now in the database, atomically.

The persister then announces the new events over replication, one RDATA token per stream ordering — rows are only merged into one token when they share a position, and 100 and 101 don't. So the sync worker's events-stream position steps 99 → 100 → 101, and on reaching 100 it pokes the notifier.

Alice's long-poll wakes at exactly that moment. Her response is built at the worker's current position — end = 100 — with RDATA 101 still in the queue:

Sync A  (since=99, end=100):
  timeline:     events   99 < ordering ≤ 100  →  [Carol's message]
  state_after:  deltas   99 < stream_id ≤ 100 →  [$bob_join_call]  ← delivered EARLY
  next_batch:   s100                                               ← mid-batch token

No race on the client's side is needed: the server hands out the mid-batch token as next_batch. Alice's client re-polls with it, as every sync client does. The worker has meanwhile processed RDATA 101:

Sync B  (since=100, end=101):
  timeline:     events   100 < ordering ≤ 101  →  [Bob's m.call.member @101]  ✓
  state_after:  deltas   100 < stream_id ≤ 101 →  []     row is stamped 100   ✗

A state event in the timeline with an empty state_after. An MSC4222 client trusts state_after over timeline state events, so Alice's copy of Bob's call membership never updates from this response.

On a single process this cannot happen: the batch's stream IDs are released as a whole, so the position visible to /sync jumps 99 → 101 and s100 is never handed out. Only a process that learns its position from replication — any sync worker — ticks through the middle of a batch.

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
    • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
  • Code style is correct (run the linters)

A since token that falls inside a persist batch drops the deltas of the
batch's state events, because current_state_delta_stream rows are
stamped with the batch minimum stream ordering while the timeline is
selected on the event's own ordering. A worker reading the events
stream from replication routinely observes such a token, so on worker
deployments a state event can be served in the sync timeline of a room
the user is still joined to while being absent from state_after,
breaking the MSC4222 invariant that `state at since` + `state_after`
equals the state at the end of the timeline.

The tests document the storage-level and replication-level
preconditions, reproduce the bug end-to-end (including the gappy
variant where the affected state event is truncated out of the
timeline), and guard that state_after reports the resolved state at the
end of the timeline rather than a replay of the timeline.

Three tests fail at this commit; the next commit fixes them.
current_state_delta_stream rows are stamped with the *minimum* stream
ordering of the persist batch of their event, so bounding the MSC4222
delta query on the row's stream_id drops the deltas of a batch's state
events for any since token that falls inside the batch. A worker
reading the events stream from replication routinely observes such a
token, since RDATA advances the stream one event at a time; the state
event is then in the sync timeline but missing from state_after, or --
when the timeline is truncated -- silently missing altogether.

Add get_current_state_deltas_for_room_by_event_position, which bounds
each delta on the maximum of the row's stream_id and its event's own
stream ordering:

- a state event persisted mid-batch is tracked at its own position
- rows with no event (the last-local-user state clearance) keep the
  row's position
- rows stamped after their event (the partial-state resync path, which
  re-announces existing state at a fresh position) also keep the row's
  position, preserving the re-announcement

That maximum is not a bound an index can serve, so the window is
fetched as the union of two index-driven sets, with exact per-writer
filtering in Python via _filter_results_by_stream as the existing query
does: rows whose own stream_id is in the window (an index range on
(room_id, stream_id), the same cost as the existing query), and rows
whose *event* is in the window (driven by the events
(room_id, stream_ordering) index over the window's state events, joined
back via a new partial index on current_state_delta_stream(event_id) --
a row stamped below the window with an effective position inside it
must have its event inside the window). Overall cost is proportional to
the window, as before.

The early-return optimisation consults the events stream cache as well
as the delta stream cache, since a delta's effective position can now
exceed the row stamp the delta cache tracks.

Only the MSC4222 sync path uses the new method; every other consumer of
current_state_delta_stream is untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant