MSC4242: State DAGs

[kegsay]

Rendered

Synapse impl:

• CSAPI
• Federation

Complement tests: matrix-org/complement#841

No client changes required as it's not client visible.

[turt2live]

(for clarity: this is using the process defined by matrix-org/matrix-spec#2157)

[turt2live]

Implementation requirements:

• Server (unstable room version)
• Complement tests (for unstable room version)

[turt2live]

This MSC now has content.

[MadLittleMods]

I'm assuming the list of PSE should be sorted before put onto the end of the queue?

(needs clarification)

[kegsay]

Doesn't matter, as it's still deterministic if we use the wire-format ordering.

[MadLittleMods]

I was thinking this was necessary in order to "walk these events in lexicographical order" 🤔

If we don't do this then part-way down the forks, we could wander down in whatever path.

[kegsay]

Aha, you're right, thanks. Will fix.

[kegsay]

d741371

[kegsay]

Reopened this because this ordering is harder to implement efficiently than other orderings. In particular, when working on the CTE SQL for this, expressing "process one frontier element at a time, query its predecessors, sort them lexicographically then append them to results" results in a subtle ordering which depends on the processing order of parents. The CTE SQL works on a hop-based system, finding all events 1 hop away, sorting them, rinse and repeat. This loses the distinction of sorting based on parent order, causing some unit tests to fail.

As an example:

  A
  |
  B
 / \
D   C
|   |
E   F  <-- latest={E,F}, earliest={}, limit=2

The old order would sort latest=[E,F], then pop E, process it to produce D, then pop F and process it to process C, resulting in [D, C] as the output.

The proposed ordering would sort latest=[E,F] then simultaneously lookup all prevs to process [D, C] (everything 1 hop away) then sort them, resulting in [C, D] as the output.

The main important thing here to have a deterministic output, I don't particularly care which ordering we use, and given the CTE one is faster, that one should probably be the one we do.

[gamesguru]

This seems insufficient for guaranteeing fully up-to-date reconciliations. While walking prev_state_events guarantees full causal history for an event, it fails to uncover concurrent branches of state events (hidden forward extremities generated during network partitions). Relying on the "forward-push" mechanism (new events being sent) to merge branches means partitions will lead to semi-permanent divergence in inactive rooms. Active partition-healing and discovery mechanisms are essential to the eventual consistency this MSC aims for... relegating them to "follow-up" MSCs feels inadequate. I'm sure there has been some work/investigation here, but I have not seen or come across those proposals. I have some ideas here regarding /room_digest endpoints and gossip mechanisms.

If state DAGs are to be their own fully contiguous, self-sufficient sub-graphs, they definitely deserve a dedicated /_matrix/federation/v2/get_missing_state_events endpoint rather than overloading /get_missing_events with a boolean param. Because this MSC introduces completely different traversal logic when walking the state DAG, overloading the legacy API endpoint is surely an anti-pattern. We need a clear, dedicated endpoint for this specific BFS walk.

The proposed exponential limit markers may result in network bottlenecks when resolving deep partitions. Having recently implemented binary-lifted jump pointers (shorteventid_statejumppointers) to alleviate database thrashing under the O(N) hop-by-hop walk of large test cases, I feel a pressing need for that same O(log N) efficiency over the network wire.

On the network side, I caution against leaning too heavily on Bloom Filters. While convenient at shallow depths, their footprint scales linearly to keep a low false-positive rate. A hybrid approach seems better. Similar to how C++ switches from quicksort to heapsort to bound worst-case overhead, federation syncs should enforce an algorithmic "switch" above a critical divergence depth... explicitly shifting to the O(log N) techniques already mentioned (Merkle search trees, Merkle-Patricia tries, Range-based reconciliation) to maintain a healthy and efficient network.

[kegsay]

I don't disagree with most of what you have written, but you don't seem to understand that we have a working protocol and large protocol changes are difficult to land.

I explicitly call out that we don't have an anti-entropy mechanism today and consciously defer it. You may disagree, that's fine. It is worth noting that in practice the lack of such a mechanism hasn't materially impacted the protocol because in practice relying on forward-push is enough. Anti-entropy does have its own costs (constant background chatter gossiping latest forward extremities) so it's not as clear cut as you're portraying.

It's a taste issue if you want a new endpoint or not. Both operations walk a DAG and have the same inputs. If you have any non-taste issues which may lean to one or the other then let me know.

Yes, it can take a while to reconcile with a server which has diverged significantly from you. I'm very open to suggestions to improve this, though you need to bear in mind we are working in a Byzantine environment. Most optimisations assume servers follow the protocol and degrade to a DAG walk (or worse, break entirely). If you have a way to ensure the jump pointers can't be outright malicious, please comment with a suggestion.

We're not using bloom filters in this proposal.

[gamesguru]

1. I certainly see the value in not attaching too many subordinate MSCs, and how that could slow things down more. However, I insist on proper separations of concerns. If providing "eventual consistency" (preferably: rapid consistency) falls under the purview of a different card or MSC, that should be clearly admitted, explained, and documented. This can be especially misleading given Synapse's tendency not to re-evaluate anything it already processed. Advertising State DAGs intrinsically as a sufficient standalone solution causes some confusion to the serious reader. My MSC4521 aims optimistically to reconcile differences of 100,000 elements over populations of 10 million in about 5-10 seconds. Smaller reconciliations (like under 1000 differences) are nearly instant, under 200 ms—theoretically fast enough to keep all partitions unified in real-time (except for deliberate Byzantine defectors).

2. /get_missing_state_events will cache dedicated ETags and consume a homogeneous JSON structure... also allows homeserver admins to route state-DAG syncs to dedicated worker pools or apply custom reverse proxy rules and rate limits. Better to keep it separate.

3. Merkle proofs can attest to the credibility of a path or series of binary hops. While state resolution must still be run (i.e., the corresponding event chain must be fully fetched and validated), a valid Merkle path is instantly more credible, worthy, and promising as a lead than an opaque, ostensibly Byzantine input.

   A "graceful" fallback to linear walking O(N) could just as easily be Byzantine. I could self-sign 10 million PDUs and stream them to you one after another, all from the same domain. Linear walks should probably be a worst-case fallback if all else fails.

   To avoid enforcing the full download back to m.room.create, we can leverage the finality concepts you recently outlined in your PaPoC '26 ERA paper. By utilizing "epoch events" or an m.room.finality seal issued by the finality arbiter (e.g., the immutable Creator role), we can certify or commit a given snapshot, allowing nodes to act as "thin servers" and trust the cryptographic seal.

   This perfectly aligns with the ERA paper's note that finalized events can be compacted. We could establish a precise cadence (e.g., checkpointing every 5000 state events and adjacent merge points), preemptively compress the deterministic JSON with xzip, and seal the bundle cryptographically to maintain high availability indefinitely.

   For room versions or customizations not opting into a finality clause, any rebuilds triggered later by backdated admin events remain the server's responsibility to process from the conflict out to the frontier (existing implementations and runtime characteristics can be substantially improved, to the point of allowing arguably even the largest servers to rebuild huge rooms leisurely).

[gamesguru]

It may also be worth analyzing or mentioning m.space.parent. This introduces some cross-room validation complexity, per the spec:

To avoid situations where a room falsely claims it is part of a given space, m.space.parent events should be ignored unless one of the following is true:

• A corresponding m.space.child event can be found in the supposed parent space.
• The sender of the m.space.parent event has sufficient power level in the supposed parent space to send m.space.child state events (there doesn't need to be a matching child event).

https://spec.matrix.org/v1.2/client-server-api/#mspaceparent-relationships

---

As a detail, m.room.tombstone seems to have very low overhead (not having any forward extremities or being referenced). But is worth including for completeness.

Lastly, I'd like to bring up m.policy.rule.* and m.room.policy events. These events are very common in some rooms, and I'm curious if they have the potential to slow things down at all. Seems these events would carry the cost of standard membership events. Rooms today like the KDE help lounge (with over 500K membership churn due to their timeout/rejoin model) can be almost impossible to join and require significant disk/CPU usage if ever successfully joined.

[gamesguru]

Hi, I am doing a deep dive on redactions in my state res library lately.

Since a new redaction algo was not specified or updated, and auth_events were gutted out in favor of prev_state_events, this would mean redactions strip and break a lot of things. Am I missing something?

We would need to clarify protected fields and hashing semantics.

Technically just overloading the old auth_events field name would be an easier refactor, at the expense of confusion/semantic poisoning (and possible resolution failures if the create event and room version can't be fetched)... for a year or two, mere presence of the prev_state_events will be a distinguishing factor (e.g., maps numerically to v13+ and state res v2.2+, where there is no v14 nor v2.3 yet).

I think some edge cases about how redacted state events with content: {} can influence fork resolution are worth adding, either as manual notes or formally as unit, or rather I guess Complement tests? I did a little work on an internal PR (against my own Complement fork's master/main). I'll invite you there, we can maybe discuss some ideas at leisure. I may be able to put a small demo together.

I have a feeling some of the unintuitive logic around the same-event-ID despite redaction and the hashes: { ... } logic/conundrums can be addressed and simplified by choosing the right Merkle tree construction and combining header leaves correctly, so content can be stripped without mutating event_root. Or content_hash can be checked if it matches the "empty commitment." This sounds like a big lift, but if redactions require a lot of work anyway we may want to do things a more "intuitive" way and refactor while we are at it (I'm not saying I'm married to the idea of Merkle trees/paths/proofs here, but it's certainly worth considering over the coming months so we can get more understanding of the benefits and any possible limits).

Starting a separate review for this, since I have a hunch redactions may deserve a dedicated conversation thread.