MSC3871: Gappy timelines - #3871
Conversation
turt2live
added
proposal
As suggested by @benkuly, #3871 (comment) The synthetic event approach is still listed as an alternative
d90bb0b to
5f7bb0e
Compare
…n explanation just above
| messages anyway but they do probably see our slow `/messages` pagination. | ||
|
|
||
|
|
||
| ### Expose `prev_events` to the client |
There was a problem hiding this comment.
That would simplify things dramatically, seems.
Same approach of gap-filling that is used for server-server could be employed for client-server.
Additionally, would make it possible for clients to issue full events, as wondered in MSC4080, since it becomes possible for clients to detect forward extremeties.
| This describes the new `gaps` response field being added to the `200 response` | ||
| of `/messages`: |
There was a problem hiding this comment.
We should probably also add a gaps field to the timeline in /sync. It's important for clients to know wherever they might be missing a message.
| { | ||
| "prev_pagination_token": "t9", | ||
| "event_id": "$fred", | ||
| "next_pagination_token": "t10", | ||
| } |
There was a problem hiding this comment.
(this is in the context of paginating forwards)
I'm not sure whether we should consider the gap in front of the latest events to actually be a gap in the gaps list.
The problem is that the client has no way to know whether it's just a forward extremity (one of the latest events in the room) to know whether it should display the gap in the timeline. I.e. we probably don't want to display a "gap" after the latest events in the room.
We should either omit the gap for forward extremities in the room or add a field to indicate it's a forward extremity.
There was a problem hiding this comment.
I'm currently thinking that we should just omit the gap after the latest messages in the room. The /messages endpoint already returns start/end pagination tokens for people wanting to continue off either side. And this way clients can simply just display a gap wherever we say there is one without needing complex logic.
See [MSC: Gappy timelines](matrix-org/matrix-spec-proposals#3871)
To try out the flow:
- **Default to fast responses with gaps**: As a default, we can always
respond quickly and indicate gaps ([MSC3871]
(matrix-org/matrix-spec-proposals#3871)) for
clients to paginate at their leisure.
- **Fast back-pagination**: Clients back-paginate with
`/messages?dir=b&backfill=false`, and Synapse skips backfilling
entirely, returning only local history with gaps as necessary.
- **Explicit gap filling**: To fill in gaps, clients use
`/messages?dir=b&backfill=true` which works just like today to do a best
effort backfill.
This allows the client to back-paginate the history we already have without
delay. And can fill in the gaps as they see fit.
This is basically a simplified version of [MSC4282]
(matrix-org/matrix-spec-proposals#4282).
|
|
||
| Name | Type | Description | required | ||
| --- | --- | --- | --- | ||
| `gaps` | `[GapEntry]` | A list of gaps indicating where events are missing in the `chunk` | no |
There was a problem hiding this comment.
For ease of processing, we should declare that the order of the gaps is the in the same direction as the chunk of messages.
| "prev_pagination_token": "t6", | ||
| "event_id": "$corge", | ||
| "next_pagination_token": "t5", |
There was a problem hiding this comment.
Per https://spec.matrix.org/v1.18/appendices/#pagination, we should probably align these fields to be prev_batch and next_batch
The juicy details and explanation are in the diff itself. Split out from #18873 in order to fix paginating from [MSC3871](matrix-org/matrix-spec-proposals#3871) gap tokens actually backfilling history. To be clear, this is a good change to make outside of the [MSC3871](matrix-org/matrix-spec-proposals#3871) use case. For example (as the new Complement test shows), fixes a problem where if you try to paginate `/messages` from tokens returned by `/context`, we could fail to backfill anything new and hide away history. Also fixes matrix-org/complement#853
The juicy details and explanation are in the diff itself. Split out from element-hq#18873 in order to fix paginating from [MSC3871](matrix-org/matrix-spec-proposals#3871) gap tokens actually backfilling history. To be clear, this is a good change to make outside of the [MSC3871](matrix-org/matrix-spec-proposals#3871) use case. For example (as the new Complement test shows), fixes a problem where if you try to paginate `/messages` from tokens returned by `/context`, we could fail to backfill anything new and hide away history. Also fixes matrix-org/complement#853
9 participants
A proposal for letting homeservers give feedback to clients where there is a gap in the timeline from a
/messagesresponse.Spawned from a discussion with @erikjohnston about making
/messagesfasterRendered
Matrix Live demo: https://www.youtube.com/watch?v=Bo7GaLRhtc0&t=409s
Implementations:
gapsin/messageselement-hq/synapse#18873