perf(backfill): let the homeserver withhold what the walk would drop (CHOO-1436) - #382
Open
amaudruz wants to merge 1 commit into
Open
perf(backfill): let the homeserver withhold what the walk would drop (CHOO-1436)#382amaudruz wants to merge 1 commit into
amaudruz wants to merge 1 commit into
Conversation
…(CHOO-1436) A room's history is mostly things the message log does not keep — runtime state, task transitions, tool-call and LLM reports. The backfill pulled all of it, decoded it, and dropped it locally. `limit` counts what the homeserver sends, so a page thick with discards is a page mostly wasted: on a busy room the walk spends most of its pages, and most of its time, transferring events it already knows it will not write. `read_history` now takes `exclude_types`, which the Matrix transport passes as a `not_types` event filter. Same rows at the end, far fewer round trips. Three things this is careful about: **The denial is derived, not restated.** `NOT_RECORDED_FILTER` is built from the sets in `recorded_types.py`, because the two drifting apart would mean the homeserver silently withholding something the log wanted. It stays a denial for the same reason the log does: an allowlist would have the server skip a type nobody has classified yet, turning "record what we do not recognise" into "lose it" with nothing to notice. **Reconcile deliberately does not filter.** That walk reports what it discarded, and the disclosure is part of its answer — a homeserver that withheld those events would make the report read as though the room had never carried any. **The parameter is required.** A walk that filters and a walk that counts what it drops want opposite answers, and neither should get one by omission. Direct access to the homeserver's own store is not an alternative, for the record: Tuwunel is an embedded RocksDB, held open by the running process, with an undocumented internal encoding. The client API is the only stable reader. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The backfill walks a room's whole history to reconstruct it into the message log. Most of what it walks is not conversation — runtime state, task transitions, tool-call and LLM reports — and it was pulling all of it, decoding it, and dropping it locally.
limitcounts what the homeserver sends, so a page thick with discards is a page mostly wasted. On a busy room that is most of the pages and most of the time.read_historynow takesexclude_types, which the Matrix transport passes as anot_typesevent filter. Same rows written at the end, far fewer round trips.Three things this is careful about
The denial is derived, not restated.
NOT_RECORDED_FILTERis built from the sets already inrecorded_types.py. Restating them would let the two drift, and drift here means the homeserver withholding something the log wanted.It stays a denylist, for the same reason the log is one: an allowlist would have the server skip a type nobody has classified yet, turning the deliberate "record what we do not recognise" into "lose it", with nothing left to notice.
Reconcile deliberately does not filter. That walk reports what it discarded (
ignored_by_type) and the disclosure is part of its answer — a homeserver that withheld those events would make the report read as though the room had never carried any. The parameter is required rather than defaulted for exactly this reason: a walk that filters and a walk that counts want opposite answers, and neither should get one by omission.On reading the homeserver's database directly
Asked, and worth recording as a no: Tuwunel is an embedded RocksDB, not SQL. There is nothing to
SELECT, the store is held open by the running process (which must stay up during the migration), and the key layout and PDU encoding are conduwuit/tuwunel internals free to change in any release. The client API is the only stable reader.Checks
2,652 tests pass; ruff and mypy clean. New coverage: the filter reaches nio, an empty exclusion sends no filter at all, the walk asks for the right denial, and the denial cannot drift from the denylist it comes from.
🤖 Generated with Claude Code