Skip to content
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions .claude/prompts/01-notification-counts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
Add notification counting for events

- spec states "The updated notification count from a new event MUST appear in the same /sync response as the event itself."
- store as rooms.users.notificationVersions (userid, roomid, version) -> types.Notifications{notifs, highlights, ...}
- store types.Notifications as msgpack with single letter keys
- for every event send we must (SendLocalEvents, SendFederatedEvents):
- get local users in the room
- get all their push rules (stub these for now), before any write txn
- inside the txn for each event, eval each users rules, map eventsToUserNotifications[id.EventID]types.Notifications{}
- pass to txnStoreEvents, we apply notificationVersions (userid, roomid, eventVersion) -> types.Notifications{}
- version is the event version (so can get back to the eventid)
- on sync, just
- range notificationVersions (userid, roomid) => sum counts
- on receipt just
- clearrange up to (userid, roomid, eventVersionFromReceipt)

Explore the codebase and come up with a plan to implement the above changes.

... implemented, second prompt:

Now we need to implement an EventNotificationIterator to compact notificationVersions:

- compact notificationVersions by aggregating old -> new (userid, roomid, version)
- just iter events constantly compact
- so should just be merging 2 -> 1 constantly, unless falls behind
- note: in future this worker will also actually turn each (unaggregated) notification into an actual notification for each of the users configured push targets

Explore the codebase and come up with a plan to implement the above changes.

... implemented, third prompt

Let's extend notification counts to handle threads. We need to:

- add ThreadID to types.Notifications (already done)
- in eventsend.go, we:
- move the notification generation into a new read txn, just before each write txn
- include threadID in generated notifications, this is:
- "" if event has no relation
- $event_id of thread root (found by walking thread relations of m.thread type until no more)
- we need two ways to sum notifications:
- the current one is fine for non-threading clients
- new sum by threadID version
- update sync
- add SyncOption to enable threaded notification counts
- when set, sum by threadID and update sync response accordingly

Explore the codebase and come up with a plan to implement the above changes.
26 changes: 26 additions & 0 deletions .claude/prompts/02-push-rules.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
Add push rules to the accounts database

- store Matrix push rules in a new directory in the accounts database, keys:
- userPushRules (userID, groupName, kind, ruleID) -> partial mautrix.PushRule
- kind is one of: override, underride, sender, room, content
- userPushVersions (userID) -> versionstamp of last written rule
- new methods:
- AccountsDatabase.GetRulesForUser
- AccountsDatabase.GetRuleForUser
- AccountsDatabase.PutRuleForUser
- AccountsDatabase.DeleteRuleForUser
- sync must return all the users push rules if the userPushVersion > the sync token as m.push_rules account data event

Explore the codebase and come up with a plan to implement the above changes.

... implemented, second prompt:

Now we need to evaluate the push rules during event sending.

- add databases.SendLocalEvents which calls rooms.SendLocalEvents
- modify rooms.SendLocalEvents to take userid -> pushrules map
- databases.SendLocalEvents fetches local users in room -> makes the map
- rooms.SendLocalEvents then uses push rules for evaluation
- same for databases.SendFederatedEvents -> rooms.SendFederatedEvents

Explore the codebase and come up with a plan to implement.
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ ed25519-*
*.pem
babbleserv
external/
.claude/
.claude/settings.local.json
3 changes: 3 additions & 0 deletions docker/complement-tests.list
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ TestKeysQueryWithDeviceIDAsObjectFails
TestNotPresentUserCannotBanOthers
TestPresence
TestPresenceSyncDifferentRooms
TestPushRuleCacheHealth
TestPushSync
TestRoomAlias
TestRoomCreate
TestRoomReceipts
Expand All @@ -21,6 +23,7 @@ TestServerCapabilities
TestSync
TestSyncFilter
TestSyncOmitsStateChangeOnFilteredEvents
TestThreadedReceipts
TestToDeviceMessages
TestUploadKey
TestUploadKeyIdempotency
Expand Down
38 changes: 22 additions & 16 deletions docs/data-flows.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,17 @@ Some high level flow charts describing how routes, databases and workers interac
## Room Events

```
┌─────────────────────────────┐ ┌──────────────────────────┐
│ │ │ │
│ CompactNotificationIterator │ │ PushNotificationIterator │
│ │ │ │
└─────────────────────────────┘ └──────────────────────────┘

┌───────────────────┐ ┌──────────────────────────┐
│ │ │ │
┌─────────────►│ FederationRoutes ├────►│ RoomsDatabase │◄────────────────┐
│ │ │ │ - SendLocalEvents │ │
│ └───────────────────┘ │ - SendFederatedEvents │ send events
│ └───────────────────┘ │ - SendFederatedEvents │ send events
│ │ │ │
│ └───────────┬──────────────┘ │
│ │ │
Expand All @@ -23,7 +29,7 @@ Federation Transaction PDUs │ │
│ │ └────────────────────┘
Federation outgoing events ◄────────────────────┤ FederationSender │
│ (per server) │
└────────────────────────┘
└────────────────────────┘
```

## Key & device management (user xs keys, device list updates)
Expand All @@ -37,20 +43,20 @@ Federation outgoing events ◄────────────────
│ │ user send events
┌────────┼──────────┐ ┌───────────────────┐ │ │
│ │ │ │ │ │
┌─────────────►│ FederationRoutes ├─►│ AccountsDatabase │◄───┼───────────────┐
│ │ │ │ │ │
│ └───────────────────┘ └──────┬────────────┘ │ user upload keys │
│ │ member events
│ │ │
│ │device change │
│ │ │
Federation Transaction EDUs │ │
│ │
┌───────────────────────┐ ┌─────▼──────────┐ ┌──┴────────────────┐
│ │ │ │ │
│ DeviceChangeIterator │ │ EventsIterator │ │ ClientRoutes │
│ │ │ │ │
Federation outgoing └───────────────────────┘ └───────────────┘ └────────────────────┘
┌─────────────►│ FederationRoutes ├─►│ AccountsDatabase │◄───┼──────────────────
│ │ │ │ │ │
│ └───────────────────┘ └──────┬────────────┘ │ user upload keys │
│ │ member events
│ │ │
│ │device change │
│ │ │
Federation Transaction EDUs │ │
│ │
┌───────────────────────┐ ┌──────────▼─────────────┐ ┌──┴────────────────┐
│ │ │ │ │
│ DeviceChangeIterator │ │DeviceJoinEventIterator │ │ ClientRoutes │
│ │ │ │ │
Federation outgoing └───────────────────────┘ └─────┬──────────────────┘ └────────────────────┘
- m.device_list_update │ │ ▲
- m.signing_key_update │ to-device │ │
▲ │ │ │
Expand Down
16 changes: 8 additions & 8 deletions docs/matrix-spec-compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ This document explores Babbleserv's compatability (or not) with the Matrix speci

These seem incredibly expensive to calculate for little benefit - clients must still implement all of their own aggregation logic because servers cannot guarantee their own aggregations are correct [citation needed]. So what's the point.

Note: backfilling still presents an issue here, but the `/reations` and threads APIs are supported and are more suitable for gathering this information.
Note: backfilling still presents an issue here, but the `/reations` and threads APIs _will be_ supported and are more suitable for gathering this information.

- see: [MSC2675 limitations](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/2675-aggregations-server.md#limitations), also see [MSC2677 (reactions) explicitly states server should NOT aggregate](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/2677-reactions.md#server-side-aggregation-of-mannotation-relationships), despite MSC2575 recommending this exact thing
- note that this also means edits are not applied by the server, clients should (and do) handle these appropriately - from the server perspective events are immutable unless redacted
Expand Down Expand Up @@ -40,20 +40,20 @@ Cheap alternative:

## Linearized Matrix

See [MSC3995](https://github.com/matrix-org/matrix-spec-proposals/pull/3995) - Babbleserv's data model means that within the local database state is always resolved before storage. There may be multiple dangling events in a room but the current state is always a resolved state in those cases. As such in many ways Babbleserv is similar to linearized Matrix hub servers. Events will be synced in version order, always. The `prev_events` are only relevant when ingesting events over Federation.
See [MSC3995](https://github.com/matrix-org/matrix-spec-proposals/pull/3995) - Babbleserv's data model means that within the local database state is always resolved before storage. There may be multiple dangling events in a room but the current state is always a resolved state in those cases. As such in many ways Babbleserv is similar to linearized Matrix hub servers. Events will be synced in version order, always.

## No Reactions in Relations API

The `/relations` API will not return `m.annotation` evens unless the `rel_type` is explicitly specified (and only `m.annotation` events are returned).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Typo: "evens" should be "events".

-The `/relations` API will not return `m.annotation` evens unless the `rel_type` is explicitly specified (and only `m.annotation` events are returned).
+The `/relations` API will not return `m.annotation` events unless the `rel_type` is explicitly specified (and only `m.annotation` events are returned).
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
The `/relations` API will not return `m.annotation` evens unless the `rel_type` is explicitly specified (and only `m.annotation` events are returned).
The `/relations` API will not return `m.annotation` events unless the `rel_type` is explicitly specified (and only `m.annotation` events are returned).
🧰 Tools
🪛 LanguageTool

[grammar] ~47-~47: Ensure spelling is correct
Context: ...onsAPI will not returnm.annotationevens unless therel_type` is explicitly spe...

(QB_NEW_EN_ORTHOGRAPHY_ERROR_IDS_1)

🤖 Prompt for AI Agents
In `@docs/matrix-spec-compatibility.md` at line 47, Fix the typo in the
documentation sentence for the /relations API: change "evens" to "events" in the
sentence that currently reads "The `/relations` API will not return
`m.annotation` evens unless the `rel_type` is explicitly specified (and only
`m.annotation` events are returned)." Ensure the corrected text reads "events"
and keep the rest of the wording unchanged, referencing the `/relations` API and
`m.annotation` mention.


## Push Rules
## Profile Updates and Device List Changes are Asynchronous

Not implemented.
Request to update/change will return before the changes are applied. Does this even deviate from the spec?

## Profile Updates are NOT Considered Room State
## Push Rules/Notifications/Notification Counts

Deviates from the spec. Synthetic events used.
Not implemented yet.

## Profile Updates and Device List Changes are Asynchronous
## Device last_seen and last_seen_ip aren't populated

Request to update/change will return before the changes are applied. Does this even deviate from the spec?
Not implemented yet.
3 changes: 0 additions & 3 deletions internal/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,12 @@ type BabbleConfig struct {
SigningKeyRefreshInterval time.Duration `yaml:"signingKeyRefreshInterval"`

Rooms struct {
Enabled bool `yaml:"enabled"`
Database databaseConfig `yaml:"database"`
Notifier NotifierConfig `yaml:"notifier"`
DefaultVersion string `yaml:"defaultVersion"`
} `yaml:"rooms"`

Accounts struct {
Enabled bool `yaml:"enabled"`
Database databaseConfig `yaml:"database"`
Notifier NotifierConfig `yaml:"notifier"`

Expand All @@ -59,7 +57,6 @@ type BabbleConfig struct {
} `yaml:"accounts"`

Transient struct {
Enabled bool `yaml:"enabled"`
Database databaseConfig `yaml:"database"`
Notifier NotifierConfig `yaml:"notifier"`

Expand Down
3 changes: 3 additions & 0 deletions internal/databases/accounts/accounts.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import (
"github.com/beeper/babbleserv/internal/config"
"github.com/beeper/babbleserv/internal/databases/accounts/accountdata"
"github.com/beeper/babbleserv/internal/databases/accounts/devices"
"github.com/beeper/babbleserv/internal/databases/accounts/pushrules"
"github.com/beeper/babbleserv/internal/databases/accounts/tokens"
"github.com/beeper/babbleserv/internal/databases/accounts/users"
"github.com/beeper/babbleserv/internal/notifier"
Expand All @@ -30,6 +31,7 @@ type AccountsDatabase struct {
tokens *tokens.TokensDirectory
devices *devices.DevicesDirectory
accountdata *accountdata.AccountDataDirectory
pushrules *pushrules.PushRulesDirectory
}

func NewAccountsDatabase(
Expand Down Expand Up @@ -69,6 +71,7 @@ func NewAccountsDatabase(
tokens: tokens.NewTokensDirectory(log, db, accountsDir),
devices: devices.NewDevicesDirectory(log, db, accountsDir),
accountdata: accountdata.NewAccountDataDirectory(log, db, accountsDir),
pushrules: pushrules.NewPushRulesDirectory(log, db, accountsDir),
}
}

Expand Down
2 changes: 1 addition & 1 deletion internal/databases/accounts/devices/devices.go
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ func (d *DevicesDirectory) TxnGetDevice(txn fdb.ReadTransaction, userID id.UserI
}

func (d *DevicesDirectory) TxnStoreDevice(txn fdb.Transaction, userID id.UserID, device *types.Device) {
txn.Set(d.keyForDevice(userID, device.ID), device.ToMsgpack())
txn.Set(d.keyForDevice(userID, device.ID), device.ToBytes())
}

func (d *DevicesDirectory) TxnGetOrCreateDevice(txn fdb.Transaction, userID id.UserID, deviceID id.DeviceID, initialDisplayName string) (*types.Device, error) {
Expand Down
64 changes: 64 additions & 0 deletions internal/databases/accounts/pushrules.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
package accounts

import (
"context"

"github.com/apple/foundationdb/bindings/go/src/fdb"
"github.com/apple/foundationdb/bindings/go/src/fdb/tuple"
"maunium.net/go/mautrix/id"
"maunium.net/go/mautrix/pushrules"

"github.com/beeper/babbleserv/internal/notifier"
"github.com/beeper/babbleserv/internal/types"
"github.com/beeper/babbleserv/internal/util"
)

func (a *AccountsDatabase) GetPushRulesForUser(ctx context.Context, userID id.UserID) (*pushrules.PushRuleset, error) {
return util.DoReadTransaction(ctx, a.db, func(txn fdb.ReadTransaction) (*pushrules.PushRuleset, error) {
return a.pushrules.TxnGetRulesForUser(txn, userID)
})
}

func (a *AccountsDatabase) GetPushRulesForUserByKind(ctx context.Context, userID id.UserID, kind pushrules.PushRuleType) ([]*pushrules.PushRule, error) {
return util.DoReadTransaction(ctx, a.db, func(txn fdb.ReadTransaction) ([]*pushrules.PushRule, error) {
return a.pushrules.TxnGetRulesForUserByKind(txn, userID, kind)
})
}

func (a *AccountsDatabase) GetPushRuleForUser(ctx context.Context, userID id.UserID, kind pushrules.PushRuleType, ruleID string) (*pushrules.PushRule, error) {
return util.DoReadTransaction(ctx, a.db, func(txn fdb.ReadTransaction) (*pushrules.PushRule, error) {
return a.pushrules.TxnGetRuleForUser(txn, userID, kind, ruleID)
})
}

func (a *AccountsDatabase) PutPushRuleForUser(ctx context.Context, userID id.UserID, kind pushrules.PushRuleType, ruleID string, rule *types.StoredPushRule) error {
_, err := util.DoWriteTransactionWithVersion(ctx, a.db, func(txn fdb.Transaction) (types.Nil, error) {
a.pushrules.TxnPutRuleForUser(txn, userID, kind, ruleID, rule)
return nil, nil
})
if err == nil {
a.notifier.SendChange(notifier.Change{
UserIDs: []id.UserID{userID},
})
}
return err
}

func (a *AccountsDatabase) DeletePushRuleForUser(ctx context.Context, userID id.UserID, kind pushrules.PushRuleType, ruleID string) error {
_, err := util.DoWriteTransactionWithVersion(ctx, a.db, func(txn fdb.Transaction) (types.Nil, error) {
a.pushrules.TxnDeleteRuleForUser(txn, userID, kind, ruleID)
return nil, nil
})
if err == nil {
a.notifier.SendChange(notifier.Change{
UserIDs: []id.UserID{userID},
})
}
return err
}

func (a *AccountsDatabase) GetUserPushRulesVersion(ctx context.Context, userID id.UserID) (tuple.Versionstamp, error) {
return util.DoReadTransaction(ctx, a.db, func(txn fdb.ReadTransaction) (tuple.Versionstamp, error) {
return a.pushrules.TxnGetUserPushVersion(txn, userID), nil
})
}
Loading
Loading