Implement notification counts

.claude/prompts/01-notification-counts.md

@@ -0,0 +1,47 @@
+Add notification counting for events
+
+- spec states "The updated notification count from a new event MUST appear in the same /sync response as the event itself."
+- store as rooms.users.notificationVersions (userid, roomid, version) -> types.Notifications{notifs, highlights, ...}
+- store types.Notifications as msgpack with single letter keys
+- for every event send we must (SendLocalEvents, SendFederatedEvents):
+    - get local users in the room
+    - get all their push rules (stub these for now), before any write txn
+    - inside the txn for each event, eval each users rules, map eventsToUserNotifications[id.EventID]types.Notifications{}
+    - pass to txnStoreEvents, we apply notificationVersions (userid, roomid, eventVersion) -> types.Notifications{}
+- version is the event version (so can get back to the eventid)
+- on sync, just
+    - range notificationVersions (userid, roomid) => sum counts
+- on receipt just
+    - clearrange up to (userid, roomid, eventVersionFromReceipt)
+
+Explore the codebase and come up with a plan to implement the above changes.
+
+... implemented, second prompt:
+
+Now we need to implement an EventNotificationIterator to compact notificationVersions:
+
+- compact notificationVersions by aggregating old -> new (userid, roomid, version)
+- just iter events constantly compact
+    - so should just be merging 2 -> 1 constantly, unless falls behind
+- note: in future this worker will also actually turn each (unaggregated) notification into an actual notification for each of the users configured push targets
+
+Explore the codebase and come up with a plan to implement the above changes.
+
+... implemented, third prompt
+
+Let's extend notification counts to handle threads. We need to:
+
+- add ThreadID to types.Notifications (already done)
+- in eventsend.go, we:
+    - move the notification generation into a new read txn, just before each write txn
+    - include threadID in generated notifications, this is:
+        - "" if event has no relation
+        - $event_id of thread root (found by walking thread relations of m.thread type until no more)
+- we need two ways to sum notifications:
+    - the current one is fine for non-threading clients
+    - new sum by threadID version
+- update sync
+    - add SyncOption to enable threaded notification counts
+    - when set, sum by threadID and update sync response accordingly
+
+Explore the codebase and come up with a plan to implement the above changes.

.claude/prompts/02-push-rules.md

@@ -0,0 +1,55 @@
+Add push rules to the accounts database
+
+- store Matrix push rules in a new directory in the accounts database, keys:
+    - userPushRules (userID, groupName, kind, ruleID) -> partial mautrix.PushRule
+        - kind is one of: override, underride, sender, room, content
+    - userPushVersions (userID) -> versionstamp of last written rule
+- new methods:
+    - AccountsDatabase.GetRulesForUser
+    - AccountsDatabase.GetRuleForUser
+    - AccountsDatabase.PutRuleForUser
+    - AccountsDatabase.DeleteRuleForUser
+- sync must return all the users push rules if the userPushVersion > the sync token as m.push_rules account data event
+
+Explore the codebase and come up with a plan to implement the above changes.
+
+... implemented, second prompt:
+
+Now we need to evaluate the push rules during event sending.
+
+- add databases.SendLocalEvents which calls rooms.SendLocalEvents
+    - modify rooms.SendLocalEvents to take userid -> pushrules map
+    - databases.SendLocalEvents fetches local users in room -> makes the map
+    - rooms.SendLocalEvents then uses push rules for evaluation
+- same for databases.SendFederatedEvents -> rooms.SendFederatedEvents
+
+Explore the codebase and come up with a plan to implement.
+
+... implemented, second prompt:
+
+Now we need to implement Matrix pushers APIs:
+
+- store Matrix pushers (mautrix pushgateway.Pusher) for users in UsersDirectory
+    - userPushers subspace (userID, pushKey) -> pushgateway.Pusher
+    - methods:
+        - AccountsDatabase.GetPushersForUser
+        - AccountsDatabase.SetPusherForUser
+
+Explore the codebase with a few agents (databases, routes) and come up with a plan to implement.
+
+... implemented, second prompt:
+
+Finally, now that we've implemented the various push components, let's actually send some push notifications!
+
+- we're going to base this on the CompactNotificationIterator, which is currently disabled
+- to prevent the notifications keyspace growing indefinitely (UsersDirectory.notificationVersions), we add a configurable limit to the number of notifications per user/room to keep, this worker will handle deleting the oldest N to maintain the limit (this means read receipt accuracy over the most recent X events per room)
+- let's call it PushNotificationIterator, it now has two responsibilities:
+    - as events come in, send pushes as required
+    - remove old notification count keys
+- to implement this, for every event that comes in:
+    - fetch all the local users in the room
+    - for each user, fetch push notification keys up to (including) the event version
+        - if notification was generated for this event (ie userid/roomid/eventversion notification exists), fetch users pushers and send it to each in parallel
+    - delete oldest push notifications > the configurable limit
+
+Explore the codebase with a few agents (databases, routes) and come up with a plan to implement.

.gitignore

@@ -5,4 +5,5 @@ ed25519-*
 *.pem
 babbleserv
 external/
-.claude/
+.claude/settings.local.json
+*.frpc

docker/complement-tests.list

@@ -13,6 +13,8 @@ TestKeysQueryWithDeviceIDAsObjectFails
 TestNotPresentUserCannotBanOthers
 TestPresence
 TestPresenceSyncDifferentRooms
+TestPushRuleCacheHealth
+TestPushSync
 TestRoomAlias
 TestRoomCreate
 TestRoomReceipts
@@ -21,6 +23,7 @@ TestServerCapabilities
 TestSync
 TestSyncFilter
 TestSyncOmitsStateChangeOnFilteredEvents
+TestThreadedReceipts
 TestToDeviceMessages
 TestUploadKey
 TestUploadKeyIdempotency

docs/data-flows.md

@@ -5,11 +5,17 @@ Some high level flow charts describing how routes, databases and workers interac
 ## Room Events
 
 ```
+                            ┌─────────────────────────────┐  ┌──────────────────────────┐               
+                            │                             │  │                          │               
+                            │ CompactNotificationIterator │  │ PushNotificationIterator │               
+                            │                             │  │                          │               
+                            └─────────────────────────────┘  └──────────────────────────┘               
+
                      ┌───────────────────┐     ┌──────────────────────────┐                             
                      │                   │     │                          │                             
       ┌─────────────►│ FederationRoutes  ├────►│  RoomsDatabase           │◄────────────────┐           
       │              │                   │     │   - SendLocalEvents      │                 │           
-      │              └───────────────────┘     │   - SendFederatedEvents  │             send events        
+      │              └───────────────────┘     │   - SendFederatedEvents  │             send events     
       │                                        │                          │                 │           
       │                                        └───────────┬──────────────┘                 │           
       │                                                    │                                │           
@@ -23,7 +29,7 @@ Federation Transaction PDUs                     │                        │
                                                 │                        │        └────────────────────┘
 Federation outgoing events ◄────────────────────┤    FederationSender    │                              
                                                 │      (per server)      │                              
-                                                └────────────────────────┘                              
+                                                └────────────────────────┘                                                 
 ```
 
 ## Key & device management (user xs keys, device list updates)
@@ -37,20 +43,20 @@ Federation outgoing events ◄────────────────
                               │                                      │                 user send events 
                      ┌────────┼──────────┐  ┌───────────────────┐    │                             │    
                      │                   │  │                   │    │                             │    
-      ┌─────────────►│ FederationRoutes  ├─►│ AccountsDatabase  │◄───┼───────────────┐             │    
-      │              │                   │  │                   │    │               │             │    
-      │              └───────────────────┘  └──────┬────────────┘    │           user upload keys  │    
-      │                                            │           member events         │             │    
-      │                                            │                 │               │             │    
-      │                                            │device change    │               │             │    
-      │                                            │                 │               │             │    
-Federation Transaction EDUs                        │                 │               │             │    
-                                                   │                 │               │             │    
-                                    ┌──────────────▼─────────┐ ┌─────▼──────────┐ ┌──┴─────────────┴───┐
-                                    │                        │ │                │ │                    │
-                                    │  DeviceChangeIterator  │ │ EventsIterator │ │    ClientRoutes    │
-                                    │                        │ │                │ │                    │
-Federation outgoing                 └─────────────┬──────────┘ └┬───────────────┘ └────────────────────┘
+      ┌─────────────►│ FederationRoutes  ├─►│ AccountsDatabase  │◄───┼──────────────────┐          │   
+      │              │                   │  │                   │    │                  │          │   
+      │              └───────────────────┘  └──────┬────────────┘    │           user upload keys  │   
+      │                                            │           member events            │          │     
+      │                                            │                 │                  │          │     
+      │                                            │device change    │                  │          │      
+      │                                            │                 │                  │          │      
+Federation Transaction EDUs                        │                 │                  │          │  
+                                                   │                 │                  │          │      
+                               ┌───────────────────▼────┐ ┌──────────▼─────────────┐ ┌──┴──────────┴──────┐
+                               │                        │ │                        │ │                    │
+                               │  DeviceChangeIterator  │ │DeviceJoinEventIterator │ │    ClientRoutes    │
+                               │                        │ │                        │ │                    │
+Federation outgoing            └──────────────────┬─────┘ └─────┬──────────────────┘ └────────────────────┘
     - m.device_list_update                        │             │                           ▲           
     - m.signing_key_update                        │  to-device  │                           │           
       ▲                                           │             │                           │           

docs/matrix-spec-compatibility.md

@@ -6,7 +6,7 @@ This document explores Babbleserv's compatability (or not) with the Matrix speci
 
 These seem incredibly expensive to calculate for little benefit - clients must still implement all of their own aggregation logic because servers cannot guarantee their own aggregations are correct [citation needed]. So what's the point.
 
-Note: backfilling still presents an issue here, but the `/reations` and threads APIs are supported and are more suitable for gathering this information.
+Note: backfilling still presents an issue here, but the `/reations` and threads APIs _will be_ supported and are more suitable for gathering this information.
 
 - see: [MSC2675 limitations](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/2675-aggregations-server.md#limitations), also see [MSC2677 (reactions) explicitly states server should NOT aggregate](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/2677-reactions.md#server-side-aggregation-of-mannotation-relationships), despite MSC2575 recommending this exact thing
 - note that this also means edits are not applied by the server, clients should (and do) handle these appropriately - from the server perspective events are immutable unless redacted
@@ -40,20 +40,20 @@ Cheap alternative:
 
 ## Linearized Matrix
 
-See [MSC3995](https://github.com/matrix-org/matrix-spec-proposals/pull/3995) - Babbleserv's data model means that within the local database state is always resolved before storage. There may be multiple dangling events in a room but the current state is always a resolved state in those cases. As such in many ways Babbleserv is similar to linearized Matrix hub servers. Events will be synced in version order, always. The `prev_events` are only relevant when ingesting events over Federation.
+See [MSC3995](https://github.com/matrix-org/matrix-spec-proposals/pull/3995) - Babbleserv's data model means that within the local database state is always resolved before storage. There may be multiple dangling events in a room but the current state is always a resolved state in those cases. As such in many ways Babbleserv is similar to linearized Matrix hub servers. Events will be synced in version order, always.
 
 ## No Reactions in Relations API
 
 The `/relations` API will not return `m.annotation` evens unless the `rel_type` is explicitly specified (and only `m.annotation` events are returned).
 
-## Push Rules
+## Profile Updates and Device List Changes are Asynchronous
 
-Not implemented.
+Request to update/change will return before the changes are applied. Does this even deviate from the spec?
 
-## Profile Updates are NOT Considered Room State
+## Push Rules/Notifications/Notification Counts
 
-Deviates from the spec. Synthetic events used.
+Not implemented yet.
 
-## Profile Updates and Device List Changes are Asynchronous
+## Device last_seen and last_seen_ip aren't populated
 
-Request to update/change will return before the changes are applied. Does this even deviate from the spec?
+Not implemented yet.

docs/project-structure.md

@@ -21,15 +21,22 @@ Ingesting federated events is a good example of this - all the network fetching
 
 ## Module Layout
 
+Babbleserv is roughly divided in three:
+
+- databases talk to FoundationDB, no access to network/federation, implements Matrix spec on top of the databases (event auth, state res) using FDB transactions
+- routes implement the client/federation Matrix APIs, pre-fetch anything before passing to relevant database call
+- workers handle asynchronous tasks after database changes (federation outgoing, profile updates, presence, push notifications)
+
 ### `internal/databases/*/`
 
 - each represents a FDB cluster containing a logical group of sub-databases
 - top level database transactions called by routes
 - call through to the domain specific directories nested modules
+- each database lives under a key prefix
 
 #### `internal/databases/*/*/`
 
-- individual database "directories" (FDB thing)
+- individual database "directories" (key prefix)
 - group together common key prefix operations (ie events, users)
 - not exported/available outside of database
 

internal/config/config.go

@@ -38,14 +38,24 @@ type BabbleConfig struct {
 	SigningKeyRefreshInterval time.Duration        `yaml:"signingKeyRefreshInterval"`
 
 	Rooms struct {
-		Enabled        bool           `yaml:"enabled"`
 		Database       databaseConfig `yaml:"database"`
 		Notifier       NotifierConfig `yaml:"notifier"`
 		DefaultVersion string         `yaml:"defaultVersion"`
+
+		// Max notifications per user/room to keep count of, is not accurately applied, ie counts
+		// may go over this before being trimmed back down after the timeout below or sufficient
+		// traffic in the room.
+		// Default: 100
+		MaxNotificationsPerUserRoom int `yaml:"maxNotificationsPerUserRoom"`
+
+		// Timeout after which we compact a rooms notifications even if less than max notifications
+		// have been sent. This accounts for process restarts - the notification compactor stores
+		// events sent per room in memory only.
+		// Default: 3h
+		CompactRoomNotificationsTimeout time.Duration `yaml:"compactRoomNotificationsTimeout"`
 	} `yaml:"rooms"`
 
 	Accounts struct {
-		Enabled  bool           `yaml:"enabled"`
 		Database databaseConfig `yaml:"database"`
 		Notifier NotifierConfig `yaml:"notifier"`
 
@@ -59,7 +69,6 @@ type BabbleConfig struct {
 	} `yaml:"accounts"`
 
 	Transient struct {
-		Enabled  bool           `yaml:"enabled"`
 		Database databaseConfig `yaml:"database"`
 		Notifier NotifierConfig `yaml:"notifier"`
 
@@ -85,9 +94,6 @@ type BabbleConfig struct {
 		Servers []serverConfig `yaml:"servers"`
 	} `yaml:"routes"`
 
-	Workers struct {
-	} `yaml:"workers"`
-
 	Federation struct {
 		MaxFetchMissingEvents       int  `yaml:"maxFetchMissingEvents"`
 		FetchProfileForMemberEvents bool `yaml:"fetchProfileForMemberEvents"`
@@ -159,6 +165,13 @@ func NewBabbleConfig(filename string, commitHash string) BabbleConfig {
 		cfg.Transient.PresenceTimeoutCheckInterval = time.Minute
 	}
 
+	if cfg.Rooms.MaxNotificationsPerUserRoom == 0 {
+		cfg.Rooms.MaxNotificationsPerUserRoom = 100
+	}
+	if cfg.Rooms.CompactRoomNotificationsTimeout == 0 {
+		cfg.Rooms.CompactRoomNotificationsTimeout = 3 * time.Hour
+	}
+
 	return cfg
 }