From b6caabe0b0d3e347674efa87e27008210d0c95ed Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Tue, 13 Jun 2023 18:37:11 +0200 Subject: [PATCH 01/20] New proposal: Add a new predefined override push rule (`.m.rule.encrypted_event`) to be able to push all encrypted events --- ...encrypted-events-except-for-muted rooms.md | 104 ++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100644 proposals/40xx-push-all-encrypted-events-except-for-muted rooms.md diff --git a/proposals/40xx-push-all-encrypted-events-except-for-muted rooms.md b/proposals/40xx-push-all-encrypted-events-except-for-muted rooms.md new file mode 100644 index 00000000000..7daf65a2965 --- /dev/null +++ b/proposals/40xx-push-all-encrypted-events-except-for-muted rooms.md @@ -0,0 +1,104 @@ +# MSC40xx: Push all encrypted events except for muted rooms + +About notifications handling, it is currently not possible for mobile clients which get push notifications to properly implement a mentions-and-keywords-only room. + +Like it was already described in [MSC3996: Encrypted mentions-only rooms](https://github.com/matrix-org/matrix-spec-proposals/pull/3996): + +Currently, every new event in an encrypted room might be pushed to mobile clients due to the default **underride** rule [`.m.rule.encrypted`](https://spec.matrix.org/v1.7/client-server-api/#default-underride-rules). + +However, a room can be configured to be mentions-and-keywords-only by creating a [room-specific push rule](https://spec.matrix.org/v1.7/client-server-api/#push-rules) +with the `id` property set to the room ID & `actions` set to do nothing.[^1] Since this is evaluated *before* the `.m.rule.encrypted` rule (almost) **no events get pushed for a mentions-and-keywords-only room**. + +Additionally, a room can be configured to be muted by creating the **override** [push rule](https://spec.matrix.org/v1.7/client-server-api/#push-rules) which matches the room ID & has `actions` set to do nothing, e.g.: + +```json +{ + "rule_id": "!abcdef:example.com", + "conditions": [ + { + "key": "room_id", + "kind": "event_match", + "pattern": "!abcdef:example.com" + } + ], + "default": false, + "enabled": true, + "actions": [] +} +``` + +## Proposal + +A new predefined **override** push rule (`.m.rule.encrypted_event`) is added just after the `.m.rule.master` push rule to be able to push all encrypted events: + +```json +{ + "rule_id": ".m.rule.encrypted_event", + "default": true, + "enabled": true, + "conditions": [ + { + "kind": "event_match", + "key": "type", + "pattern": "m.room.encrypted" + } + ], + "actions": ["notify"] +} +``` + +Note: The “user-defined rules” are evaluated with a higher priority than “server-default rules”, except for the `.m.rule.master` rule which has always a higher priority than any other rule (see [here](https://spec.matrix.org/v1.7/client-server-api/#predefined-rules)). So all the **override** push rules created to mute rooms will be evaluated before this new one. The new rule will be evaluated before all the other “server-default rules”. This new rule deprecates the existing underride rule [`.m.rule.encrypted`](https://spec.matrix.org/v1.7/client-server-api/#default-underride-rules), which will become useless. + +### Receiving notifications + +When this push rule matches then homeserver would push the event to all registered clients, similar to how other rules work. Clients would [decrypt the event and re-run push rules](https://spec.matrix.org/v1.7/client-server-api/#receiving-notifications) as normal, but they will consider at the same time the original event is encrypted. The following algorithm has to be implemented client side as soon as it received a push for an encrypted event **and** `.m.rule.encrypted_event` is present and enabled in the account push rules set: + +1. If the decryption failed, the new `.m.rule.encrypted_event` rule will match on client side too. There is no reason to notify the end user. The push is discarded [^2]. The client decrements locally the number of unread notifications received from the homeserver for this room. We decided to discard here the push because this use case should not happen except if the event was not encrypted for the current session, +or if we are in front of an unexpected "Unable To Decrypt" (UTD) issue. +2. If the decryption succeeded, there are 3 cases: +- the decrypted event resulted in a highlight notification -> the client increments locally the highlight notifications count for this room +- the decrypted event results in a notification -> no change is required on the notifications count +- the decrypted event results in no notification -> the client decrements locally the number of unread notifications for this room + +The overall tradeoff is that clients will receive extra pushes some of the time. + +### Listening notifications + +This new push rule will impact the existing endpoint [`GET /_matrix/client/v3/notifications`](https://spec.matrix.org/v1.7/client-server-api/#get_matrixclientv3notifications). + +When this rule will be present and enabled in the account push rules set, the clients which retrieve the list of events that it has been notified about with this endpoint, will receive most of the encrypted events (except for muted rooms). They will be able to decrypt and re-run push rules locally. This should fix the notifications panel observed in some web clients where currently the notifications of encrypted events are missing. + +### Polling + +The clients without push services (Web or F-Droid for example) will have to take into account this new push rule (if it is enabled) during their polling mechanism. Indeed they should update the unread notifications count by following the same algorithm as above. This algorithm should replace the potential current one. + +## Potential issues + +The clients without push may observe encrypted rooms with a high number of unread notifications in case of gappy sync. The workaround would be to not display the count for the room until the gap is completed or the count is reset (mark all as read). + +## Alternatives + +[MSC3996: Encrypted mentions-only rooms](https://github.com/matrix-org/matrix-spec-proposals/pull/3996): the suggested top-level property `m.has_mentions` may be an option to reduce the volume of pushes. But we would not be able to support notifications on keywords then. + +## Security considerations + +None + +## Future extensions + +None + +## Unstable prefix + +During development the new push rule shall use `org.matrix.msc40xx.encrypted_event` instead of `.m.rule.encrypted_event`. + +## Dependencies + +None + + +[^1]: Via either an explicit `"dont_notify"` action or an empty array. These are +equivalent, see [issue #643](https://github.com/matrix-org/matrix-spec/issues/643). + +[^2]: In the past it was not possible to discard notifications on iOS: if a push +notification was received it *had to be displayed*. This is [no longer the case](https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_usernotifications_filtering). From 910d33596ae2e5b0305cbecfb309614265330327 Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Tue, 13 Jun 2023 18:45:18 +0200 Subject: [PATCH 02/20] Update MSC number --- ...=> 4028-push-all-encrypted-events-except-for-muted-rooms.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename proposals/{40xx-push-all-encrypted-events-except-for-muted rooms.md => 4028-push-all-encrypted-events-except-for-muted-rooms.md} (99%) diff --git a/proposals/40xx-push-all-encrypted-events-except-for-muted rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md similarity index 99% rename from proposals/40xx-push-all-encrypted-events-except-for-muted rooms.md rename to proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 7daf65a2965..ccd28e5fe36 100644 --- a/proposals/40xx-push-all-encrypted-events-except-for-muted rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -1,4 +1,4 @@ -# MSC40xx: Push all encrypted events except for muted rooms +# MSC4028: Push all encrypted events except for muted rooms About notifications handling, it is currently not possible for mobile clients which get push notifications to properly implement a mentions-and-keywords-only room. From 1a04b423cbe04ce07ed54ae66286d93e58450a3a Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Tue, 13 Jun 2023 19:06:22 +0200 Subject: [PATCH 03/20] Add an alternative --- .../4028-push-all-encrypted-events-except-for-muted-rooms.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index ccd28e5fe36..0ee3cad6c61 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -80,6 +80,8 @@ The clients without push may observe encrypted rooms with a high number of unrea [MSC3996: Encrypted mentions-only rooms](https://github.com/matrix-org/matrix-spec-proposals/pull/3996): the suggested top-level property `m.has_mentions` may be an option to reduce the volume of pushes. But we would not be able to support notifications on keywords then. +Another alternative would be to define a new [`action`](https://spec.matrix.org/v1.7/client-server-api/#actions): `push_without_notify` or `silently_notify` in order to push all encrypted events without incrementing the notifications count. The client implementation to handle these counts would be then less complex. But this would need to implement more changes in Synapse. The current proposal is just to add a new push rules without changing the rules handling server side. This alternative should be discussed as soon as possible. + ## Security considerations None From 4e02acd6119b29f3a2a64402cf93181a39fbd5a1 Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Tue, 20 Jun 2023 19:20:46 +0200 Subject: [PATCH 04/20] Update the expected changes client side - how to manage the received pushes - how to handle the unread notifications counts in encrypted rooms --- ...encrypted-events-except-for-muted-rooms.md | 45 +++++++++++++------ 1 file changed, 31 insertions(+), 14 deletions(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 0ee3cad6c61..51c74e2b7da 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -51,30 +51,47 @@ Note: The “user-defined rules” are evaluated with a higher priority than “ ### Receiving notifications -When this push rule matches then homeserver would push the event to all registered clients, similar to how other rules work. Clients would [decrypt the event and re-run push rules](https://spec.matrix.org/v1.7/client-server-api/#receiving-notifications) as normal, but they will consider at the same time the original event is encrypted. The following algorithm has to be implemented client side as soon as it received a push for an encrypted event **and** `.m.rule.encrypted_event` is present and enabled in the account push rules set: - -1. If the decryption failed, the new `.m.rule.encrypted_event` rule will match on client side too. There is no reason to notify the end user. The push is discarded [^2]. The client decrements locally the number of unread notifications received from the homeserver for this room. We decided to discard here the push because this use case should not happen except if the event was not encrypted for the current session, -or if we are in front of an unexpected "Unable To Decrypt" (UTD) issue. -2. If the decryption succeeded, there are 3 cases: -- the decrypted event resulted in a highlight notification -> the client increments locally the highlight notifications count for this room -- the decrypted event results in a notification -> no change is required on the notifications count -- the decrypted event results in no notification -> the client decrements locally the number of unread notifications for this room +When this push rule matches then homeserver would push the event to all registered clients, similar to how other rules work. Clients would [decrypt the event and re-run push rules](https://spec.matrix.org/v1.7/client-server-api/#receiving-notifications) as normal. If the decrypted event results in no notification, the push is discarded [^2]. If the decryption failed, the new `.m.rule.encrypted_event` rule will match on client side, but there is no reason to notify the end user, so the push is discarded [^2] too. Globally the client must discard the push when the event stays encrypted locally because this use case should not happen except if the event was not encrypted for the current session, or if we are in front of an unexpected "Unable To Decrypt" (UTD) issue. The overall tradeoff is that clients will receive extra pushes some of the time. +### Unread notification counts + +As soon as the rule `.m.rule.encrypted_event` is present and enabled in the account push rules set, the clients have to track unread notifications themselves in **encrypted rooms**. All the clients with or without push services will have to compute the unread notification count for each encrypted room during the client-server sync, by implementing the algorithm described below. + +This algorithm depends on the unread notification count `unread_notifications` received from the server for each joined room in the [sync response](https://spec.matrix.org/v1.7/client-server-api/#get_matrixclientv3sync). It has to be adapted to the potential unread thread notification counts `unread_thread_notifications` that we will ignore here to simplify the description. + +Let's consider we receive the following data for a joined encrypted rooms in the sync response: +- number of new encrypted events in the timeline: n +- unread notification count: k + +1. if k = 0 -> reset the local notification counts for this room (the unread notification count and the highlight count). +2. else if k <= n -> reset the local notification counts, then decrypt and run push rules on the k most recent events: + - if the decrypted event resulted in a highlight notification -> increment the local highlight notifications count and the local unread notification count. + - if the decrypted event results in a notification -> increment only the local unread notifications count. + - if the decrypted event results in no notification -> no change is required on the local notification counts. + - if the decryption failed -> increment the local unread notifications count. +3. else if k > n and no gap is mentioned in the sync response for this room -> update the local notification counts by taking into account the potential change of the read receipt position of the current user, then decrypt and run push rules on the n received events by applying the same rules as above. +4. else if k > n and a gap is mentioned in the sync response -> there is 2 options: +- flush the local room storage, this will reset the local notification counts, then decrypt and run push rules on the n received events by applying the same rules as above. In order to compute locally accurate notification counts, back paginate to retrieve (k-n) more encrypted events. + +or + +- update the local notification counts by taking into account the potential change of the read receipt position of the current user, then decrypt and run push rules on the n received events by applying the same rules as above. In order to compute locally accurate notification counts, back paginate to close the gap or at least retrieve (k-n) more encrypted events. + +About the case 4, the back pagination should be mandatory for mentions-and-keywords-only rooms to detect if there are some mentions in the gap. The client may decide to not trigger back pagination for rooms in "all messages" mode, but in that case the local unread notification count is unknown. No badge can be displayed at the application side, only an unread marker. + +Note: some unencrypted events (for example `m.room.redaction`) may be taken into account or not in this unreads count k. This depends on the push rules set. If you force "all messages" mode on a room by creating a room-specific push rule, some unencrypted events (like redaction events) will be included in this count. This is not the case if "all messages" mode is obtained by the combination of server predefined rules. The client have locally all the information (= the push rules) to handle this edge case during the implementation of the algorithm. + ### Listening notifications This new push rule will impact the existing endpoint [`GET /_matrix/client/v3/notifications`](https://spec.matrix.org/v1.7/client-server-api/#get_matrixclientv3notifications). When this rule will be present and enabled in the account push rules set, the clients which retrieve the list of events that it has been notified about with this endpoint, will receive most of the encrypted events (except for muted rooms). They will be able to decrypt and re-run push rules locally. This should fix the notifications panel observed in some web clients where currently the notifications of encrypted events are missing. -### Polling - -The clients without push services (Web or F-Droid for example) will have to take into account this new push rule (if it is enabled) during their polling mechanism. Indeed they should update the unread notifications count by following the same algorithm as above. This algorithm should replace the potential current one. - ## Potential issues -The clients without push may observe encrypted rooms with a high number of unread notifications in case of gappy sync. The workaround would be to not display the count for the room until the gap is completed or the count is reset (mark all as read). +Until a client is updated to support this MSC, it may display a high number of unread notifications for encrypted rooms. The first workaround would be to not display the count for the encrypted rooms when the rule `.m.rule.encrypted_event` is present and enabled in the account push rules set, until the suggested algorithm is implemented. ## Alternatives @@ -92,7 +109,7 @@ None ## Unstable prefix -During development the new push rule shall use `org.matrix.msc40xx.encrypted_event` instead of `.m.rule.encrypted_event`. +During development the new push rule shall use `org.matrix.msc4028.encrypted_event` instead of `.m.rule.encrypted_event`. ## Dependencies From 4c9266b17a62c755441d37f9ce77c0cf3b3a1304 Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Wed, 28 Jun 2023 10:10:37 +0200 Subject: [PATCH 05/20] Update "Potential issues" section --- ...-encrypted-events-except-for-muted-rooms.md | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 51c74e2b7da..15fd8cea43e 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -91,7 +91,23 @@ When this rule will be present and enabled in the account push rules set, the cl ## Potential issues -Until a client is updated to support this MSC, it may display a high number of unread notifications for encrypted rooms. The first workaround would be to not display the count for the encrypted rooms when the rule `.m.rule.encrypted_event` is present and enabled in the account push rules set, until the suggested algorithm is implemented. +### Client side + +#### Unexpected unread counts +Until a client is updated to support this MSC, it may display a high number of unread notifications for encrypted rooms. The first workaround would be to not display the count for the encrypted rooms when the rule `.m.rule.encrypted_event` is present and enabled in the account push rules set, until the suggested algorithm is implemented, at least for the mentions-and-keywords-only encrypted rooms. + +#### Battery life +The mobile client will wake up more frequently than currently. We will have to control any potential issue with battery life when we will implement this MSC. The impact on the battery life will be definitly lower than the one observed on clients without push service which poll for events in the background. The end user will be able to reduce the impact by muting some encrypted rooms. + +### Server side + +#### Event fetching +The clients with push services will request more frequently the server to retrieve the event contents. I'm mainly thinking about an encrypted room with a high number of members, each mobile client of these members will try to retrieve simultaneously the event content of each new encrypted event (pushed by the server). This scenario may happen today only when most of the members configure the room in "All messages" mode. + +This may be a potential performance issue if each client fetchs separately each pushed event. But this should not be the case, because the clients have to run a sync if they want to be able to receive potential new encryption keys through to_devices. So they will retrieve more than one event at the time. + +#### Push sending +Push is a huge bottleneck on the sending side. So calculating what to push and the unread counts (its the same process) is a big performance bottleneck currently when sending in large rooms. A bunch of optimisation has taken place to make that better, but its still quite expensive. We should plan an effort to limit potential issue server side here. ## Alternatives From 0bb7c52cfad08953a944ad342cc3ef3af1c59b22 Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Wed, 28 Jun 2023 11:17:47 +0200 Subject: [PATCH 06/20] fix typos --- ...4028-push-all-encrypted-events-except-for-muted-rooms.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 15fd8cea43e..e8370fe02b6 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -97,14 +97,14 @@ When this rule will be present and enabled in the account push rules set, the cl Until a client is updated to support this MSC, it may display a high number of unread notifications for encrypted rooms. The first workaround would be to not display the count for the encrypted rooms when the rule `.m.rule.encrypted_event` is present and enabled in the account push rules set, until the suggested algorithm is implemented, at least for the mentions-and-keywords-only encrypted rooms. #### Battery life -The mobile client will wake up more frequently than currently. We will have to control any potential issue with battery life when we will implement this MSC. The impact on the battery life will be definitly lower than the one observed on clients without push service which poll for events in the background. The end user will be able to reduce the impact by muting some encrypted rooms. +The mobile client will wake up more frequently than currently. We will have to control any potential issue with battery life when we will implement this MSC. The impact on the battery life will be definitely lower than the one observed on clients without push service which poll for events in the background. The end user will be able to reduce the impact by muting some encrypted rooms. ### Server side #### Event fetching The clients with push services will request more frequently the server to retrieve the event contents. I'm mainly thinking about an encrypted room with a high number of members, each mobile client of these members will try to retrieve simultaneously the event content of each new encrypted event (pushed by the server). This scenario may happen today only when most of the members configure the room in "All messages" mode. -This may be a potential performance issue if each client fetchs separately each pushed event. But this should not be the case, because the clients have to run a sync if they want to be able to receive potential new encryption keys through to_devices. So they will retrieve more than one event at the time. +This may be a potential performance issue if each client fetches separately each pushed event. But this should not be the case, because the clients have to run a sync if they want to be able to receive potential new encryption keys through to_devices. So they will retrieve more than one event at the time. #### Push sending Push is a huge bottleneck on the sending side. So calculating what to push and the unread counts (its the same process) is a big performance bottleneck currently when sending in large rooms. A bunch of optimisation has taken place to make that better, but its still quite expensive. We should plan an effort to limit potential issue server side here. @@ -125,7 +125,7 @@ None ## Unstable prefix -During development the new push rule shall use `org.matrix.msc4028.encrypted_event` instead of `.m.rule.encrypted_event`. +During development the new push rule shall use `org.matrix.msc4028.encrypted_event` instead of `.m.rule.encrypted_event`. Contrary to the stable push rule, this unstable one should not be enabled by default until the clients are ready to support a minimum this MSC. ## Dependencies From b0cd5e5f09f4f9b319126fee69c19231a6ea5ca9 Mon Sep 17 00:00:00 2001 From: giomfo Date: Wed, 9 Aug 2023 14:21:33 +0200 Subject: [PATCH 07/20] Update proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md typo Co-authored-by: Patrick Cloke --- .../4028-push-all-encrypted-events-except-for-muted-rooms.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index e8370fe02b6..8a556b082be 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -83,7 +83,7 @@ About the case 4, the back pagination should be mandatory for mentions-and-keywo Note: some unencrypted events (for example `m.room.redaction`) may be taken into account or not in this unreads count k. This depends on the push rules set. If you force "all messages" mode on a room by creating a room-specific push rule, some unencrypted events (like redaction events) will be included in this count. This is not the case if "all messages" mode is obtained by the combination of server predefined rules. The client have locally all the information (= the push rules) to handle this edge case during the implementation of the algorithm. -### Listening notifications +### Listing notifications This new push rule will impact the existing endpoint [`GET /_matrix/client/v3/notifications`](https://spec.matrix.org/v1.7/client-server-api/#get_matrixclientv3notifications). From 132ec4ad15520f16b208c04ee1d525643ed8ec8f Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Wed, 9 Aug 2023 16:06:10 +0200 Subject: [PATCH 08/20] update "unread notification counts" section intro --- ...4028-push-all-encrypted-events-except-for-muted-rooms.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 8a556b082be..4fbaa4b762f 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -57,9 +57,11 @@ The overall tradeoff is that clients will receive extra pushes some of the time. ### Unread notification counts -As soon as the rule `.m.rule.encrypted_event` is present and enabled in the account push rules set, the clients have to track unread notifications themselves in **encrypted rooms**. All the clients with or without push services will have to compute the unread notification count for each encrypted room during the client-server sync, by implementing the algorithm described below. +Since the encryption has been introduced, the actual unread notification counts have to be updated/computed at the client side for the **encrypted rooms**. Nothing has been specified for that, so each client implemented its own mechanism. We want to harmonize this here. -This algorithm depends on the unread notification count `unread_notifications` received from the server for each joined room in the [sync response](https://spec.matrix.org/v1.7/client-server-api/#get_matrixclientv3sync). It has to be adapted to the potential unread thread notification counts `unread_thread_notifications` that we will ignore here to simplify the description. +As soon as the rule `.m.rule.encrypted_event` is present and enabled in the account push rules set, all the clients with or without push services should compute the unread notification count for each encrypted room during the client-server sync, by implementing the algorithm described below. Its complexity is similar to the existing implementations. + +This algorithm depends on the unread notification count `unread_notifications` received from the server for each joined room in the [sync response](https://spec.matrix.org/v1.7/client-server-api/#get_matrixclientv3sync). It has to be adapted to the potential unread thread notification counts `unread_thread_notifications` that we will ignore here to ease the description. Let's consider we receive the following data for a joined encrypted rooms in the sync response: - number of new encrypted events in the timeline: n From 48fe8947826b95532086579b7d3ecdd142418d87 Mon Sep 17 00:00:00 2001 From: giomfo Date: Wed, 9 Aug 2023 16:18:25 +0200 Subject: [PATCH 09/20] Update proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md Co-authored-by: Stuart Mumford --- .../4028-push-all-encrypted-events-except-for-muted-rooms.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 4fbaa4b762f..2d7dd04efca 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -1,6 +1,6 @@ # MSC4028: Push all encrypted events except for muted rooms -About notifications handling, it is currently not possible for mobile clients which get push notifications to properly implement a mentions-and-keywords-only room. +About notifications handling, it is currently not possible for mobile clients which get push notifications to properly implement a mentions-and-keywords-only room when encryption is enabled. Like it was already described in [MSC3996: Encrypted mentions-only rooms](https://github.com/matrix-org/matrix-spec-proposals/pull/3996): From bac66b8c5dd7526d402f0edb8d1a21b5f0840741 Mon Sep 17 00:00:00 2001 From: giomfo Date: Wed, 27 Sep 2023 17:01:37 +0200 Subject: [PATCH 10/20] Update proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md Co-authored-by: Patrick Cloke --- .../4028-push-all-encrypted-events-except-for-muted-rooms.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 2d7dd04efca..defa29aca03 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -38,9 +38,9 @@ A new predefined **override** push rule (`.m.rule.encrypted_event`) is added jus "enabled": true, "conditions": [ { - "kind": "event_match", + "kind": "event_property_is", "key": "type", - "pattern": "m.room.encrypted" + "value": "m.room.encrypted" } ], "actions": ["notify"] From 2207b525d44d121e9be5e5ee113de19b51caffea Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Wed, 27 Sep 2023 18:59:03 +0200 Subject: [PATCH 11/20] Add a reference to MSC3013 Encrypted push --- .../4028-push-all-encrypted-events-except-for-muted-rooms.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index defa29aca03..b5ae3970f70 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -108,6 +108,8 @@ The clients with push services will request more frequently the server to retrie This may be a potential performance issue if each client fetches separately each pushed event. But this should not be the case, because the clients have to run a sync if they want to be able to receive potential new encryption keys through to_devices. So they will retrieve more than one event at the time. +BTW [MSC3013: Encrypted Push](https://github.com/matrix-org/matrix-spec-proposals/pull/3013) could potentially mitigate this. + #### Push sending Push is a huge bottleneck on the sending side. So calculating what to push and the unread counts (its the same process) is a big performance bottleneck currently when sending in large rooms. A bunch of optimisation has taken place to make that better, but its still quite expensive. We should plan an effort to limit potential issue server side here. From 443f07cccba5b9df01b7a1165f8060208dfe9633 Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Wed, 18 Oct 2023 14:58:33 +0200 Subject: [PATCH 12/20] Add a section on Email notifications --- ...sh-all-encrypted-events-except-for-muted-rooms.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index b5ae3970f70..1fbcd391394 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -91,6 +91,18 @@ This new push rule will impact the existing endpoint [`GET /_matrix/client/v3/no When this rule will be present and enabled in the account push rules set, the clients which retrieve the list of events that it has been notified about with this endpoint, will receive most of the encrypted events (except for muted rooms). They will be able to decrypt and re-run push rules locally. This should fix the notifications panel observed in some web clients where currently the notifications of encrypted events are missing. +### Email notifications + +A user may set up a pusher to receive emails with unread notifications (see the spec [here](https://spec.matrix.org/v1.8/client-server-api/#post_matrixclientv3pushersset) with `kind` = "email"). Note that it sends emails on a delay of ~10 minutes to give people time to see the notification and mark it as read. + +The number of this email notifications may increase when this new rule will be present and enabled in the account push rules set. Indeed the email notifications will then include the encrypted rooms for which the notifications are configured in mentions-and-keywords-only mode. These rooms are not supported for the moment in the email notifications, so the users may miss some mentions. + +Currently the existing email notifications are not really relevant in case of encrypted rooms. The users may receive emails with a long bunch of encrypted messages (only the room names and the sender names are in plain text). This will happen more frequently when this new push rule will be enabled. **We should just list the encrypted room names in the email notification when the pushed events stayed encrypted server side**. The users need at least to be notified by email when there is activity in these unmuted encrypted rooms without expecting more details (except unencrypted events). They should mute the room if they don't need this information. + +If the email notifications become too noisy because of this new rule, we would have to work on them in another MSC by considering eventually one of the following options: +1. Ignore in the email notifications the encrypted events which have been pushed only because the condition "event type == m.room.encrypted" was satisfied. The server is not able to send relevant email notification in that specific case. +2. Define a new action which is "push but don't notify", to exclude some pushed events in the email. + ## Potential issues ### Client side From 5f04a997c4e8720e9d88d8bf07759fb8c6ec56a6 Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Thu, 19 Oct 2023 13:30:41 +0200 Subject: [PATCH 13/20] Update "Email notifications" section --- ...ush-all-encrypted-events-except-for-muted-rooms.md | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 1fbcd391394..8b58d0de115 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -93,15 +93,14 @@ When this rule will be present and enabled in the account push rules set, the cl ### Email notifications -A user may set up a pusher to receive emails with unread notifications (see the spec [here](https://spec.matrix.org/v1.8/client-server-api/#post_matrixclientv3pushersset) with `kind` = "email"). Note that it sends emails on a delay of ~10 minutes to give people time to see the notification and mark it as read. +A user may set up a pusher to receive emails with unread notifications (see the spec [here](https://spec.matrix.org/v1.8/client-server-api/#post_matrixclientv3pushersset) with `kind` = "email"). Note that in the current Synapse implementation this pusher sends emails on a delay of ~10 minutes to give people time to see the notification and mark it as read. It looks like some other server implementations don't support email pushers though (see details in [this comment](https://github.com/matrix-org/matrix-spec-proposals/pull/4028#discussion_r1364373223)). -The number of this email notifications may increase when this new rule will be present and enabled in the account push rules set. Indeed the email notifications will then include the encrypted rooms for which the notifications are configured in mentions-and-keywords-only mode. These rooms are not supported for the moment in the email notifications, so the users may miss some mentions. +Currently the existing email notifications are not really relevant in case of encrypted rooms. The users may receive emails with a long bunch of encrypted messages when the room is configured in "all messages" notification mode. This feature should be reviewed again with a Product point of view. -Currently the existing email notifications are not really relevant in case of encrypted rooms. The users may receive emails with a long bunch of encrypted messages (only the room names and the sender names are in plain text). This will happen more frequently when this new push rule will be enabled. **We should just list the encrypted room names in the email notification when the pushed events stayed encrypted server side**. The users need at least to be notified by email when there is activity in these unmuted encrypted rooms without expecting more details (except unencrypted events). They should mute the room if they don't need this information. +In order to not disturb the existing email notifications mechanism, the server implementations which support it should ignore in the email notifications the events pushed because of the new push rule. +Otherwise the number of emails will increase by including the encrypted rooms for which the notifications are configured in mentions-and-keywords-only mode. These rooms are not handled for the moment in the email notifications. -If the email notifications become too noisy because of this new rule, we would have to work on them in another MSC by considering eventually one of the following options: -1. Ignore in the email notifications the encrypted events which have been pushed only because the condition "event type == m.room.encrypted" was satisfied. The server is not able to send relevant email notification in that specific case. -2. Define a new action which is "push but don't notify", to exclude some pushed events in the email. +As a first option, the server should ignore these pushed events by using the push rule Id: `.m.rule.encrypted_event` (`org.matrix.msc4028.encrypted_event`). We may consider to work later on a new MSC to introduce a new [action](https://spec.matrix.org/latest/client-server-api/#actions) _"push but don't notify"_ or a new tweak _"dont_email"_ to exclude some pushed events from the email. ## Potential issues From b55877727c640a5a67e13e471eb5b4c3691ec49f Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Fri, 20 Oct 2023 14:57:43 +0200 Subject: [PATCH 14/20] Update email notification handling by considering the new MSC4062 --- ...-all-encrypted-events-except-for-muted-rooms.md | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 8b58d0de115..727054886e3 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -43,7 +43,12 @@ A new predefined **override** push rule (`.m.rule.encrypted_event`) is added jus "value": "m.room.encrypted" } ], - "actions": ["notify"] + "actions": [ + "notify", + { + "set_tweak": "org.matrix.msc4062.dont_email" + } + ] } ``` @@ -95,12 +100,11 @@ When this rule will be present and enabled in the account push rules set, the cl A user may set up a pusher to receive emails with unread notifications (see the spec [here](https://spec.matrix.org/v1.8/client-server-api/#post_matrixclientv3pushersset) with `kind` = "email"). Note that in the current Synapse implementation this pusher sends emails on a delay of ~10 minutes to give people time to see the notification and mark it as read. It looks like some other server implementations don't support email pushers though (see details in [this comment](https://github.com/matrix-org/matrix-spec-proposals/pull/4028#discussion_r1364373223)). -Currently the existing email notifications are not really relevant in case of encrypted rooms. The users may receive emails with a long bunch of encrypted messages when the room is configured in "all messages" notification mode. This feature should be reviewed again with a Product point of view. +Currently the existing email notifications are not really relevant in case of encrypted rooms. The users may receive emails with a long bunch of encrypted messages when the room is configured in "all messages" notification mode. This feature should be reviewed again but this is not the topic of this MSC. -In order to not disturb the existing email notifications mechanism, the server implementations which support it should ignore in the email notifications the events pushed because of the new push rule. -Otherwise the number of emails will increase by including the encrypted rooms for which the notifications are configured in mentions-and-keywords-only mode. These rooms are not handled for the moment in the email notifications. +In order to not disturb the existing email notifications mechanism, the server implementations which support it should ignore in the email notifications the events pushed because of the new push rule. Otherwise the number of emails will increase by including the encrypted rooms for which the notifications are configured in mentions-and-keywords-only mode. These rooms are not handled for the moment in the email notifications. -As a first option, the server should ignore these pushed events by using the push rule Id: `.m.rule.encrypted_event` (`org.matrix.msc4028.encrypted_event`). We may consider to work later on a new MSC to introduce a new [action](https://spec.matrix.org/latest/client-server-api/#actions) _"push but don't notify"_ or a new tweak _"dont_email"_ to exclude some pushed events from the email. +As a first option, we suggested to ignore these pushed events by using the push rule Id: `org.matrix.msc4028.encrypted_event`. But this would require significant rework to the way Synapse handles push notifications (in particular the database will need to be modified -- this is a massive table so modifying the database is non-trival). To avoid database modifications, we wrote [MSC4062](https://github.com/matrix-org/matrix-spec-proposals/pull/4062). The new tweak `org.matrix.msc4062.dont_email` would let us disable the email notification in the push rule definition. ## Potential issues From 080032eaa685beb03a4f8e557ea71cc7fbacadac Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Fri, 20 Oct 2023 15:00:26 +0200 Subject: [PATCH 15/20] typo --- .../4028-push-all-encrypted-events-except-for-muted-rooms.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 727054886e3..79ba2bbc2a3 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -104,7 +104,7 @@ Currently the existing email notifications are not really relevant in case of en In order to not disturb the existing email notifications mechanism, the server implementations which support it should ignore in the email notifications the events pushed because of the new push rule. Otherwise the number of emails will increase by including the encrypted rooms for which the notifications are configured in mentions-and-keywords-only mode. These rooms are not handled for the moment in the email notifications. -As a first option, we suggested to ignore these pushed events by using the push rule Id: `org.matrix.msc4028.encrypted_event`. But this would require significant rework to the way Synapse handles push notifications (in particular the database will need to be modified -- this is a massive table so modifying the database is non-trival). To avoid database modifications, we wrote [MSC4062](https://github.com/matrix-org/matrix-spec-proposals/pull/4062). The new tweak `org.matrix.msc4062.dont_email` would let us disable the email notification in the push rule definition. +As a first option, we suggested to ignore these pushed events by using the push rule Id: `org.matrix.msc4028.encrypted_event`. But this would require significant rework to the way Synapse handles push notifications (in particular the database will need to be modified -- this is a massive table so modifying the database is non-trivial). To avoid database modifications, we wrote [MSC4062](https://github.com/matrix-org/matrix-spec-proposals/pull/4062). The new tweak `org.matrix.msc4062.dont_email` would let us disable the email notification in the push rule definition. ## Potential issues From 0375295dd06bdc5fa4600f35f49521abee017d44 Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Mon, 23 Oct 2023 11:42:43 +0200 Subject: [PATCH 16/20] limit lines to approximately 120 characters --- ...encrypted-events-except-for-muted-rooms.md | 157 ++++++++++++++---- 1 file changed, 122 insertions(+), 35 deletions(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 79ba2bbc2a3..77b727e5584 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -1,15 +1,22 @@ # MSC4028: Push all encrypted events except for muted rooms -About notifications handling, it is currently not possible for mobile clients which get push notifications to properly implement a mentions-and-keywords-only room when encryption is enabled. +About notifications handling, it is currently not possible for mobile clients which get push notifications to properly +implement a mentions-and-keywords-only room when encryption is enabled. -Like it was already described in [MSC3996: Encrypted mentions-only rooms](https://github.com/matrix-org/matrix-spec-proposals/pull/3996): +Like it was already described in +[MSC3996: Encrypted mentions-only rooms](https://github.com/matrix-org/matrix-spec-proposals/pull/3996): -Currently, every new event in an encrypted room might be pushed to mobile clients due to the default **underride** rule [`.m.rule.encrypted`](https://spec.matrix.org/v1.7/client-server-api/#default-underride-rules). +Currently, every new event in an encrypted room might be pushed to mobile clients due to the default **underride** +rule [`.m.rule.encrypted`](https://spec.matrix.org/v1.7/client-server-api/#default-underride-rules). -However, a room can be configured to be mentions-and-keywords-only by creating a [room-specific push rule](https://spec.matrix.org/v1.7/client-server-api/#push-rules) -with the `id` property set to the room ID & `actions` set to do nothing.[^1] Since this is evaluated *before* the `.m.rule.encrypted` rule (almost) **no events get pushed for a mentions-and-keywords-only room**. +However, a room can be configured to be mentions-and-keywords-only by creating a +[room-specific push rule](https://spec.matrix.org/v1.7/client-server-api/#push-rules) +with the `id` property set to the room ID & `actions` set to do nothing.[^1] Since this is evaluated *before* the +`.m.rule.encrypted` rule (almost) **no events get pushed for a mentions-and-keywords-only room**. -Additionally, a room can be configured to be muted by creating the **override** [push rule](https://spec.matrix.org/v1.7/client-server-api/#push-rules) which matches the room ID & has `actions` set to do nothing, e.g.: +Additionally, a room can be configured to be muted by creating the **override** +[push rule](https://spec.matrix.org/v1.7/client-server-api/#push-rules) which matches the room ID & has `actions` set +to do nothing, e.g.: ```json { @@ -29,7 +36,8 @@ Additionally, a room can be configured to be muted by creating the **override** ## Proposal -A new predefined **override** push rule (`.m.rule.encrypted_event`) is added just after the `.m.rule.master` push rule to be able to push all encrypted events: +A new predefined **override** push rule (`.m.rule.encrypted_event`) is added just after the `.m.rule.master` push rule +to be able to push all encrypted events: ```json { @@ -52,87 +60,163 @@ A new predefined **override** push rule (`.m.rule.encrypted_event`) is added jus } ``` -Note: The “user-defined rules” are evaluated with a higher priority than “server-default rules”, except for the `.m.rule.master` rule which has always a higher priority than any other rule (see [here](https://spec.matrix.org/v1.7/client-server-api/#predefined-rules)). So all the **override** push rules created to mute rooms will be evaluated before this new one. The new rule will be evaluated before all the other “server-default rules”. This new rule deprecates the existing underride rule [`.m.rule.encrypted`](https://spec.matrix.org/v1.7/client-server-api/#default-underride-rules), which will become useless. +Note: The “user-defined rules” are evaluated with a higher priority than “server-default rules”, except for the +`.m.rule.master` rule which has always a higher priority than any other rule (see +[here](https://spec.matrix.org/v1.7/client-server-api/#predefined-rules)). So all the **override** push rules created +to mute rooms will be evaluated before this new one. The new rule will be evaluated before all the other +“server-default rules”. When it is enabled, this new rule makes useless the 2 existing underride rules: +`.m.rule.encrypted_room_one_to_one` and `.m.rule.encrypted` (predefined +[here](https://spec.matrix.org/v1.7/client-server-api/#default-underride-rules)). ### Receiving notifications -When this push rule matches then homeserver would push the event to all registered clients, similar to how other rules work. Clients would [decrypt the event and re-run push rules](https://spec.matrix.org/v1.7/client-server-api/#receiving-notifications) as normal. If the decrypted event results in no notification, the push is discarded [^2]. If the decryption failed, the new `.m.rule.encrypted_event` rule will match on client side, but there is no reason to notify the end user, so the push is discarded [^2] too. Globally the client must discard the push when the event stays encrypted locally because this use case should not happen except if the event was not encrypted for the current session, or if we are in front of an unexpected "Unable To Decrypt" (UTD) issue. +When this push rule matches then homeserver would push the event to all registered clients, similar to how other rules +work. Clients would +[decrypt the event and re-run push rules](https://spec.matrix.org/v1.7/client-server-api/#receiving-notifications) as +normal. If the decrypted event results in no notification, the push is discarded [^2]. If the decryption failed, the +new `.m.rule.encrypted_event` rule will match on client side, but there is no reason to notify the end user, so the +push is discarded [^2] too. Globally the client must discard the push when the event stays encrypted locally because +this use case should not happen except if the event was not encrypted for the current session, or if we are in front +of an unexpected "Unable To Decrypt" (UTD) issue. The overall tradeoff is that clients will receive extra pushes some of the time. ### Unread notification counts -Since the encryption has been introduced, the actual unread notification counts have to be updated/computed at the client side for the **encrypted rooms**. Nothing has been specified for that, so each client implemented its own mechanism. We want to harmonize this here. +Since the encryption has been introduced, the actual unread notification counts have to be updated/computed at the +client side for the **encrypted rooms**. Nothing has been specified for that, so each client implemented its own +mechanism. We want to harmonize this here. -As soon as the rule `.m.rule.encrypted_event` is present and enabled in the account push rules set, all the clients with or without push services should compute the unread notification count for each encrypted room during the client-server sync, by implementing the algorithm described below. Its complexity is similar to the existing implementations. +As soon as the rule `.m.rule.encrypted_event` is present and enabled in the account push rules set, all the clients +with or without push services should compute the unread notification count for each encrypted room during the +client-server sync, by implementing the algorithm described below. Its complexity is similar to the existing +implementations. -This algorithm depends on the unread notification count `unread_notifications` received from the server for each joined room in the [sync response](https://spec.matrix.org/v1.7/client-server-api/#get_matrixclientv3sync). It has to be adapted to the potential unread thread notification counts `unread_thread_notifications` that we will ignore here to ease the description. +This algorithm depends on the unread notification count `unread_notifications` received from the server for each joined +room in the [sync response](https://spec.matrix.org/v1.7/client-server-api/#get_matrixclientv3sync). It has to be +adapted to the potential unread thread notification counts `unread_thread_notifications` that we will ignore here to +ease the description. Let's consider we receive the following data for a joined encrypted rooms in the sync response: - number of new encrypted events in the timeline: n - unread notification count: k -1. if k = 0 -> reset the local notification counts for this room (the unread notification count and the highlight count). +1. if k = 0 -> reset the local notification counts for this room (the unread notification count and the highlight +count). 2. else if k <= n -> reset the local notification counts, then decrypt and run push rules on the k most recent events: - - if the decrypted event resulted in a highlight notification -> increment the local highlight notifications count and the local unread notification count. + - if the decrypted event resulted in a highlight notification -> increment the local highlight notifications count + and the local unread notification count. - if the decrypted event results in a notification -> increment only the local unread notifications count. - if the decrypted event results in no notification -> no change is required on the local notification counts. - if the decryption failed -> increment the local unread notifications count. -3. else if k > n and no gap is mentioned in the sync response for this room -> update the local notification counts by taking into account the potential change of the read receipt position of the current user, then decrypt and run push rules on the n received events by applying the same rules as above. +3. else if k > n and no gap is mentioned in the sync response for this room -> update the local notification counts by +taking into account the potential change of the read receipt position of the current user, then decrypt and run push +rules on the n received events by applying the same rules as above. 4. else if k > n and a gap is mentioned in the sync response -> there is 2 options: -- flush the local room storage, this will reset the local notification counts, then decrypt and run push rules on the n received events by applying the same rules as above. In order to compute locally accurate notification counts, back paginate to retrieve (k-n) more encrypted events. +- flush the local room storage, this will reset the local notification counts, then decrypt and run push rules on the +n received events by applying the same rules as above. In order to compute locally accurate notification counts, back +paginate to retrieve (k-n) more encrypted events. or -- update the local notification counts by taking into account the potential change of the read receipt position of the current user, then decrypt and run push rules on the n received events by applying the same rules as above. In order to compute locally accurate notification counts, back paginate to close the gap or at least retrieve (k-n) more encrypted events. +- update the local notification counts by taking into account the potential change of the read receipt position of the +current user, then decrypt and run push rules on the n received events by applying the same rules as above. In order +to compute locally accurate notification counts, back paginate to close the gap or at least retrieve (k-n) more +encrypted events. -About the case 4, the back pagination should be mandatory for mentions-and-keywords-only rooms to detect if there are some mentions in the gap. The client may decide to not trigger back pagination for rooms in "all messages" mode, but in that case the local unread notification count is unknown. No badge can be displayed at the application side, only an unread marker. +About the case 4, the back pagination should be mandatory for mentions-and-keywords-only rooms to detect if there are +some mentions in the gap. The client may decide to not trigger back pagination for rooms in "all messages" mode, but +in that case the local unread notification count is unknown. No badge can be displayed at the application side, only +an unread marker. -Note: some unencrypted events (for example `m.room.redaction`) may be taken into account or not in this unreads count k. This depends on the push rules set. If you force "all messages" mode on a room by creating a room-specific push rule, some unencrypted events (like redaction events) will be included in this count. This is not the case if "all messages" mode is obtained by the combination of server predefined rules. The client have locally all the information (= the push rules) to handle this edge case during the implementation of the algorithm. +Note: some unencrypted events (for example `m.room.redaction`) may be taken into account or not in this unreads +count k. This depends on the push rules set. If you force "all messages" mode on a room by creating a room-specific +push rule, some unencrypted events (like redaction events) will be included in this count. This is not the case if +"all messages" mode is obtained by the combination of server predefined rules. The client have locally all the +information (= the push rules) to handle this edge case during the implementation of the algorithm. ### Listing notifications -This new push rule will impact the existing endpoint [`GET /_matrix/client/v3/notifications`](https://spec.matrix.org/v1.7/client-server-api/#get_matrixclientv3notifications). +This new push rule will impact the existing endpoint +[`GET /_matrix/client/v3/notifications`](https://spec.matrix.org/v1.7/client-server-api/#get_matrixclientv3notifications). -When this rule will be present and enabled in the account push rules set, the clients which retrieve the list of events that it has been notified about with this endpoint, will receive most of the encrypted events (except for muted rooms). They will be able to decrypt and re-run push rules locally. This should fix the notifications panel observed in some web clients where currently the notifications of encrypted events are missing. +When this rule will be present and enabled in the account push rules set, the clients which retrieve the list of +events that it has been notified about with this endpoint, will receive most of the encrypted events (except for +muted rooms). They will be able to decrypt and re-run push rules locally. This should fix the notifications panel +observed in some web clients where currently the notifications of encrypted events are missing. ### Email notifications -A user may set up a pusher to receive emails with unread notifications (see the spec [here](https://spec.matrix.org/v1.8/client-server-api/#post_matrixclientv3pushersset) with `kind` = "email"). Note that in the current Synapse implementation this pusher sends emails on a delay of ~10 minutes to give people time to see the notification and mark it as read. It looks like some other server implementations don't support email pushers though (see details in [this comment](https://github.com/matrix-org/matrix-spec-proposals/pull/4028#discussion_r1364373223)). +A user may set up a pusher to receive emails with unread notifications (see the spec +[here](https://spec.matrix.org/v1.8/client-server-api/#post_matrixclientv3pushersset) with `kind` = "email"). +Note that in the current Synapse implementation this pusher sends emails on a delay of ~10 minutes to give people time +to see the notification and mark it as read. It looks like some other server implementations don't support email +pushers though (see details in +[this comment](https://github.com/matrix-org/matrix-spec-proposals/pull/4028#discussion_r1364373223)). -Currently the existing email notifications are not really relevant in case of encrypted rooms. The users may receive emails with a long bunch of encrypted messages when the room is configured in "all messages" notification mode. This feature should be reviewed again but this is not the topic of this MSC. +Currently the existing email notifications are not really relevant in case of encrypted rooms. +The users may receive emails with a long bunch of encrypted messages when the room is configured in "all messages" +notification mode. This feature should be reviewed again but this is not the topic of this MSC. -In order to not disturb the existing email notifications mechanism, the server implementations which support it should ignore in the email notifications the events pushed because of the new push rule. Otherwise the number of emails will increase by including the encrypted rooms for which the notifications are configured in mentions-and-keywords-only mode. These rooms are not handled for the moment in the email notifications. +In order to not disturb the existing email notifications mechanism, the server implementations which support it should +ignore in the email notifications the events pushed because of the new push rule. Otherwise the number of emails will +increase by including the encrypted rooms for which the notifications are configured in mentions-and-keywords-only mode. +These rooms are not handled for the moment in the email notifications. -As a first option, we suggested to ignore these pushed events by using the push rule Id: `org.matrix.msc4028.encrypted_event`. But this would require significant rework to the way Synapse handles push notifications (in particular the database will need to be modified -- this is a massive table so modifying the database is non-trivial). To avoid database modifications, we wrote [MSC4062](https://github.com/matrix-org/matrix-spec-proposals/pull/4062). The new tweak `org.matrix.msc4062.dont_email` would let us disable the email notification in the push rule definition. +As a first option, we suggested to ignore these pushed events by using the push rule Id: `org.matrix.msc4028.encrypted_event`. +But this would require significant rework to the way Synapse handles push notifications (in particular the database +will need to be modified -- this is a massive table so modifying the database is non-trivial). +To avoid database modifications, we wrote [MSC4062](https://github.com/matrix-org/matrix-spec-proposals/pull/4062). +The new tweak `org.matrix.msc4062.dont_email` would let us disable the email notification in the push rule definition. ## Potential issues ### Client side #### Unexpected unread counts -Until a client is updated to support this MSC, it may display a high number of unread notifications for encrypted rooms. The first workaround would be to not display the count for the encrypted rooms when the rule `.m.rule.encrypted_event` is present and enabled in the account push rules set, until the suggested algorithm is implemented, at least for the mentions-and-keywords-only encrypted rooms. +Until a client is updated to support this MSC, it may display a high number of unread notifications for encrypted rooms. +The first workaround would be to not display the count for the encrypted rooms when the rule `.m.rule.encrypted_event` +is present and enabled in the account push rules set, until the suggested algorithm is implemented, at least for the +mentions-and-keywords-only encrypted rooms. #### Battery life -The mobile client will wake up more frequently than currently. We will have to control any potential issue with battery life when we will implement this MSC. The impact on the battery life will be definitely lower than the one observed on clients without push service which poll for events in the background. The end user will be able to reduce the impact by muting some encrypted rooms. +The mobile client will wake up more frequently than currently. We will have to control any potential issue with battery +life when we will implement this MSC. The impact on the battery life will be definitely lower than the one observed +on clients without push service which poll for events in the background. The end user will be able to reduce +the impact by muting some encrypted rooms. ### Server side #### Event fetching -The clients with push services will request more frequently the server to retrieve the event contents. I'm mainly thinking about an encrypted room with a high number of members, each mobile client of these members will try to retrieve simultaneously the event content of each new encrypted event (pushed by the server). This scenario may happen today only when most of the members configure the room in "All messages" mode. +The clients with push services will request more frequently the server to retrieve the event contents. +I'm mainly thinking about an encrypted room with a high number of members, each mobile client of these members will +try to retrieve simultaneously the event content of each new encrypted event (pushed by the server). +This scenario may happen today only when most of the members configure the room in "All messages" mode. -This may be a potential performance issue if each client fetches separately each pushed event. But this should not be the case, because the clients have to run a sync if they want to be able to receive potential new encryption keys through to_devices. So they will retrieve more than one event at the time. +This may be a potential performance issue if each client fetches separately each pushed event. +But this should not be the case, because the clients have to run a sync if they want to be able to receive potential +new encryption keys through to_devices. So they will retrieve more than one event at the time. -BTW [MSC3013: Encrypted Push](https://github.com/matrix-org/matrix-spec-proposals/pull/3013) could potentially mitigate this. +BTW [MSC3013: Encrypted Push](https://github.com/matrix-org/matrix-spec-proposals/pull/3013) could potentially +mitigate this. #### Push sending -Push is a huge bottleneck on the sending side. So calculating what to push and the unread counts (its the same process) is a big performance bottleneck currently when sending in large rooms. A bunch of optimisation has taken place to make that better, but its still quite expensive. We should plan an effort to limit potential issue server side here. +Push is a huge bottleneck on the sending side. So calculating what to push and the unread counts (its the same process) +is a big performance bottleneck currently when sending in large rooms. +A bunch of optimisation has taken place to make that better, but its still quite expensive. +We should plan an effort to limit potential issue server side here. ## Alternatives -[MSC3996: Encrypted mentions-only rooms](https://github.com/matrix-org/matrix-spec-proposals/pull/3996): the suggested top-level property `m.has_mentions` may be an option to reduce the volume of pushes. But we would not be able to support notifications on keywords then. +[MSC3996: Encrypted mentions-only rooms](https://github.com/matrix-org/matrix-spec-proposals/pull/3996): the suggested +top-level property `m.has_mentions` may be an option to reduce the volume of pushes. But we would not be able +to support notifications on keywords then. -Another alternative would be to define a new [`action`](https://spec.matrix.org/v1.7/client-server-api/#actions): `push_without_notify` or `silently_notify` in order to push all encrypted events without incrementing the notifications count. The client implementation to handle these counts would be then less complex. But this would need to implement more changes in Synapse. The current proposal is just to add a new push rules without changing the rules handling server side. This alternative should be discussed as soon as possible. +Another alternative would be to define a new [`action`](https://spec.matrix.org/v1.7/client-server-api/#actions): +`push_without_notify` or `silently_notify` in order to push all encrypted events without incrementing +the notifications count. The client implementation to handle these counts would be then less complex. +But this would need to implement more changes in Synapse. The current proposal is just to add a new push rules +without changing the rules handling server side. This alternative should be discussed as soon as possible. ## Security considerations @@ -144,7 +228,9 @@ None ## Unstable prefix -During development the new push rule shall use `org.matrix.msc4028.encrypted_event` instead of `.m.rule.encrypted_event`. Contrary to the stable push rule, this unstable one should not be enabled by default until the clients are ready to support a minimum this MSC. +During development the new push rule shall use `org.matrix.msc4028.encrypted_event` instead of `.m.rule.encrypted_event`. +Contrary to the stable push rule, this unstable one should not be enabled by default until the clients are ready +to support a minimum this MSC. ## Dependencies @@ -155,4 +241,5 @@ None equivalent, see [issue #643](https://github.com/matrix-org/matrix-spec/issues/643). [^2]: In the past it was not possible to discard notifications on iOS: if a push -notification was received it *had to be displayed*. This is [no longer the case](https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_usernotifications_filtering). +notification was received it *had to be displayed*. This is +[no longer the case](https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_usernotifications_filtering). From 49ace6d8831fea8af3cb9f17218df1ad3c1b5f33 Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Wed, 8 Nov 2023 16:47:06 +0100 Subject: [PATCH 17/20] Update: this unstable push rule is enabled by default (like the stable one) --- .../4028-push-all-encrypted-events-except-for-muted-rooms.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 77b727e5584..0d62675855d 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -229,8 +229,8 @@ None ## Unstable prefix During development the new push rule shall use `org.matrix.msc4028.encrypted_event` instead of `.m.rule.encrypted_event`. -Contrary to the stable push rule, this unstable one should not be enabled by default until the clients are ready -to support a minimum this MSC. +Caution: this unstable push rule will be enabled by defaut like the stable push rule, the server owners should wait for +the clients to support a minimum this MSC before enabling the MSC server side. ## Dependencies From 4852fc819e3b4e540dc86cefbf81e5b0270dcb50 Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Wed, 8 Nov 2023 16:49:42 +0100 Subject: [PATCH 18/20] typo --- .../4028-push-all-encrypted-events-except-for-muted-rooms.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index 0d62675855d..d0fd5e88242 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -229,7 +229,8 @@ None ## Unstable prefix During development the new push rule shall use `org.matrix.msc4028.encrypted_event` instead of `.m.rule.encrypted_event`. -Caution: this unstable push rule will be enabled by defaut like the stable push rule, the server owners should wait for + +Caution: this unstable push rule will be enabled by default like the stable push rule, the server owners should wait for the clients to support a minimum this MSC before enabling the MSC server side. ## Dependencies From d3a1e87c26bd811372bf4819413ed685242612b7 Mon Sep 17 00:00:00 2001 From: Giom Foret Date: Wed, 10 Jan 2024 11:02:37 +0100 Subject: [PATCH 19/20] Add msc4028 to the /_matrix/clients/versions endpoint --- .../4028-push-all-encrypted-events-except-for-muted-rooms.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index d0fd5e88242..df32988cbd9 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -233,6 +233,10 @@ During development the new push rule shall use `org.matrix.msc4028.encrypted_eve Caution: this unstable push rule will be enabled by default like the stable push rule, the server owners should wait for the clients to support a minimum this MSC before enabling the MSC server side. +To ensure the server supports the functionality before a spec release, clients should look for the unstable feature `org.matrix.msc4028` in the response of the `/_matrix/clients/versions` endpoint. Once released in the specification, clients should be checking for server support through advertised spec versions instead. + +Once this MSC has successfully passed a merge FCP, clients should use `.m.rule.encrypted_event` as the right push rule. The unstable `org.matrix.msc4028.encrypted_event` can be used only as a fallback if the right one is missing (backwards compatibility with slightly old servers). + ## Dependencies None From 5c96969c724a7776da492f7c02fdda54f27b6de5 Mon Sep 17 00:00:00 2001 From: giomfo Date: Wed, 10 Jan 2024 17:54:51 +0100 Subject: [PATCH 20/20] Update proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- .../4028-push-all-encrypted-events-except-for-muted-rooms.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md index df32988cbd9..4931044c013 100644 --- a/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md +++ b/proposals/4028-push-all-encrypted-events-except-for-muted-rooms.md @@ -233,7 +233,7 @@ During development the new push rule shall use `org.matrix.msc4028.encrypted_eve Caution: this unstable push rule will be enabled by default like the stable push rule, the server owners should wait for the clients to support a minimum this MSC before enabling the MSC server side. -To ensure the server supports the functionality before a spec release, clients should look for the unstable feature `org.matrix.msc4028` in the response of the `/_matrix/clients/versions` endpoint. Once released in the specification, clients should be checking for server support through advertised spec versions instead. +To ensure the server supports the functionality before a spec release, clients should interpret an unstable feature `org.matrix.msc4028` with a value of `true` in the response of the `/_matrix/clients/versions` endpoint as the feature being supported by the homeserver. Once released in the specification, clients should be checking for server support through advertised spec versions instead. Once this MSC has successfully passed a merge FCP, clients should use `.m.rule.encrypted_event` as the right push rule. The unstable `org.matrix.msc4028.encrypted_event` can be used only as a fallback if the right one is missing (backwards compatibility with slightly old servers).