From 5dadeaa9a19315e65d0a6b3b40995289a050d781 Mon Sep 17 00:00:00 2001 From: "Olivier Wilkinson (reivilibre)" Date: Thu, 6 Aug 2020 15:59:35 +0100 Subject: [PATCH 1/5] WIP Troubleshooting doc Signed-off-by: Olivier Wilkinson (reivilibre) --- TROUBLESHOOTING.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 TROUBLESHOOTING.md diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md new file mode 100644 index 00000000..0bf59f2c --- /dev/null +++ b/TROUBLESHOOTING.md @@ -0,0 +1,20 @@ +# Troubleshooting Sygnal deployments + +Push notifications can be rather hard to get right, and there are plenty of +places to trip up and have nothing but silence to show for your efforts. + +This document offers some suggestions of what to check and lists some common +pitfalls you may encounter. + + +## Troubleshooting Firebase Notifications + +### Troubles with iOS apps on Firebase + +Sygnal currently only sends 'data messages' (also called 'silent notifications', +but this name could be misleading). Data messages + + +## Troubleshooting APNs Notifications + + From 1abdaafe9648c41f573d418d6eddd8aaa5f7add9 Mon Sep 17 00:00:00 2001 From: "Olivier Wilkinson (reivilibre)" Date: Fri, 21 Aug 2020 15:34:35 +0100 Subject: [PATCH 2/5] Add more documentation! --- README.rst | 16 ++++ TROUBLESHOOTING.md | 20 ---- docs/applications.md | 202 ++++++++++++++++++++++++++++++++++++++++ docs/troubleshooting.md | 200 +++++++++++++++++++++++++++++++++++++++ 4 files changed, 418 insertions(+), 20 deletions(-) delete mode 100644 TROUBLESHOOTING.md create mode 100644 docs/applications.md create mode 100644 docs/troubleshooting.md diff --git a/README.rst b/README.rst index b80b4505..cc57b39e 100644 --- a/README.rst +++ b/README.rst @@ -9,6 +9,7 @@ for a high level overview of how notifications work in Matrix. https://matrix.org/docs/spec/push_gateway/r0.1.0 describes the protocol that Matrix Home Servers use to send notifications to Push Gateways such as Sygnal. + Setup ===== Sygnal is configured through a YAML configuration file. @@ -27,6 +28,7 @@ Keys in this section take the form of the ``app_id``, as specified when setting See the sample configuration for examples. + App Types --------- There are two supported App Types: @@ -65,6 +67,7 @@ gcm to contain the 'Server key', which can be acquired from Firebase Console at: ``https://console.firebase.google.com/project//settings/cloudmessaging/`` + Using an HTTP Proxy for outbound traffic ---------------------------------------- @@ -78,6 +81,7 @@ Currently only HTTP proxies with the CONNECT method are supported. If you wish, you can instead configure a HTTP CONNECT proxy in ``sygnal.yaml``. + Pusher ``data`` configuration ============================= @@ -90,6 +94,7 @@ via `POST /_matrix/client/r0/pushers/set `_ +* `Troubleshooting <./docs/troubleshooting.md>`_ diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md deleted file mode 100644 index 0bf59f2c..00000000 --- a/TROUBLESHOOTING.md +++ /dev/null @@ -1,20 +0,0 @@ -# Troubleshooting Sygnal deployments - -Push notifications can be rather hard to get right, and there are plenty of -places to trip up and have nothing but silence to show for your efforts. - -This document offers some suggestions of what to check and lists some common -pitfalls you may encounter. - - -## Troubleshooting Firebase Notifications - -### Troubles with iOS apps on Firebase - -Sygnal currently only sends 'data messages' (also called 'silent notifications', -but this name could be misleading). Data messages - - -## Troubleshooting APNs Notifications - - diff --git a/docs/applications.md b/docs/applications.md new file mode 100644 index 00000000..6335e898 --- /dev/null +++ b/docs/applications.md @@ -0,0 +1,202 @@ +# Notes for Application Developers + +This document aims to illustrate some of the quirks, peculiarities and other +notable aspects of Sygnal for developers of applications that need to receive +push. + +Sygnal has been somewhat flavoured by the development of the Element iOS +and Element Android clients, but nevertheless is intended to be useful for +other iOS and Android applications that have 'typical' requirements, without +need to resort to customising the pushkins. + +(It is possible to extend Sygnal with other kinds of pushkins, but this is +out of scope for this document.) + +Once you have read this document, you may also find [the troubleshooting document](./troubleshooting.md) +useful if you are running into trouble whilst deploying Sygnal. + + +## Definitions + +* **APNs**: Apple Push Notification service, a notification service for iOS + applications. +* **FCM**: Firebase Cloud Messaging (previously Google Cloud Messaging), a + notification service primarily for Android applications but also usable for + iOS and Chrome applications. +* **push key**: Also known as registration token (FCM) or device token (APNs), + this ID identifies a device to which notifications can be sent. + + +## Outlines of Flows + +An understanding of the flows of push notifications may be useful. +This section will provide an outline. + + +### Registration flow + +0) The client needs to be configured with the address of its Sygnal instance, + and any configuration that FCM or APNs requires for client apps. + +1) The client needs to use the FCM or APNs library to acquire a push key, which + identifies the client to the notification service. + +2) The client registers a pusher on the user's homeserver, giving the address of + its Sygnal instance, its push key, and perhaps some other parameters. + To register a pusher, it uses [POST /_matrix/client/r0/pushers/set](https://matrix.org/docs/spec/client_server/latest#post-matrix-client-r0-pushers-set). + + +#### Worth noting + +* In general, there is no contract between the operators of the homeserver(s) + and the operators of the Sygnal push gateways. + + - Sygnal (push gateway) instances are deployed by the owner of the application. + + - Unless the application restricts it, the user is usually free to choose any + homeserver which may be operated by anyone. + +* It is not feasible to allow end-users to configure their own Sygnal instance, + because the Sygnal instance needs the appropriate FCM or APNs secrets that + belong to the application. + + +### Notification flow + +1) The user's homeserver receives an event (from federation, another local user, + or even an application service — the source is unimportant). + +2) The user's homeserver applies push rules to the event on behalf of the user, + in order to determine whether to send a push notification or not (as well as + a few other tweaks). + + - For example, the push rules can decide to exclude certain rooms, or to + notify for mentions on keywords (so the user could, for example, be notified + when anyone mentions 'lunch'). + +3) If the homeserver decides to send a notification, we continue. + +4) The homeserver calls [POST /_matrix/push/v1/notify](https://matrix.org/docs/spec/push_gateway/latest#post-matrix-push-v1-notify) + on the Sygnal instance that the user's client registered. + +5) The Sygnal receives this request and rewrites it into a request for the + appropriate notification service (APNs or FCM), which it then sends. + +6) Sygnal responds to the homeserver about the push's success and whether or not + the push key was rejected. + + +#### Worth noting + +* When Sygnal only sends data messages (also known as 'silent notifications') to + target devices — the application needs to wake up and trigger its own + notification, perhaps after downloading and decrypting the event. + + - Data-only messages are always sent to FCM. + + - Data-only messages are sent to APNs when the `event_id_only` format is in + use. In other cases, the app may still need to perform additional processing + as encrypted events would need to be decrypted, for example. + +* When encrypted events are present, the homeserver is unable to conclusively + run push rules — in this case, the client will need to run them locally to + decide whether or not a notification should be displayed, a sound needs to be + played and/or the message needs to be highlighted. + +* **Consider user privacy**: if you use the `event_id_only` format, then data + sent to the notification service (FCM or APNs) is minimal. If you do not, then + unencrypted messages will have their content sent to the notification service, + which some may prefer to avoid. + + +## Platform-specific Notes + +### Apple Push Notification service + +By default, the client will receive a message with this structure: + +```json +{ + "room_id": "!slw48wfj34rtnrf:example.com", + "event_id": "$qTOWWTEL48yPm3uT-gdNhFcoHxfKbZuqRVnnWWSkGBs", + "aps": { + "alert": { + "loc-key": "MSG_FROM_USER_IN_ROOM_WITH_CONTENT", + "loc-args": [ + "Major Tom", + "Mission Control", + "I'm floating in a most peculiar way." + ] + }, + "badge": 3 + } +} +``` + +Please note that fields may be truncated if they are large, so that they fit +within APNs's limit. +Please also note that some fields will be unavailable if you registered a pusher +with `event_id_only` format. + + +#### iOS Applications Beware! + +When registering your iOS pusher, you have the ability to specify a default +payload that will be sent to APNs. This allows you to set custom flags for APNs. + +Of particular interest are `content-available` and `mutable-content` (which you +can set to `1` to enable). + +Consult [APNs documentation] for a more in-depth explanation, but: + +* `content-available` wakes up your application for up to 30 seconds in which it + can process the message. +* `mutable-content` allows your application to mutate (modify) the notification. + +An example `data` dictionary to specify on `POST /_matrix/client/r0/pushers/set`: + +```json +{ + "url": "https://push-gateway.location.here/_matrix/push/v1/notify", + "format": "event_id_only", + "default_payload": { + "aps": { + "mutable-content": 1, + "content-available": 1, + "alert": {"loc-key": "SINGLE_UNREAD", "loc-args": []}, + } + } +} +``` + +[APNs documentation]: https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/CreatingtheNotificationPayload.html + + +### Firebase Cloud Messaging + +The client will receive a message with an FCM `data` payload with this structure: + +```json +{ + "event_id": "$3957tyerfgewrf384", + "type": "m.room.message", + "sender": "@exampleuser:example.org", + "room_name": "Mission Control", + "room_alias": "#exampleroom:example.org", + "sender_display_name": "Major Tom", + "content": { + "msgtype": "m.text", + "body": "I'm floating in a most peculiar way." + }, + "room_id": "!slw48wfj34rtnrf:example.org", + "prio": "high", + "unread": 2, + "missed_calls": 1 +} +``` + +Please note that fields may be truncated if they are large, so that they fit +within FCM's limit. +Please also note that some fields will be unavailable if you registered a pusher +with `event_id_only` format. + diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 00000000..1b9b398c --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,200 @@ +# Troubleshooting Sygnal deployments + +Push notifications can be rather hard to get right, and there are plenty of +places to trip up and have nothing but silence to show for your efforts. + +This document offers some suggestions of what to check and lists some common +pitfalls you may encounter, based on experience. + +There are also appendices with information which may be useful for manual +debugging. + +Your first steps are to ensure that you have logging so you can see what is going +on; a level of `INFO` or even `DEBUG` will be useful. + + +## Narrowing in on the problem + +### Check the pusher is registered in your homeserver + +Typically, applications will register a pusher on startup. + +If you have access to your homeserver, you can check that it is making it there. + +Start your application and then run a query against the database. + +#### On Synapse + +Use `sqlite3 /path/to/homeserver.db` or `psql synapse` as required for your +deployment. + +```sql +SELECT app_id, data FROM pushers +WHERE user_name = '@my.user:example.org' AND kind='http'; +``` + +You should see something like: + +``` + app_id | data +-------------------+-------------------------------------------------------- + org.example.chat | {"format":"event_id_only", + | "url":"https://example.org/_matrix/push/v1/notify"} +``` + + +#### On other homeserver implementations + +No details available, but contributions welcomed. + + +### Check the push gateway (Sygnal) is reachable from the homeserver + +Following on from the example above, the homeserver's database contains the +push gateway URL of `https://example.org/_matrix/push/v1/notify`. + +It may be worth manually checking that the push gateway is reachable; e.g. with +curl: + +``` +$ curl https://example.org/_matrix/push/v1/notify + + 405 - Method Not Allowed + +

Method Not Allowed

+

Your browser approached me (at /_matrix/push/v1/notify) with the method "GET". I only allow the methods HEAD, POST here.

+ + +``` + +If you get a response, such as an error like 405 Method Not Allowed, as above, +this would suggest that the push gateway is at least reachable. + +If you get a **404 No Such Resource** error on the `/_matrix/push/v1/notify` endpoint, +then chances are that your reverse proxy is not configured to pass through the +full URL. + +If you don't get a HTTP response, then it is probably worth investigation. +Check that: + +* Sygnal is running +* Sygnal's configuration makes it listen on the desired port +* Any reverse proxies are correctly set up and running +* The firewall permits inbound traffic on the port in question. + + +## Troubleshooting Firebase Notifications + +### iOS-specific troubles with apps using Firebase + +#### App doesn't receive notifications when inactive + +Sygnal currently only sends 'data messages' (also called 'silent notifications', +but this name could be misleading). + +Whereas data messages will wake up apps on Android with no additional changes, +iOS needs to be told that a notification is meant to wake up an inactive app. +This is done with the `content_available` flag, which you can set in your +`fcm_options` dictionary for the Firebase pushkin. +(See `sygnal.yaml.sample`.) + + +## Troubleshooting APNs Notifications + +### Base64 decoding error in the logs + +#### Common cause 1: Hex rather than base64 encoding + +Sygnal's APNs support expects your pushkeys to be base64 encoded rather than +hexadecimally encoded. + +*(Why? The previous APNs API which Sygnal supported was binary and didn't define +a text-safe encoding, so it was chosen to use base64 in Sygnal. Now the new API +exists and specifies hexadecimal encoding, but Sygnal retains backwards +compatibility and will do the base64-to-hex conversion.)* + + +#### Common cause 2: Firebase token given + +If you are using Firebase for your iOS app, you will get Firebase tokens +(looking a bit like `blahblahblah:APA91blahblahblah`… note the presence of a +colon which is not valid base64). + +In this case, you need to **configure Sygnal to use a FCM (gcm) pushkin rather +than an APNs one, as Firebase talks to APNs on your behalf**. +Instead of configuring Sygnal with your APNs secrets, you need to configure +Firebase with your APNs secrets, and Sygnal with your Firebase secrets. + + +### App doesn't receive notifications when inactive + +If you want your application to be woken up to be able to process APNs messages +received when your application is in the background, you need to set the +`content-available` flag in your pusher's default payload — see +[the notes for iOS applications](./applications.md#ios-applications-beware). + + +# Appendices + +## Sending a notification to Sygnal manually with `curl` + +Note: this depends on the heredoc syntax of the `bash` shell. + +```bash +curl -i -H "Content-Type: application/json" --request POST -d '@-' http://syg1:8008/_matrix/push/v1/notify <", + "pushkey": "", + "pushkey_ts": 12345678, + "data": {}, + "tweaks": { + "sound": "bing" + } + } + ] + } +} +EOF +``` + + +## Example of an FCM request + +HTTP data sent to `https://fcm.googleapis.com/fcm/send`: + +``` +POST /fcm/send HTTP/1.1 +User-Agent: sygnal +Content-Type: application/json +Authorization: key= +Host: fcm.googleapis.com + +{"data": {"event_id": "$3957tyerfgewrf384", "type": "m.room.message", "sender": "@exampleuser:example.org", "room_name": "Mission Control", "room_alias": "#exampleroom:example.org", "membership": null, "sender_display_name": "Major Tom", "content": {"msgtype": "m.text", "body": "I'm floating in a most peculiar way."}, "room_id": "!slw48wfj34rtnrf:example.org", "prio": "high", "unread": 2, "missed_calls": 1}, "priority": "high", "to": ""} +``` + +You can send using curl using: + +```bash +curl -i -H "Content-Type: application/json" -H "Authorization: key=" --request POST -d '@-' https://fcm.googleapis.com/fcm/send <"} +EOF +``` From 3c4c7114ef9374cc883dcfc97d0e3fbd872d7929 Mon Sep 17 00:00:00 2001 From: "Olivier Wilkinson (reivilibre)" Date: Fri, 21 Aug 2020 15:35:44 +0100 Subject: [PATCH 3/5] Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) --- changelog.d/150.doc | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelog.d/150.doc diff --git a/changelog.d/150.doc b/changelog.d/150.doc new file mode 100644 index 00000000..783994f8 --- /dev/null +++ b/changelog.d/150.doc @@ -0,0 +1 @@ +Add preliminary documentation (Troubleshooting and Application Developers' Notes). From 0eacc69ad6f702aa8bf17a996ea14ab61795e618 Mon Sep 17 00:00:00 2001 From: "Olivier Wilkinson (reivilibre)" Date: Tue, 25 Aug 2020 13:32:31 +0100 Subject: [PATCH 4/5] Incorporate suggestions --- README.rst | 4 ++-- docs/applications.md | 25 ++++++++++++++----------- docs/troubleshooting.md | 22 +++++++++++----------- 3 files changed, 27 insertions(+), 24 deletions(-) diff --git a/README.rst b/README.rst index cc57b39e..3d2cdaef 100644 --- a/README.rst +++ b/README.rst @@ -121,5 +121,5 @@ More Documentation More documentation for Sygnal is available in the ``docs`` directory: -* `Notes for Application Developers <./docs/applications.md>`_ -* `Troubleshooting <./docs/troubleshooting.md>`_ +* `Notes for Application Developers `_ +* `Troubleshooting `_ diff --git a/docs/applications.md b/docs/applications.md index 6335e898..7441135b 100644 --- a/docs/applications.md +++ b/docs/applications.md @@ -1,4 +1,4 @@ -# Notes for Application Developers +# Notes for application developers This document aims to illustrate some of the quirks, peculiarities and other notable aspects of Sygnal for developers of applications that need to receive @@ -7,9 +7,9 @@ push. Sygnal has been somewhat flavoured by the development of the Element iOS and Element Android clients, but nevertheless is intended to be useful for other iOS and Android applications that have 'typical' requirements, without -need to resort to customising the pushkins. +need to resort to customising the application types (Pushkins). -(It is possible to extend Sygnal with other kinds of pushkins, but this is +(It is possible to extend Sygnal with other application types, but this is out of scope for this document.) Once you have read this document, you may also find [the troubleshooting document](./troubleshooting.md) @@ -25,9 +25,12 @@ useful if you are running into trouble whilst deploying Sygnal. iOS and Chrome applications. * **push key**: Also known as registration token (FCM) or device token (APNs), this ID identifies a device to which notifications can be sent. +* **Pushkin**: A module which adds support to Sygnal for an application type. + Sygnal comes with an APNs pushkin and an FCM pushkin out of the box, but you + may also use custom ones. -## Outlines of Flows +## Outlines of flows An understanding of the flows of push notifications may be useful. This section will provide an outline. @@ -38,7 +41,7 @@ This section will provide an outline. 0) The client needs to be configured with the address of its Sygnal instance, and any configuration that FCM or APNs requires for client apps. -1) The client needs to use the FCM or APNs library to acquire a push key, which +1) The client needs to use an FCM or APNs library to acquire a push key, which identifies the client to the notification service. 2) The client registers a pusher on the user's homeserver, giving the address of @@ -95,8 +98,8 @@ This section will provide an outline. - Data-only messages are always sent to FCM. - Data-only messages are sent to APNs when the `event_id_only` format is in - use. In other cases, the app may still need to perform additional processing - as encrypted events would need to be decrypted, for example. + use. In other cases, the app may still need to perform additional processing, + for example if encrypted events would need to be decrypted. * When encrypted events are present, the homeserver is unable to conclusively run push rules — in this case, the client will need to run them locally to @@ -109,7 +112,7 @@ This section will provide an outline. which some may prefer to avoid. -## Platform-specific Notes +## Platform-specific notes ### Apple Push Notification service @@ -134,12 +137,12 @@ By default, the client will receive a message with this structure: ``` Please note that fields may be truncated if they are large, so that they fit -within APNs's limit. +within APNs' limit. Please also note that some fields will be unavailable if you registered a pusher with `event_id_only` format. -#### iOS Applications Beware! +#### iOS applications beware! When registering your iOS pusher, you have the ability to specify a default payload that will be sent to APNs. This allows you to set custom flags for APNs. @@ -163,7 +166,7 @@ An example `data` dictionary to specify on `POST /_matrix/client/r0/pushers/set` "aps": { "mutable-content": 1, "content-available": 1, - "alert": {"loc-key": "SINGLE_UNREAD", "loc-args": []}, + "alert": {"loc-key": "SINGLE_UNREAD", "loc-args": []} } } } diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 1b9b398c..9eb0f81b 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -30,7 +30,7 @@ deployment. ```sql SELECT app_id, data FROM pushers -WHERE user_name = '@my.user:example.org' AND kind='http'; + WHERE user_name = '@my.user:example.org' AND kind='http'; ``` You should see something like: @@ -45,7 +45,7 @@ You should see something like: #### On other homeserver implementations -No details available, but contributions welcomed. +No details available, but contributions welcome. ### Check the push gateway (Sygnal) is reachable from the homeserver @@ -53,8 +53,8 @@ No details available, but contributions welcomed. Following on from the example above, the homeserver's database contains the push gateway URL of `https://example.org/_matrix/push/v1/notify`. -It may be worth manually checking that the push gateway is reachable; e.g. with -curl: +It may be worth manually checking that the push gateway is reachable from the +homeserver; e.g. with curl: ``` $ curl https://example.org/_matrix/push/v1/notify @@ -67,23 +67,23 @@ $ curl https://example.org/_matrix/push/v1/notify ``` -If you get a response, such as an error like 405 Method Not Allowed, as above, +If you get a response, such as an error like **405 Method Not Allowed**, as above, this would suggest that the push gateway is at least reachable. If you get a **404 No Such Resource** error on the `/_matrix/push/v1/notify` endpoint, then chances are that your reverse proxy is not configured to pass through the full URL. -If you don't get a HTTP response, then it is probably worth investigation. +If you don't get an HTTP response, then it is probably worth investigation. Check that: * Sygnal is running * Sygnal's configuration makes it listen on the desired port * Any reverse proxies are correctly set up and running -* The firewall permits inbound traffic on the port in question. +* The firewall permits inbound traffic on the port in question -## Troubleshooting Firebase Notifications +## Troubleshooting Firebase notifications ### iOS-specific troubles with apps using Firebase @@ -96,10 +96,10 @@ Whereas data messages will wake up apps on Android with no additional changes, iOS needs to be told that a notification is meant to wake up an inactive app. This is done with the `content_available` flag, which you can set in your `fcm_options` dictionary for the Firebase pushkin. -(See `sygnal.yaml.sample`.) +(See [`sygnal.yaml.sample`](../sygnal.yaml.sample).) -## Troubleshooting APNs Notifications +## Troubleshooting APNs notifications ### Base64 decoding error in the logs @@ -131,7 +131,7 @@ Firebase with your APNs secrets, and Sygnal with your Firebase secrets. If you want your application to be woken up to be able to process APNs messages received when your application is in the background, you need to set the `content-available` flag in your pusher's default payload — see -[the notes for iOS applications](./applications.md#ios-applications-beware). +[the notes for iOS applications](applications.md#ios-applications-beware). # Appendices From 49c0edf3991834f9a3cc6824d57726566a7e3e4e Mon Sep 17 00:00:00 2001 From: reivilibre <38398653+reivilibre@users.noreply.github.com> Date: Tue, 25 Aug 2020 15:44:54 +0100 Subject: [PATCH 5/5] Update docs/troubleshooting.md Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> --- docs/troubleshooting.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 9eb0f81b..8e58d645 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -94,7 +94,7 @@ but this name could be misleading). Whereas data messages will wake up apps on Android with no additional changes, iOS needs to be told that a notification is meant to wake up an inactive app. -This is done with the `content_available` flag, which you can set in your +This is done with FCM's `content_available` flag, which you can set in your `fcm_options` dictionary for the Firebase pushkin. (See [`sygnal.yaml.sample`](../sygnal.yaml.sample).)