From f82e463ea329417538f2d79b1dd0e2e25a0ebe5b Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 23 Mar 2023 12:40:54 -0600 Subject: [PATCH 01/13] Sending One-Time Key (OTK) claims to appservices --- .../3983-sending-otk-claims-to-appservices.md | 132 ++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 proposals/3983-sending-otk-claims-to-appservices.md diff --git a/proposals/3983-sending-otk-claims-to-appservices.md b/proposals/3983-sending-otk-claims-to-appservices.md new file mode 100644 index 00000000000..71386fca322 --- /dev/null +++ b/proposals/3983-sending-otk-claims-to-appservices.md @@ -0,0 +1,132 @@ +# MSC3983: Sending One-Time Key (OTK) claims to appservices + +Presently in Matrix, the public portion of OTKs are [uploaded](https://spec.matrix.org/v1.6/client-server-api/#uploading-keys) +to the homeserver to ensure other devices can encrypt new messages without requiring the device to +be online and responsive. This works for devices operating exclusively over the Client-Server API, +however [appservices](https://spec.matrix.org/v1.6/application-service-api/) looking to support +encryption (through [MSC3202](https://github.com/matrix-org/matrix-spec-proposals/pull/3202) or +similar) could have millions or billions of users on them, which can easily translate to quite a +few public keys needing to be uploaded to the homeserver. + +Given appservices *generally* have an uptime which is equivalent to the homeserver itself, and will +have already stored the public portion of its OTKs somewhere, we can save a bit of duplication by +having the homeserver delegate [`/keys/claim`](https://spec.matrix.org/v1.6/client-server-api/#post_matrixclientv3keysclaim) +requests to the appservice. + +In numbers, a conservative estimate for an interoperable messaging bridge (appservice) would be +500 million users. Each user generates between 50 and 100 OTKs, so we'll pick the low end at 50. +That's 25 **billion** public keys. Currently in Matrix, that means the appservice stores 25 billion +keys and the homeserver stores a copy of those 25 billion keys. + +This proposal introduces a mechanism for saving the homeserver from duplicating 25 billion keys. + +## Background + +Appservices can register a [namespace](https://spec.matrix.org/v1.6/application-service-api/#registration) +of users either exclusively (no one else can register users matching the regex) or implicitly (the +appservice receives events about those users, but can't prevent registration). Implicit namespaces +can be shared across multiple appservices. + +## Proposal + +For users under an appservice's explicit namespace, if that user has no unused OTKs (excluding fallback +keys) on the homeserver, the homeserver proxies the following APIs to the appservice using the new +API described below: +* [`/_matrix/client/v3/keys/claim`](https://spec.matrix.org/v1.6/client-server-api/#post_matrixclientv3keysclaim) +* [`/_matrix/federation/v1/user/keys/claim`](https://spec.matrix.org/v1.6/server-server-api/#post_matrixfederationv1userkeysclaim) + +**`POST /_matrix/app/v1/keys/claim`** +```jsonc +// Request +{ + "@alice:example.org": { + "DEVICEID": ["signed_curve25519", "signed_curve25519"] // device ID to algorithm names + }, + // ... +} +``` +```jsonc +// Response +{ + "@alice:example.org": { + "DEVICEID": { + "signed_curve25519:AAAAHg": { + "key": "...", + "signatures": { + "@alice:example.org": { + "ed25519:DEVICEID": "..." + } + } + }, + "signed_curve25519:BBBBHg": { + "key": "...", + "signatures": { + "@alice:example.org": { + "ed25519:DEVICEID": "..." + } + } + } + } + }, + // ... +} +``` + +Multiple users, devices, and keys for those devices can be claimed in a single request. This is to +allow homeservers to batch multiple client/federation requests into a single request on the appservice, +if desirable. This is an optional optimization for homeserver implementations. In the example above, 2 +keys are claimed for one device. + +If the appservice responds with an error of any kind (including timeout), the homeserver uses the +fallback key, if known. + +In this case, the appservice is responsible for ensuring it doesn't use a key twice. The +`device_one_time_keys_count` field for the appservice (over MSC3202, for example) would be zero. In +many implementations, when this field falls below a threshold it is common for upload requests to +happen: appservices intending on using the new API should not perform those uploads as it means, +quite simply, not using the new API. + +Normally, a client (including appservices) would consider a OTK "used" once it sees it in a to-device +message. Given the claim query is proxied directly to where the keys are generated in this proposal, +appservices *can* consider a key "used" when it is claimed, before seeing it in a to-device message. + +If the homeserver uses the fallback key, that will be communicated in the traditional ways to the +appservice (namely through `device_unused_fallback_key_types` in the case of MSC3202). + +We don't apply this API to implicit (non-exclusive) users as it's possible for multiple appservices +to have a namespace covering the user: instead of guessing or going around to each, we require the +user to be in an exclusive namespace. This guarantees that there's only one appservice responsible +for the user. + +## Potential issues + +As described, the appservice could be offline or in fact experience a worse uptime than the homeserver. +This new API is optional for appservices: if they don't want to use it (because they know their uptime +will be bad), they can simply upload keys in advance, just like before this proposal. Similarly, if +the appservice is trying to use the API but is offline, they *should* have a fallback key to continue +using as, well, a fallback. + +## Alternatives + +No major alternatives. + +It could be argued that supporting a fallback key for appservices is too much considering their uptime, +however in practice a protocol should not be designed around 100% uptime. This is additionally why the +proposal doesn't suggest proxying device/signing key queries to the appservice: the appservice could be +down, leaving encryption broken until it comes back online. + +## Security considerations + +No major considerations. + +## Unstable prefix + +While this MSC is not considered stable, implementations should use +`/_matrix/app/unstable/org.matrix.msc3983/keys/claim` as the endpoint instead. There is no version +compatibility check: homeservers implementing this functionality would receive an error from appservices +which don't support the endpoint and thus engage in the behaviour described by the MSC. + +## Dependencies + +This MSC has no direct dependencies, however is of little use without being partnered with something +like [MSC3202](https://github.com/matrix-org/matrix-spec-proposals/pull/3202). From 90006ad0434253584f2ea598a890d43f514ba7af Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 23 Mar 2023 12:44:18 -0600 Subject: [PATCH 02/13] Fix typos --- .github/_typos.toml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/_typos.toml b/.github/_typos.toml index 7c107218e20..956ee4334c8 100644 --- a/.github/_typos.toml +++ b/.github/_typos.toml @@ -1,2 +1,6 @@ [default] check-filename = true + +[default.extend-words] +OTK = "OTK" +OTKs = "OTKs" \ No newline at end of file From b128030edb3446e2756c04d403ab0d4f582b92fb Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 23 Mar 2023 12:44:57 -0600 Subject: [PATCH 03/13] Actually fix typos? --- .github/_typos.toml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/_typos.toml b/.github/_typos.toml index 956ee4334c8..676f0fbaeec 100644 --- a/.github/_typos.toml +++ b/.github/_typos.toml @@ -1,6 +1,6 @@ [default] check-filename = true -[default.extend-words] +[default.extend-identifiers] OTK = "OTK" -OTKs = "OTKs" \ No newline at end of file +OTKs = "OTKs" From 0630814439723aeb9aa456d5cb36dc5a753e683c Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 23 Mar 2023 14:42:05 -0600 Subject: [PATCH 04/13] Oops, auth is important --- proposals/3983-sending-otk-claims-to-appservices.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/proposals/3983-sending-otk-claims-to-appservices.md b/proposals/3983-sending-otk-claims-to-appservices.md index 71386fca322..e016ea66e34 100644 --- a/proposals/3983-sending-otk-claims-to-appservices.md +++ b/proposals/3983-sending-otk-claims-to-appservices.md @@ -72,6 +72,9 @@ API described below: } ``` +*Note*: Like other appservice endpoints, this endpoint should *not* be ratelimited and *does* require +normal [authentication](https://spec.matrix.org/v1.6/application-service-api/#authorization). + Multiple users, devices, and keys for those devices can be claimed in a single request. This is to allow homeservers to batch multiple client/federation requests into a single request on the appservice, if desirable. This is an optional optimization for homeserver implementations. In the example above, 2 From bbc79ab607dba2c1e869ddd44a1500c9f1f4e512 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Thu, 23 Mar 2023 15:13:00 -0600 Subject: [PATCH 05/13] Clarify that OTKs should only be used once still --- proposals/3983-sending-otk-claims-to-appservices.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/3983-sending-otk-claims-to-appservices.md b/proposals/3983-sending-otk-claims-to-appservices.md index e016ea66e34..cf1524e88e6 100644 --- a/proposals/3983-sending-otk-claims-to-appservices.md +++ b/proposals/3983-sending-otk-claims-to-appservices.md @@ -89,9 +89,9 @@ many implementations, when this field falls below a threshold it is common for u happen: appservices intending on using the new API should not perform those uploads as it means, quite simply, not using the new API. -Normally, a client (including appservices) would consider a OTK "used" once it sees it in a to-device -message. Given the claim query is proxied directly to where the keys are generated in this proposal, -appservices *can* consider a key "used" when it is claimed, before seeing it in a to-device message. +Normally the homeserver would be [ensuring](https://spec.matrix.org/v1.6/client-server-api/#one-time-and-fallback-keys) +OTKs are only used once, however with the appservice serving the endpoint it becomes the responsibility +of the appservice to perform this check. If the homeserver uses the fallback key, that will be communicated in the traditional ways to the appservice (namely through `device_unused_fallback_key_types` in the case of MSC3202). From 91e6d0273cc87e7fc6b2882e2878193726c74890 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 24 Mar 2023 10:52:39 -0600 Subject: [PATCH 06/13] Acknowledge performance hit --- proposals/3983-sending-otk-claims-to-appservices.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/proposals/3983-sending-otk-claims-to-appservices.md b/proposals/3983-sending-otk-claims-to-appservices.md index cf1524e88e6..3114d1188b7 100644 --- a/proposals/3983-sending-otk-claims-to-appservices.md +++ b/proposals/3983-sending-otk-claims-to-appservices.md @@ -109,6 +109,11 @@ will be bad), they can simply upload keys in advance, just like before this prop the appservice is trying to use the API but is offline, they *should* have a fallback key to continue using as, well, a fallback. +For appservices which never intend to upload keys there is a bit of a wasted lookup to see if there are +any keys for the user(s). This could be mitigated with an implementation-specific flag to skip the lookup +and just do proxying, though for the general case in this MSC the fallback key consideration is kept for +reliability concerns. + ## Alternatives No major alternatives. From 0b9f6d9005710f0526a1825bae31208dd41d7668 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 24 Mar 2023 10:53:00 -0600 Subject: [PATCH 07/13] Mention e2be approach currently in use --- proposals/3983-sending-otk-claims-to-appservices.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/proposals/3983-sending-otk-claims-to-appservices.md b/proposals/3983-sending-otk-claims-to-appservices.md index 3114d1188b7..35c5d5eadff 100644 --- a/proposals/3983-sending-otk-claims-to-appservices.md +++ b/proposals/3983-sending-otk-claims-to-appservices.md @@ -116,7 +116,11 @@ reliability concerns. ## Alternatives -No major alternatives. +Many encryption-capable bridges today can avoid uploading OTKs (and sometimes even device keys) because +they have a bot user in the room. The bot user uploads its keys, but the remaining bridge users do not. +This works if the bridge users don't need to be involved in rooms without the bot user present, though +being able to (securely) DM bridge users is a valuable consideration for this MSC. In future, scalable +encryption for appservices might take the shape of an appservice-wide device of some sort. It could be argued that supporting a fallback key for appservices is too much considering their uptime, however in practice a protocol should not be designed around 100% uptime. This is additionally why the From 8a1dd86c69b32d1717dde8765ca362a363ae9df4 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 24 Mar 2023 10:59:48 -0600 Subject: [PATCH 08/13] Mention encryption bridge usecase --- proposals/3983-sending-otk-claims-to-appservices.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/proposals/3983-sending-otk-claims-to-appservices.md b/proposals/3983-sending-otk-claims-to-appservices.md index 35c5d5eadff..8c34acd7728 100644 --- a/proposals/3983-sending-otk-claims-to-appservices.md +++ b/proposals/3983-sending-otk-claims-to-appservices.md @@ -126,6 +126,12 @@ It could be argued that supporting a fallback key for appservices is too much co however in practice a protocol should not be designed around 100% uptime. This is additionally why the proposal doesn't suggest proxying device/signing key queries to the appservice: the appservice could be down, leaving encryption broken until it comes back online. +## Additional uses + +An appservice aiming to bridge two different encryption systems might use this endpoint to save on data, +though currently the encryption used on both sides of the bridge would need to be compatible (ie: signatures +from device IDs and user IDs need to exist). In future, other MSCs might make encryption bridges easier to +build. ## Security considerations From 5a4a6c64d1e5b2ee928464f8e84d4f14c24d2b59 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 24 Mar 2023 11:02:09 -0600 Subject: [PATCH 09/13] Use the fallback key when missing *some* keys too --- proposals/3983-sending-otk-claims-to-appservices.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/3983-sending-otk-claims-to-appservices.md b/proposals/3983-sending-otk-claims-to-appservices.md index 8c34acd7728..d6d561a4a1f 100644 --- a/proposals/3983-sending-otk-claims-to-appservices.md +++ b/proposals/3983-sending-otk-claims-to-appservices.md @@ -81,7 +81,9 @@ if desirable. This is an optional optimization for homeserver implementations. I keys are claimed for one device. If the appservice responds with an error of any kind (including timeout), the homeserver uses the -fallback key, if known. +fallback key, if known. The homeserver additionally uses the fallback key (if known) to fill in +missing keys from the appservice. For example, if the homeserver requested 2 keys for Alice but +the appservice only provided 1, the homeserver would use the fallback key to fulfill the second. In this case, the appservice is responsible for ensuring it doesn't use a key twice. The `device_one_time_keys_count` field for the appservice (over MSC3202, for example) would be zero. In From c7701687172a464b8558e047923e32389e15445f Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 24 Mar 2023 11:38:36 -0600 Subject: [PATCH 10/13] Mention MSC3984 for key queries --- proposals/3983-sending-otk-claims-to-appservices.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/proposals/3983-sending-otk-claims-to-appservices.md b/proposals/3983-sending-otk-claims-to-appservices.md index d6d561a4a1f..2c7c2fa153e 100644 --- a/proposals/3983-sending-otk-claims-to-appservices.md +++ b/proposals/3983-sending-otk-claims-to-appservices.md @@ -125,9 +125,10 @@ being able to (securely) DM bridge users is a valuable consideration for this MS encryption for appservices might take the shape of an appservice-wide device of some sort. It could be argued that supporting a fallback key for appservices is too much considering their uptime, -however in practice a protocol should not be designed around 100% uptime. This is additionally why the -proposal doesn't suggest proxying device/signing key queries to the appservice: the appservice could be -down, leaving encryption broken until it comes back online. +however in practice appservices are not quite able to achieve 100% uptime. This proposal doesn't propose +proxying device/signing key queries to the appservice for the same reliability concerns, though appservices +which wish to opt to do so anyways could use [MSC3984](https://github.com/matrix-org/matrix-spec-proposals/pull/3984). + ## Additional uses An appservice aiming to bridge two different encryption systems might use this endpoint to save on data, @@ -150,3 +151,6 @@ which don't support the endpoint and thus engage in the behaviour described by t This MSC has no direct dependencies, however is of little use without being partnered with something like [MSC3202](https://github.com/matrix-org/matrix-spec-proposals/pull/3202). + +This MSC is additionally useful when paired with [MSC3984](https://github.com/matrix-org/matrix-spec-proposals/pull/3984), +though has no direct dependency. From a95bf4f55d02853da22c4c9dcb85d751282686dc Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Fri, 24 Mar 2023 12:39:14 -0600 Subject: [PATCH 11/13] Clarify that 405/M_UNRECOGNISED is a good idea --- proposals/3983-sending-otk-claims-to-appservices.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/proposals/3983-sending-otk-claims-to-appservices.md b/proposals/3983-sending-otk-claims-to-appservices.md index 2c7c2fa153e..23b3c0c8a50 100644 --- a/proposals/3983-sending-otk-claims-to-appservices.md +++ b/proposals/3983-sending-otk-claims-to-appservices.md @@ -116,6 +116,10 @@ any keys for the user(s). This could be mitigated with an implementation-specifi and just do proxying, though for the general case in this MSC the fallback key consideration is kept for reliability concerns. +Similarly, if an appservice doesn't intend on uploading keys (because it doesn't support encryption) and +indicates the route is [unknown](https://spec.matrix.org/v1.6/application-service-api/#unknown-routes), +the homeserver could avoid calling appservice with a backoff to prevent excessive calls. + ## Alternatives Many encryption-capable bridges today can avoid uploading OTKs (and sometimes even device keys) because From 2177b939078aecb0faf55facfcc08b0282d27cbf Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 19 Apr 2023 11:17:47 -0600 Subject: [PATCH 12/13] Add a section about overriding fallback keys --- .../3983-sending-otk-claims-to-appservices.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/proposals/3983-sending-otk-claims-to-appservices.md b/proposals/3983-sending-otk-claims-to-appservices.md index 23b3c0c8a50..7265f029758 100644 --- a/proposals/3983-sending-otk-claims-to-appservices.md +++ b/proposals/3983-sending-otk-claims-to-appservices.md @@ -103,6 +103,24 @@ to have a namespace covering the user: instead of guessing or going around to ea user to be in an exclusive namespace. This guarantees that there's only one appservice responsible for the user. +## Returning extra keys + +**TODO**: This is probably best as its own MSC. + +Independent of the appservice having `/keys/claim` proxied to it, it may be desireable for both the +fallback and one-time key to be returned. Servers should *always* include the fallback key alongside +the requested OTKs. When using this proposal's new endpoint, the server should use the fallback key +from the appservice's response rather than a previously stored fallback key, if present (if the +appservice doesn't respond with a fallback key then the server uses the stored fallback key instead, +if known). + +The server SHOULD NOT replace any uploaded fallback keys with ones returned by the appservice via +this proposal. The appservice MUST re-upload the fallback key if it wants to replace it, as it would +do upon first (known) use. + +Clients can determine which of the keys returned is the fallback key by `fallback: true` on the returned +keys. + ## Potential issues As described, the appservice could be offline or in fact experience a worse uptime than the homeserver. From 052d4800f29784571206da980a6b3855b31521c0 Mon Sep 17 00:00:00 2001 From: Travis Ralston Date: Wed, 19 Apr 2023 11:47:35 -0600 Subject: [PATCH 13/13] Update 3983-sending-otk-claims-to-appservices.md --- proposals/3983-sending-otk-claims-to-appservices.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/3983-sending-otk-claims-to-appservices.md b/proposals/3983-sending-otk-claims-to-appservices.md index 7265f029758..79141873dca 100644 --- a/proposals/3983-sending-otk-claims-to-appservices.md +++ b/proposals/3983-sending-otk-claims-to-appservices.md @@ -107,7 +107,7 @@ for the user. **TODO**: This is probably best as its own MSC. -Independent of the appservice having `/keys/claim` proxied to it, it may be desireable for both the +Independent of the appservice having `/keys/claim` proxied to it, it may be desirable for both the fallback and one-time key to be returned. Servers should *always* include the fallback key alongside the requested OTKs. When using this proposal's new endpoint, the server should use the fallback key from the appservice's response rather than a previously stored fallback key, if present (if the @@ -121,6 +121,8 @@ do upon first (known) use. Clients can determine which of the keys returned is the fallback key by `fallback: true` on the returned keys. +Servers MUST NOT mark the fallback key as "used" unless no other OTKs are returned. + ## Potential issues As described, the appservice could be offline or in fact experience a worse uptime than the homeserver.