From 1b81970a1edc19a228793441a21086b6c3b33aa2 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 17 Aug 2018 22:01:13 -0400 Subject: [PATCH 01/36] initial commit of e2e backup proposal --- .../1219-storing-megolm-keys-serverside.md | 197 ++++++++++++++++++ 1 file changed, 197 insertions(+) create mode 100644 proposals/1219-storing-megolm-keys-serverside.md diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md new file mode 100644 index 00000000000..b78fc953c1f --- /dev/null +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -0,0 +1,197 @@ +Storing megolm keys serverside +============================== + +Background +---------- + +We *optionally* let clients store a copy of their megolm inbound session keys +on the HS so that they can recover history if all devices are lost without an +explicit key export; transparently share history between a user's devices; +transparently share missing keys between a user's devices to fix UISIs; support +clients with limited local storage for keys. + +See also: + +* https://github.com/matrix-org/matrix-doc/issues/1219 +* https://github.com/vector-im/riot-web/issues/3661 +* https://github.com/vector-im/riot-web/issues/5675 +* https://docs.google.com/document/d/1MOoIA9qEKIhUQ3UmKZG-loqA8e0BzgWKKlKRUGMynVc/edit# + (old version of proposal) + +Proposal +-------- + +This proposal creates new APIs to allow clients to back up room decryption keys +on the server. Decryption keys are encrypted (using public key crypto) before +being sent to the server along with some unencrypted metadata to allow the +server to manage the backups, overwriting backups with "better" versions of the +keys. The user is given a private recovery key to save for recovering the keys +from the backup. + +Clients can create new versions of backups. A client would start a new version +of a backup when, for example, a user loses a device, and wants to ensure that +that device does not get any new decryption keys. + +### Possible UX for interactive clients + +On receipt of encryption keys (1st time): + +1. client checks if there is an existing backup: `GET /room_keys/version` + 1. if not, ask if the user wants to back up keys + 1. if yes: + 1. generate new key pair + 2. create new backup version: `POST /room_keys/version` + 3. display private key to user to save + 2. if no, exit and remember decision (user can change their mind later) + 3. while prompting, continue to poll `GET /room_keys/versions`, as + another device may have created a backup. If so, go to 1.2. + 2. if yes, get public key, prompt user to verify a device that signed the + key¹, or enter recovery key (which can derive the backup key). + 1. User can also decide to create a new backup, in which case, go to 1.1. +2. send key to backup: `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v` +3. continue backing up keys as we receive them (may receive a 403 error if a + new backup version has been created: see below) + +On 403 error when trying to `PUT` keys: + +1. get the current version +2. notify the user that there is a new backup version, and display relevant + information +3. confirm with user that they want to use the backup (user may want use the + backup, to stop backing up keys, or to create a new backup) +4. verify the device that signed the backup key¹, or enter recovery key + +¹: cross-signing (when that is completed) can be used to verify the device +that signed the key. + +On receipt of undecryptable message: + +1. ask user if they want to restore backup (ask whether to get individual key, + room keys, or all keys). (This can be done in the same place as asking if + the user wants to request keys from other devices.) +2. if yes, prompt for private key, and get keys: `GET /room_keys/keys` + +### API + +#### Backup versions + +##### `POST /room_keys/version` + +Create a new backup version. + +Body parameters: + +- `algorithm` (string): Required. The algorithm used for storing backups. + Currently, only `m.megolm_backup.v1` is defined. (FIXME: change the algorithm + name to include the encryption method) +- `auth_data` (string or object): Required. algorithm-dependent data. For + `m.megolm_backup.v1`, this is a signedjson object with the following keys: + - `public_key` (string): ... + - `signatures` (object): signatures of the public key + +Example: + +```javascript +{ + "algorithm": "m.megolm_backup.v1", + "auth_data": { + "public_key": { + "public_key": "abcdefg", + "signatures": { + "something": { + "ed25519:something": "hijklmnop" + } + } + } + } +} +``` + +On success, returns a JSON object with keys: + +- `version` (integer): the backup version + +##### `GET /room_keys/version` + +Get information about the current version. + +On success, returns a JSON object with keys: + +- `algorithm` (string): Required. Same as in the body parameters for `POST + /room_keys/version`. +- `auth_data` (string or object): Required. Same as in the body parameters for + `POST /room_keys/version`. +- `version` (integer): the backup version + + +#### Storing keys + +##### `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v` + +Store the key for the given session in the given room, using the given backup +version. + +If the server already has a backup in the backup version for the given session +and room, then it will keep the "better" one ... + +Body parameters: + +- `first_message_index` (integer): Required. The index of the first message + in the session that the key can decrypt. +- `forwarded_count` (integer): Required. The number of times this key has been + forwarded. +- `is_verified` (boolean): Whether the device backing up the key has verified + the device that the key is from. +- `session_data` (string): The backup of the key, encrypted according to the + backup algorithm. + +On success, returns ... ? + +##### `PUT /room_keys/keys/${roomId}?version=$v` + +Store several keys for the given room, using the given backup version. + +Behaves the same way as if the keys were added individually using `PUT +/room_keys/keys/${roomId}/${sessionId}?version=$v`. + +Body paremeters: +- `sessions` (object): an object where the keys are the session IDs, and the + values are objects of the same form as the body in `PUT + /room_keys/keys/${roomId}/${sessionId}?version=$v`. + +On success, returns same as `PUT +/room_keys/keys/${roomId}/${sessionId}?version=$v` + +##### `PUT /room_keys/keys/?version=$v` + +... + +#### Retrieving keys + +##### `GET /room_keys/keys/${roomId}/${sessionId}?version=$v` +##### `GET /room_keys/keys/${roomId}?version=$v` +##### `GET /room_keys/keys/?version=$v` + +#### Deleting keys + +##### `DELETE /room_keys/keys/${roomId}/${sessionId}?version=$v` +##### `DELETE /room_keys/keys/${roomId}?version=$v` +##### `DELETE /room_keys/keys/?version=$v` + +Tradeoffs +--------- + +Security Considerations +----------------------- + +An attacker who gains access to a user's account can delete or corrupt their +key backup. This proposal does not attempt to protect against that. + +Other Issues +------------ + +Since many clients will receive encryption keys at around the same time, +clients should randomly offset their requests ... + +Conclusion +---------- From 6e8ba1f7f8cd14e8540ef54d3efb1ac9694e5167 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Thu, 23 Aug 2018 23:04:21 -0400 Subject: [PATCH 02/36] add more details --- .../1219-storing-megolm-keys-serverside.md | 75 ++++++++++++++----- 1 file changed, 56 insertions(+), 19 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index b78fc953c1f..3af86f17e80 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -6,9 +6,8 @@ Background We *optionally* let clients store a copy of their megolm inbound session keys on the HS so that they can recover history if all devices are lost without an -explicit key export; transparently share history between a user's devices; -transparently share missing keys between a user's devices to fix UISIs; support -clients with limited local storage for keys. +explicit key export; fix UISIs; support clients with limited local storage for +keys. See also: @@ -28,9 +27,14 @@ server to manage the backups, overwriting backups with "better" versions of the keys. The user is given a private recovery key to save for recovering the keys from the backup. -Clients can create new versions of backups. A client would start a new version -of a backup when, for example, a user loses a device, and wants to ensure that -that device does not get any new decryption keys. +Clients can create new versions of backups. Aside from the initial backup +creation, a client might start a new version of a backup when, for example, a +user loses a device, and wants to ensure that that device does not get any new +decryption keys. + +Once one client has created a backup version, other clients can fetch the +public key for the backup from the server and add keys to the backup, if they +trust that the backup was not created by a malicious device. ### Possible UX for interactive clients @@ -49,10 +53,11 @@ On receipt of encryption keys (1st time): key¹, or enter recovery key (which can derive the backup key). 1. User can also decide to create a new backup, in which case, go to 1.1. 2. send key to backup: `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v` -3. continue backing up keys as we receive them (may receive a 403 error if a - new backup version has been created: see below) +3. continue backing up keys as we receive them (may receive a + `M_WRONG_ROOM_KEYS_VERSION` error if a new backup version has been created: + see below) -On 403 error when trying to `PUT` keys: +On `M_WRONG_ROOM_KEYS_VERSION` error when trying to `PUT` keys: 1. get the current version 2. notify the user that there is a new backup version, and display relevant @@ -71,6 +76,9 @@ On receipt of undecryptable message: the user wants to request keys from other devices.) 2. if yes, prompt for private key, and get keys: `GET /room_keys/keys` +Users can also set up or disable backups, or restore from backup via user +settings. + ### API #### Backup versions @@ -86,7 +94,7 @@ Body parameters: name to include the encryption method) - `auth_data` (string or object): Required. algorithm-dependent data. For `m.megolm_backup.v1`, this is a signedjson object with the following keys: - - `public_key` (string): ... + - `public_key` (string): the public key used to encrypt the backups - `signatures` (object): signatures of the public key Example: @@ -95,12 +103,10 @@ Example: { "algorithm": "m.megolm_backup.v1", "auth_data": { - "public_key": { - "public_key": "abcdefg", - "signatures": { - "something": { - "ed25519:something": "hijklmnop" - } + "public_key": "abcdefg", + "signatures": { + "something": { + "ed25519:something": "hijklmnop" } } } @@ -123,6 +129,10 @@ On success, returns a JSON object with keys: `POST /room_keys/version`. - `version` (integer): the backup version +Error codes: + +- `M_UNKNOWN`: No backup version has been created. FIXME: why not + `M_NOT_FOUND`? #### Storing keys @@ -142,11 +152,22 @@ Body parameters: forwarded. - `is_verified` (boolean): Whether the device backing up the key has verified the device that the key is from. -- `session_data` (string): The backup of the key, encrypted according to the - backup algorithm. +- `session_data` (string or object): Algorithm-dependent data. For + `m.megolm_backup.v1`, this is an object with the following keys: + - `ciphertext` (string): the encrypted version of the session key. See below + for how the session key is encoded. + - `ephemeral` (string): the public ephemeral key that was used to encrypt the + session key. + - `mac` (string): the message authentication code for the ciphertext. FIXME: + more details On success, returns ... ? +Error codes: + +- `M_WRONG_ROOM_KEYS_VERSION`: the version specified does not match the current + backup version + ##### `PUT /room_keys/keys/${roomId}?version=$v` Store several keys for the given room, using the given backup version. @@ -159,7 +180,7 @@ Body paremeters: values are objects of the same form as the body in `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v`. -On success, returns same as `PUT +Returns the same as `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v` ##### `PUT /room_keys/keys/?version=$v` @@ -178,6 +199,18 @@ On success, returns same as `PUT ##### `DELETE /room_keys/keys/${roomId}?version=$v` ##### `DELETE /room_keys/keys/?version=$v` +#### Key format + +Session keys are encoded as a JSON object with the properties: + +- `algorithm` (string): `m.megolm.v1.aes-sha2` +- `sender_key` (string): base64-encoded device curve25519 key +- `sender_claimed_keys` (object): object containing the identity keys for the + sending device +- `forwardingCurve25519KeyChain` (array): zero or more curve25519 keys for + devices who forwarded the session key +- `session_key` (string): base64-encoded session key + Tradeoffs --------- @@ -187,6 +220,10 @@ Security Considerations An attacker who gains access to a user's account can delete or corrupt their key backup. This proposal does not attempt to protect against that. +An attacker who gains access to a user's account can create a new backup +version using a key that they control. For this reason, clients SHOULD confirm +with users before sending keys to a new backup version. + Other Issues ------------ From 87772329875666f57f1ed5f5540a5626f6f3432e Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 5 Sep 2018 23:21:35 -0400 Subject: [PATCH 03/36] various clarifications --- proposals/1219-storing-megolm-keys-serverside.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 3af86f17e80..e24262e7df6 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -45,7 +45,8 @@ On receipt of encryption keys (1st time): 1. if yes: 1. generate new key pair 2. create new backup version: `POST /room_keys/version` - 3. display private key to user to save + 3. display private key to user to save TODO: specify how the key is + displayed 2. if no, exit and remember decision (user can change their mind later) 3. while prompting, continue to poll `GET /room_keys/versions`, as another device may have created a backup. If so, go to 1.2. @@ -76,7 +77,7 @@ On receipt of undecryptable message: the user wants to request keys from other devices.) 2. if yes, prompt for private key, and get keys: `GET /room_keys/keys` -Users can also set up or disable backups, or restore from backup via user +Users can also set up, disable, or rotate backups, or restore from backup via user settings. ### API @@ -201,7 +202,7 @@ Returns the same as `PUT #### Key format -Session keys are encoded as a JSON object with the properties: +Each session key is encoded as a JSON object with the properties: - `algorithm` (string): `m.megolm.v1.aes-sha2` - `sender_key` (string): base64-encoded device curve25519 key @@ -211,6 +212,8 @@ Session keys are encoded as a JSON object with the properties: devices who forwarded the session key - `session_key` (string): base64-encoded session key +... + Tradeoffs --------- @@ -227,8 +230,11 @@ with users before sending keys to a new backup version. Other Issues ------------ -Since many clients will receive encryption keys at around the same time, -clients should randomly offset their requests ... +Since many clients will receive encryption keys at around the same time, they +will all want to back up their copies of the keys at around the same time, +which may increase load on the server if this happens in a big room. (TODO: +how much of an issue is this?) For this reason, clients should offset their +backup requests randomly. Conclusion ---------- From 846e9e8fdc7d00a0ee163fae49eaebdc8938cd04 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Thu, 6 Sep 2018 17:52:31 -0400 Subject: [PATCH 04/36] add clarifications --- .../1219-storing-megolm-keys-serverside.md | 78 ++++++++++++++----- 1 file changed, 58 insertions(+), 20 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index e24262e7df6..72038436a94 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -43,7 +43,7 @@ On receipt of encryption keys (1st time): 1. client checks if there is an existing backup: `GET /room_keys/version` 1. if not, ask if the user wants to back up keys 1. if yes: - 1. generate new key pair + 1. generate new curve25519 key pair 2. create new backup version: `POST /room_keys/version` 3. display private key to user to save TODO: specify how the key is displayed @@ -91,18 +91,18 @@ Create a new backup version. Body parameters: - `algorithm` (string): Required. The algorithm used for storing backups. - Currently, only `m.megolm_backup.v1` is defined. (FIXME: change the algorithm - name to include the encryption method) + Currently, only `m.megolm_backup.v1.curve25519-aes-sha2` is defined. - `auth_data` (string or object): Required. algorithm-dependent data. For - `m.megolm_backup.v1`, this is a signedjson object with the following keys: - - `public_key` (string): the public key used to encrypt the backups + `m.megolm_backup.v1.curve25519-aes-sha2`, this is a signedjson object with + the following keys: + - `public_key` (string): the curve25519 public key used to encrypt the backups - `signatures` (object): signatures of the public key Example: ```javascript { - "algorithm": "m.megolm_backup.v1", + "algorithm": "m.megolm_backup.v1.curve25519-aes-sha2", "auth_data": { "public_key": "abcdefg", "signatures": { @@ -143,7 +143,10 @@ Store the key for the given session in the given room, using the given backup version. If the server already has a backup in the backup version for the given session -and room, then it will keep the "better" one ... +and room, then it will keep the "better" one. To determine which one is +"better", key backups are compared first by the `is_verified` flag (`true` is +better than `false`), then by the `first_message_index` (a lower number is better), +and finally by `forwarded_count` (a lower number is better). Body parameters: @@ -154,15 +157,16 @@ Body parameters: - `is_verified` (boolean): Whether the device backing up the key has verified the device that the key is from. - `session_data` (string or object): Algorithm-dependent data. For - `m.megolm_backup.v1`, this is an object with the following keys: - - `ciphertext` (string): the encrypted version of the session key. See below - for how the session key is encoded. - - `ephemeral` (string): the public ephemeral key that was used to encrypt the - session key. + `m.megolm_backup.v1.curve25519-aes-sha2`, this is an object with the + following keys (TODO: this stuff should be moved): + - `ciphertext` (string): the encrypted version of the session key, as an + unpadded base64 string. See below for how the session key is encoded. + - `ephemeral` (string): the public ephemeral curve25519 key that was used to + encrypt the session key, as an unpadded base64 string. - `mac` (string): the message authentication code for the ciphertext. FIXME: more details -On success, returns ... ? +On success, returns the empty JSON object. Error codes: @@ -176,7 +180,7 @@ Store several keys for the given room, using the given backup version. Behaves the same way as if the keys were added individually using `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v`. -Body paremeters: +Body parameters: - `sessions` (object): an object where the keys are the session IDs, and the values are objects of the same form as the body in `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v`. @@ -184,15 +188,43 @@ Body paremeters: Returns the same as `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v` -##### `PUT /room_keys/keys/?version=$v` +##### `PUT /room_keys/keys?version=$v` + +Store several keys, using the given backup version. + +Behaves the same way as if the keys were added individually using `PUT +/room_keys/keys/${roomId}/${sessionId}?version=$v`. + +Body parameters: +- `rooms` (object): an object where the keys are the room IDs, and the values + are objects of the same form as the body in `PUT + /room_keys/keys/${roomId}/?version=$v`. -... +Returns the same as `PUT +/room_keys/keys/${roomId}/${sessionId}?version=$v` #### Retrieving keys ##### `GET /room_keys/keys/${roomId}/${sessionId}?version=$v` + +Retrieve the key for the given session in the given room from the backup. + +On success, returns a JSON object in the same form as the request body of `PUT +/room_keys/keys/${roomId}/${sessionId}?version=$v`. + ##### `GET /room_keys/keys/${roomId}?version=$v` -##### `GET /room_keys/keys/?version=$v` + +Retrieve the all the keys for the given room from the backup. + +On success, returns a JSON object in the same form as the request body of `PUT +/room_keys/keys/${roomId}?version=$v`. + +##### `GET /room_keys/keys?version=$v` + +Retrieve all the keys from the backup. + +On success, returns a JSON object in the same form as the request body of `PUT +/room_keys/keys?version=$v`. #### Deleting keys @@ -200,7 +232,11 @@ Returns the same as `PUT ##### `DELETE /room_keys/keys/${roomId}?version=$v` ##### `DELETE /room_keys/keys/?version=$v` -#### Key format +Deletes keys from the backup. + +On success, returns the empty JSON object. + +#### `m.megolm_backup.v1.curve25519-aes-sha2` Key format Each session key is encoded as a JSON object with the properties: @@ -210,9 +246,11 @@ Each session key is encoded as a JSON object with the properties: sending device - `forwardingCurve25519KeyChain` (array): zero or more curve25519 keys for devices who forwarded the session key -- `session_key` (string): base64-encoded session key +- `session_key` (string): base64-encoded (unpadded) session key -... +The JSON object is then encrypted by generating an ephemeral curve25519 key, +performing an ECDH with the ephemeral key and the backup's public key to +generate an AES key, and encrypting the stringified object using AES. Tradeoffs --------- From 72df5fe43645acb97e6255cbf2c5368ee2aa4ad3 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 10 Oct 2018 16:28:24 -0400 Subject: [PATCH 05/36] add details on recovery key format, and some cleanups/fixes --- .../1219-storing-megolm-keys-serverside.md | 37 +++++++++++++++---- 1 file changed, 30 insertions(+), 7 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 72038436a94..ab75ccf2a90 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -45,8 +45,7 @@ On receipt of encryption keys (1st time): 1. if yes: 1. generate new curve25519 key pair 2. create new backup version: `POST /room_keys/version` - 3. display private key to user to save TODO: specify how the key is - displayed + 3. display private key to user to save (see below for the format) 2. if no, exit and remember decision (user can change their mind later) 3. while prompting, continue to poll `GET /room_keys/versions`, as another device may have created a backup. If so, go to 1.2. @@ -80,6 +79,30 @@ On receipt of undecryptable message: Users can also set up, disable, or rotate backups, or restore from backup via user settings. +### Recovery key + +The recovery key is can either be saved by the user directly, or stored +encrypted on the server (as proposed in +[MSC1687](https://github.com/matrix-org/matrix-doc/issues/1687)). If the key +is saved directly by the user, then it the code is constructed as follows: + +1. The 256-bit curve25519 private key is prepended by the bytes `0x8B` and + `0x01` +2. All the bytes in the string are above are XORed together to form a parity + byte. This parity byte is appended to the byte string. +3. The byte string is encoded using base58, using the same mapping as is used + for Bitcoin addresses. + +This 58-character string is presented to the user to save. Implementations may +add whitespace to the recovery key. + +When reading in a recovery key, clients must disregard whitespace. Clients +must base58-decode the code, ensure that the first two bytes of the decoded +string are `0x8B` and `0x01`, ensure that XOR-ing all the bytes together +results in 0, and ensure that the total length of the decoded string +is 35 bytes. Clients must then remove the first two bytes and the last byte, +and use the resulting string as the private key to decrypt backups. + ### API #### Backup versions @@ -118,9 +141,10 @@ On success, returns a JSON object with keys: - `version` (integer): the backup version -##### `GET /room_keys/version` +##### `GET /room_keys/version/{version}` -Get information about the current version. +Get information about the given version, or the current version if `{version}` +is omitted. On success, returns a JSON object with keys: @@ -128,12 +152,11 @@ On success, returns a JSON object with keys: /room_keys/version`. - `auth_data` (string or object): Required. Same as in the body parameters for `POST /room_keys/version`. -- `version` (integer): the backup version +- `version` (integer): Required. The backup version. Error codes: -- `M_UNKNOWN`: No backup version has been created. FIXME: why not - `M_NOT_FOUND`? +- `M_NOT_FOUND`: No backup version has been created. #### Storing keys From de5120335f950a53fec8374058f8c13055a0cecb Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Thu, 11 Oct 2018 10:22:42 -0400 Subject: [PATCH 06/36] change "string or object" to just "object" --- proposals/1219-storing-megolm-keys-serverside.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index ab75ccf2a90..312e43bfc4d 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -115,7 +115,7 @@ Body parameters: - `algorithm` (string): Required. The algorithm used for storing backups. Currently, only `m.megolm_backup.v1.curve25519-aes-sha2` is defined. -- `auth_data` (string or object): Required. algorithm-dependent data. For +- `auth_data` (object): Required. algorithm-dependent data. For `m.megolm_backup.v1.curve25519-aes-sha2`, this is a signedjson object with the following keys: - `public_key` (string): the curve25519 public key used to encrypt the backups @@ -150,7 +150,7 @@ On success, returns a JSON object with keys: - `algorithm` (string): Required. Same as in the body parameters for `POST /room_keys/version`. -- `auth_data` (string or object): Required. Same as in the body parameters for +- `auth_data` (object): Required. Same as in the body parameters for `POST /room_keys/version`. - `version` (integer): Required. The backup version. @@ -179,7 +179,7 @@ Body parameters: forwarded. - `is_verified` (boolean): Whether the device backing up the key has verified the device that the key is from. -- `session_data` (string or object): Algorithm-dependent data. For +- `session_data` (object): Algorithm-dependent data. For `m.megolm_backup.v1.curve25519-aes-sha2`, this is an object with the following keys (TODO: this stuff should be moved): - `ciphertext` (string): the encrypted version of the session key, as an From b45416e8b0ab432511eb73ece127f623bf7d8cc9 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 19 Oct 2018 22:19:55 -0400 Subject: [PATCH 07/36] change version from string to integer, plus other minor improvements --- proposals/1219-storing-megolm-keys-serverside.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 312e43bfc4d..14149950a60 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -139,7 +139,7 @@ Example: On success, returns a JSON object with keys: -- `version` (integer): the backup version +- `version` (string): the backup version ##### `GET /room_keys/version/{version}` @@ -152,7 +152,7 @@ On success, returns a JSON object with keys: /room_keys/version`. - `auth_data` (object): Required. Same as in the body parameters for `POST /room_keys/version`. -- `version` (integer): Required. The backup version. +- `version` (string): Required. The backup version. Error codes: @@ -275,9 +275,6 @@ The JSON object is then encrypted by generating an ephemeral curve25519 key, performing an ECDH with the ephemeral key and the backup's public key to generate an AES key, and encrypting the stringified object using AES. -Tradeoffs ---------- - Security Considerations ----------------------- @@ -286,7 +283,8 @@ key backup. This proposal does not attempt to protect against that. An attacker who gains access to a user's account can create a new backup version using a key that they control. For this reason, clients SHOULD confirm -with users before sending keys to a new backup version. +with users before sending keys to a new backup version or verify that it was +created by a trusted device by checking the signature. Other Issues ------------ @@ -299,3 +297,7 @@ backup requests randomly. Conclusion ---------- + +This proposal allows users to securely and conveniently back up and restore +their decryption keys so that users logging into a new device can decrypt old +messages. From 9d51d1e8b70a63fd36328018fde731f26f6b9b71 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Sat, 20 Oct 2018 13:45:35 -0400 Subject: [PATCH 08/36] expand the background --- .../1219-storing-megolm-keys-serverside.md | 26 ++++++++++++++++--- 1 file changed, 22 insertions(+), 4 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 14149950a60..905cd264c87 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -4,10 +4,28 @@ Storing megolm keys serverside Background ---------- -We *optionally* let clients store a copy of their megolm inbound session keys -on the HS so that they can recover history if all devices are lost without an -explicit key export; fix UISIs; support clients with limited local storage for -keys. +A user who uses end-to-end encyrption will usually have many inbound session +keys. Users who log into new devices and want to read old messages will need a +convenient way to transfer the session keys from one device to another. While +users can currently export their keys from one device and import them to +another, this is involves several steps and may be cumbersome for many users. +Users can also share keys from one device to another, but this has several +limitations, such as the fact that key shares only share one key at a time, and +require another logged-in device to be active. + +To help resolve this, we *optionally* let clients store an encrypted copy of +their megolm inbound session keys on the homeserver. Clients can keep the +backup up to date, so that users will always have the keys needed to decrypt +their conversations. The backup could be used not just for new logins, but +also to try to fix UISIs that occur after a device has logged in (as an +alternative to key sharing), or to support clients with limited local storage +for keys (clients can store old keys to the backup, and remove their local +copy, retrieving the key from the backup when needed). + +To recover keys from the backup, a user will need to enter a recovery key to +decrypt the backup. The backup will be encrypted using public key +cryptography, so that any of a user's devices can back up keys without needing +the user to enter the recovery key until they need to read from the backup. See also: From c8eac3ee2dfb7751640fb80688765484631947c9 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 24 Oct 2018 14:48:02 -0400 Subject: [PATCH 09/36] add details on how the encryption is done --- .../1219-storing-megolm-keys-serverside.md | 72 ++++++++++++------- 1 file changed, 45 insertions(+), 27 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 905cd264c87..68e23d68e7a 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -134,10 +134,8 @@ Body parameters: - `algorithm` (string): Required. The algorithm used for storing backups. Currently, only `m.megolm_backup.v1.curve25519-aes-sha2` is defined. - `auth_data` (object): Required. algorithm-dependent data. For - `m.megolm_backup.v1.curve25519-aes-sha2`, this is a signedjson object with - the following keys: - - `public_key` (string): the curve25519 public key used to encrypt the backups - - `signatures` (object): signatures of the public key + `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the definition of + this property. Example: @@ -198,14 +196,8 @@ Body parameters: - `is_verified` (boolean): Whether the device backing up the key has verified the device that the key is from. - `session_data` (object): Algorithm-dependent data. For - `m.megolm_backup.v1.curve25519-aes-sha2`, this is an object with the - following keys (TODO: this stuff should be moved): - - `ciphertext` (string): the encrypted version of the session key, as an - unpadded base64 string. See below for how the session key is encoded. - - `ephemeral` (string): the public ephemeral curve25519 key that was used to - encrypt the session key, as an unpadded base64 string. - - `mac` (string): the message authentication code for the ciphertext. FIXME: - more details + `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the definition of + this property. On success, returns the empty JSON object. @@ -277,21 +269,47 @@ Deletes keys from the backup. On success, returns the empty JSON object. -#### `m.megolm_backup.v1.curve25519-aes-sha2` Key format - -Each session key is encoded as a JSON object with the properties: - -- `algorithm` (string): `m.megolm.v1.aes-sha2` -- `sender_key` (string): base64-encoded device curve25519 key -- `sender_claimed_keys` (object): object containing the identity keys for the - sending device -- `forwardingCurve25519KeyChain` (array): zero or more curve25519 keys for - devices who forwarded the session key -- `session_key` (string): base64-encoded (unpadded) session key - -The JSON object is then encrypted by generating an ephemeral curve25519 key, -performing an ECDH with the ephemeral key and the backup's public key to -generate an AES key, and encrypting the stringified object using AES. +#### `m.megolm_backup.v1.curve25519-aes-sha2` definitions + +##### `auth_data` for backup versions + +The `auth_data` property for the backup versions endpoints for +`m.megolm_backup.v1.curve25519-aes-sha2` is a signedjson object with the +followin keys: + +- `public_key` (string): the curve25519 public key used to encrypt the backups +- `signatures` (object): signatures of the public key + +##### `session_data` for key backups + +The `session_data` field in the backups is constructed as follows: + +1. Encode the session key to be backed up as a JSON object with the properties: + - `algorithm` (string): `m.megolm.v1.aes-sha2` + - `sender_key` (string): base64-encoded device curve25519 key + - `sender_claimed_keys` (object): object containing the identity keys for the + sending device + - `forwardingCurve25519KeyChain` (array): zero or more curve25519 keys for + devices who forwarded the session key + - `session_key` (string): base64-encoded (unpadded) session key +2. Generate an ephemeral curve25519 key, and perform an ECDH with the ephemeral + key and the backup's public key to generate a shared secret. The public + half of the ephemeral key, encoded using base64, becomes the `ephemeral` + property of the `session_data`. +3. Using the shared secret, generate 80 bytes by performing an HKDF using + SHA-256 as the hash, with a salt of 32 bytes of 0, and with the empty string + as the info. The first 32 bytes are used as the AES key, the next 32 bytes + are used as the MAC key, and the last 16 bytes are used as the AES + initialization vector. +4. Stringify the JSON object, and encrypt it using AES-CBC-256 with PKCS#7 + padding. This encrypted data, encoded using base64, becomes the + `ciphertext` property of the `session_data`. +5. Pass the raw encrypted data (prior to base64 encoding) through HMAC-SHA-256 + using the MAC key generated above. The first 8 bytes of the resulting MAC + are base64-encoded, and become the `mac` property of the `session_data`. + +(The key HKDF, AES, and HMAC steps are the same as what are used for encryption +in olm and megolm.) Security Considerations ----------------------- From dc0dd18eebbe911f76e8c306d8b13fbb0210c4ea Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Thu, 25 Oct 2018 13:49:47 -0400 Subject: [PATCH 10/36] note that version is optional for GET, and say what to do when no keys are found --- .../1219-storing-megolm-keys-serverside.md | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 68e23d68e7a..9936a6cd9a6 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -238,6 +238,9 @@ Returns the same as `PUT #### Retrieving keys +When retrieving keys, the `version` parameter is optional, and defaults to +retrieving the latest backup version. + ##### `GET /room_keys/keys/${roomId}/${sessionId}?version=$v` Retrieve the key for the given session in the given room from the backup. @@ -245,6 +248,10 @@ Retrieve the key for the given session in the given room from the backup. On success, returns a JSON object in the same form as the request body of `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v`. +Error codes: + +- M_NOT_FOUND: The session is not present in the backup. + ##### `GET /room_keys/keys/${roomId}?version=$v` Retrieve the all the keys for the given room from the backup. @@ -252,6 +259,15 @@ Retrieve the all the keys for the given room from the backup. On success, returns a JSON object in the same form as the request body of `PUT /room_keys/keys/${roomId}?version=$v`. +If no keys are found, then this endpoint returns a successful response with +body: + +``` +{ + "sessions": {} +} +``` + ##### `GET /room_keys/keys?version=$v` Retrieve all the keys from the backup. @@ -259,6 +275,16 @@ Retrieve all the keys from the backup. On success, returns a JSON object in the same form as the request body of `PUT /room_keys/keys?version=$v`. + +If no keys are found, then this endpoint returns a successful response with +body: + +``` +{ + "rooms": {} +} +``` + #### Deleting keys ##### `DELETE /room_keys/keys/${roomId}/${sessionId}?version=$v` From 7b4b4a26886dec39be17816cf4d7ea3c85544ba9 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 30 Oct 2018 00:10:05 -0400 Subject: [PATCH 11/36] fix some English and some minor additions --- .../1219-storing-megolm-keys-serverside.md | 20 +++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 9936a6cd9a6..800b3814e7f 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -56,6 +56,9 @@ trust that the backup was not created by a malicious device. ### Possible UX for interactive clients +This section gives an example of how a client might handle key backups. Clients +may behave differently. + On receipt of encryption keys (1st time): 1. client checks if there is an existing backup: `GET /room_keys/version` @@ -63,7 +66,7 @@ On receipt of encryption keys (1st time): 1. if yes: 1. generate new curve25519 key pair 2. create new backup version: `POST /room_keys/version` - 3. display private key to user to save (see below for the format) + 3. display private key for user to save (see below for the format) 2. if no, exit and remember decision (user can change their mind later) 3. while prompting, continue to poll `GET /room_keys/versions`, as another device may have created a backup. If so, go to 1.2. @@ -99,10 +102,10 @@ settings. ### Recovery key -The recovery key is can either be saved by the user directly, or stored -encrypted on the server (as proposed in +The recovery key can either be saved by the user directly, or stored encrypted +on the server (as proposed in [MSC1687](https://github.com/matrix-org/matrix-doc/issues/1687)). If the key -is saved directly by the user, then it the code is constructed as follows: +is saved directly by the user, then the code is constructed as follows: 1. The 256-bit curve25519 private key is prepended by the bytes `0x8B` and `0x01` @@ -112,7 +115,8 @@ is saved directly by the user, then it the code is constructed as follows: for Bitcoin addresses. This 58-character string is presented to the user to save. Implementations may -add whitespace to the recovery key. +add whitespace to the recovery key; adding a space every 4th character is +recommended. When reading in a recovery key, clients must disregard whitespace. Clients must base58-decode the code, ensure that the first two bytes of the decoded @@ -193,9 +197,9 @@ Body parameters: in the session that the key can decrypt. - `forwarded_count` (integer): Required. The number of times this key has been forwarded. -- `is_verified` (boolean): Whether the device backing up the key has verified - the device that the key is from. -- `session_data` (object): Algorithm-dependent data. For +- `is_verified` (boolean): Required. Whether the device backing up the key has + verified the device that the key is from. +- `session_data` (object): Required. Algorithm-dependent data. For `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the definition of this property. From 982abc168a47ef6227e64a45f3af6e6489d11c86 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 30 Oct 2018 00:12:26 -0400 Subject: [PATCH 12/36] add some examples --- .../1219-storing-megolm-keys-serverside.md | 81 ++++++++++++++++++- 1 file changed, 80 insertions(+), 1 deletion(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 800b3814e7f..6fb5c905de2 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -210,6 +210,28 @@ Error codes: - `M_WRONG_ROOM_KEYS_VERSION`: the version specified does not match the current backup version +Example: + +`PUT /room_keys/keys/!room_id:example.com/sessionid?version=1` + +```javascript +{ + "first_message_index": 1, + "forwarded_count": 0, + "is_verified": true, + "session_data": { + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" + } +} +``` + +Result: + +```javascript +{} +``` ##### `PUT /room_keys/keys/${roomId}?version=$v` Store several keys for the given room, using the given backup version. @@ -223,8 +245,34 @@ Body parameters: /room_keys/keys/${roomId}/${sessionId}?version=$v`. Returns the same as `PUT -/room_keys/keys/${roomId}/${sessionId}?version=$v` +/room_keys/keys/${roomId}/${sessionId}?version=$v`. + +Example: + +`PUT /room_keys/keys/!room_id:example.com?version=1` +```javascript +{ + "sessions": { + "sessionid": { + "first_message_index": 1, + "forwarded_count": 0, + "is_verified": true, + "session_data": { + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" + } + } + } +} +``` + +Result: + +```javascript +{} +``` ##### `PUT /room_keys/keys?version=$v` Store several keys, using the given backup version. @@ -240,6 +288,37 @@ Body parameters: Returns the same as `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v` +Example: + +`PUT /room_keys/keys/!room_id:example.com?version=1` + +```javascript +{ + "rooms": { + "!room_id:example.com": { + "sessions": { + "sessionid": { + "first_message_index": 1, + "forwarded_count": 0, + "is_verified": true, + "session_data": { + "ephemeral": "base64+ephemeral+key", + "ciphertext": "base64+ciphertext+of+JSON+data", + "mac": "base64+mac+of+ciphertext" + } + } + } + } + } +} +``` + +Result: + +```javascript +{} +``` + #### Retrieving keys When retrieving keys, the `version` parameter is optional, and defaults to From 7713a0f402667b3e73efc1edcfac22537de64c5d Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 30 Oct 2018 10:05:27 -0400 Subject: [PATCH 13/36] snake-case for consistency --- proposals/1219-storing-megolm-keys-serverside.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 6fb5c905de2..194d50d108e 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -398,8 +398,8 @@ The `session_data` field in the backups is constructed as follows: - `sender_key` (string): base64-encoded device curve25519 key - `sender_claimed_keys` (object): object containing the identity keys for the sending device - - `forwardingCurve25519KeyChain` (array): zero or more curve25519 keys for - devices who forwarded the session key + - `forwarding_curve25519_key_chain` (array): zero or more curve25519 keys + for devices who forwarded the session key - `session_key` (string): base64-encoded (unpadded) session key 2. Generate an ephemeral curve25519 key, and perform an ECDH with the ephemeral key and the backup's public key to generate a shared secret. The public From 3918ed3c38389ee3058742412fb17e9ef354773a Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 30 Oct 2018 14:09:40 -0400 Subject: [PATCH 14/36] distinguish between retrieving an empty backup and a nonexistent backup --- .../1219-storing-megolm-keys-serverside.md | 20 +++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 194d50d108e..18f6f877815 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -333,7 +333,8 @@ On success, returns a JSON object in the same form as the request body of `PUT Error codes: -- M_NOT_FOUND: The session is not present in the backup. +- M_NOT_FOUND: The session is not present in the backup, or the requested + backup version does not exist. ##### `GET /room_keys/keys/${roomId}?version=$v` @@ -342,8 +343,8 @@ Retrieve the all the keys for the given room from the backup. On success, returns a JSON object in the same form as the request body of `PUT /room_keys/keys/${roomId}?version=$v`. -If no keys are found, then this endpoint returns a successful response with -body: +If the backup version exists but no keys are found, then this endpoint returns +a successful response with body: ``` { @@ -351,6 +352,10 @@ body: } ``` +Error codes: + +- `M_NOT_FOUND`: The requested backup version does not exist. + ##### `GET /room_keys/keys?version=$v` Retrieve all the keys from the backup. @@ -358,9 +363,8 @@ Retrieve all the keys from the backup. On success, returns a JSON object in the same form as the request body of `PUT /room_keys/keys?version=$v`. - -If no keys are found, then this endpoint returns a successful response with -body: +If the backup version exists but no keys are found, then this endpoint returns +a successful response with body: ``` { @@ -368,6 +372,10 @@ body: } ``` +Error codes: + +- `M_NOT_FOUND`: The requested backup version does not exist. + #### Deleting keys ##### `DELETE /room_keys/keys/${roomId}/${sessionId}?version=$v` From 2dce23564ff640f1e9e71235126db0aca7b2adef Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 13 Nov 2018 21:37:50 -0500 Subject: [PATCH 15/36] wording fixes --- proposals/1219-storing-megolm-keys-serverside.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 18f6f877815..527dfe8fe1e 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -4,7 +4,7 @@ Storing megolm keys serverside Background ---------- -A user who uses end-to-end encyrption will usually have many inbound session +A user who uses end-to-end encyrption will usually have many inbound group session keys. Users who log into new devices and want to read old messages will need a convenient way to transfer the session keys from one device to another. While users can currently export their keys from one device and import them to @@ -102,10 +102,10 @@ settings. ### Recovery key -The recovery key can either be saved by the user directly, or stored encrypted -on the server (as proposed in -[MSC1687](https://github.com/matrix-org/matrix-doc/issues/1687)). If the key -is saved directly by the user, then the code is constructed as follows: +The recovery key can be saved by the user directly, stored encrypted on the +server (as proposed in +[MSC1687](https://github.com/matrix-org/matrix-doc/issues/1687)), or both. If +the key is saved directly by the user, then the code is constructed as follows: 1. The 256-bit curve25519 private key is prepended by the bytes `0x8B` and `0x01` From b45cf4483fd9fbd06f2f5963b31bfef4e2286fae Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Tue, 13 Nov 2018 21:46:07 -0500 Subject: [PATCH 16/36] providing an alternative to key sharing is currently a non-goal --- proposals/1219-storing-megolm-keys-serverside.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 527dfe8fe1e..4d8e8f1c0e3 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -17,10 +17,9 @@ To help resolve this, we *optionally* let clients store an encrypted copy of their megolm inbound session keys on the homeserver. Clients can keep the backup up to date, so that users will always have the keys needed to decrypt their conversations. The backup could be used not just for new logins, but -also to try to fix UISIs that occur after a device has logged in (as an -alternative to key sharing), or to support clients with limited local storage -for keys (clients can store old keys to the backup, and remove their local -copy, retrieving the key from the backup when needed). +also to support clients with limited local storage for keys (clients can store +old keys to the backup, and remove their local copy, retrieving the key from +the backup when needed). To recover keys from the backup, a user will need to enter a recovery key to decrypt the backup. The backup will be encrypted using public key From c9b38cbe5310c8f0a8a637d4ac6ed57dcf1fb8e1 Mon Sep 17 00:00:00 2001 From: manuroe Date: Wed, 6 Feb 2019 11:37:19 +0100 Subject: [PATCH 17/36] Key backup: add `PUT /room_keys/version/{version}` to allow matrix clients to add signatures to an existing backup --- .../1219-storing-megolm-keys-serverside.md | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 4d8e8f1c0e3..4b1210544a2 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -177,6 +177,42 @@ Error codes: - `M_NOT_FOUND`: No backup version has been created. +##### `PUT /room_keys/version/{version}` + +Update information about the given version, or the current version if `{version}` +is omitted. Only `signatures` in `auth_data` can be updated. + +Body parameters: + +- `algorithm` (string): Optional. Must be the same as in the body parameters for `GET + /room_keys/version`. +- `auth_data` (object): Required. algorithm-dependent data. For + `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the definition of + this property. +- `version` (string): Optional. The backup version. Must be the same as the query parameter or must be the current version. + +Example: + +```javascript +{ + "auth_data": { + "public_key": "abcdefg", + "signatures": { + "something": { + "ed25519:something": "hijklmnop" + "ed25519:anotherthing": "abcdef" + } + } + } +} +``` + +On success, returns the empty JSON object. + +Error codes: + +- `M_NOT_FOUND`: No backup version has been created. + #### Storing keys ##### `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v` From e02b345c623210a718e13ac65afd66a120d90711 Mon Sep 17 00:00:00 2001 From: manuroe Date: Wed, 6 Feb 2019 11:45:11 +0100 Subject: [PATCH 18/36] Revert "Key backup: add `PUT /room_keys/version/{version}` to allow matrix clients to add signatures to an existing backup" This reverts commit c9b38cbe5310c8f0a8a637d4ac6ed57dcf1fb8e1. --- .../1219-storing-megolm-keys-serverside.md | 36 ------------------- 1 file changed, 36 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 4b1210544a2..4d8e8f1c0e3 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -177,42 +177,6 @@ Error codes: - `M_NOT_FOUND`: No backup version has been created. -##### `PUT /room_keys/version/{version}` - -Update information about the given version, or the current version if `{version}` -is omitted. Only `signatures` in `auth_data` can be updated. - -Body parameters: - -- `algorithm` (string): Optional. Must be the same as in the body parameters for `GET - /room_keys/version`. -- `auth_data` (object): Required. algorithm-dependent data. For - `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the definition of - this property. -- `version` (string): Optional. The backup version. Must be the same as the query parameter or must be the current version. - -Example: - -```javascript -{ - "auth_data": { - "public_key": "abcdefg", - "signatures": { - "something": { - "ed25519:something": "hijklmnop" - "ed25519:anotherthing": "abcdef" - } - } - } -} -``` - -On success, returns the empty JSON object. - -Error codes: - -- `M_NOT_FOUND`: No backup version has been created. - #### Storing keys ##### `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v` From 2099308d4cf1b733b88976f535590932f9b0eafb Mon Sep 17 00:00:00 2001 From: manuroe Date: Wed, 6 Feb 2019 11:50:47 +0100 Subject: [PATCH 19/36] Key backup: add `PUT /room_keys/version/{version}` to allow matrix clients to add signatures to an existing backup --- .../1219-storing-megolm-keys-serverside.md | 36 +++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 4d8e8f1c0e3..ae361d3a780 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -177,6 +177,42 @@ Error codes: - `M_NOT_FOUND`: No backup version has been created. +##### `PUT /room_keys/version/{version}` + +Update information about the given version, or the current version if `{version}` +is omitted. Only `auth_data` can be updated. + +Body parameters: + +- `algorithm` (string): Optional. Must be the same as in the body parameters for `GET + /room_keys/version`. +- `auth_data` (object): Required. algorithm-dependent data. For + `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the definition of + this property. +- `version` (string): Optional. The backup version. Must be the same as the query parameter or must be the current version. + +Example: + +```javascript +{ + "auth_data": { + "public_key": "abcdefg", + "signatures": { + "something": { + "ed25519:something": "hijklmnop" + "ed25519:anotherthing": "abcdef" + } + } + } +} +``` + +On success, returns the empty JSON object. + +Error codes: + +- `M_NOT_FOUND`: No backup version found. + #### Storing keys ##### `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v` From d43b595b5e4ec966508b09af010c3273336f10cf Mon Sep 17 00:00:00 2001 From: manuroe Date: Thu, 7 Feb 2019 10:36:34 +0100 Subject: [PATCH 20/36] Key backup: Fix PR remarks on `PUT /room_keys/version/{version}` --- proposals/1219-storing-megolm-keys-serverside.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index ae361d3a780..f4f45827503 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -179,17 +179,16 @@ Error codes: ##### `PUT /room_keys/version/{version}` -Update information about the given version, or the current version if `{version}` -is omitted. Only `auth_data` can be updated. +Update information about the given version. Only `auth_data` can be updated. Body parameters: -- `algorithm` (string): Optional. Must be the same as in the body parameters for `GET +- `algorithm` (string): Required. Must be the same as in the body parameters for `GET /room_keys/version`. - `auth_data` (object): Required. algorithm-dependent data. For `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the definition of this property. -- `version` (string): Optional. The backup version. Must be the same as the query parameter or must be the current version. +- `version` (string): Required. The backup version. Must be the same as the query parameter or must be the current version. Example: From e7f792602348f06f3c726f72902a127df0d7269e Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Thu, 7 Feb 2019 23:29:11 -0500 Subject: [PATCH 21/36] add algorithm and version to the example since they're marked as required --- proposals/1219-storing-megolm-keys-serverside.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index f4f45827503..74a148a25d3 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -194,6 +194,7 @@ Example: ```javascript { + "algorithm": "m.megolm_backup.v1.curve25519-aes-sha2", "auth_data": { "public_key": "abcdefg", "signatures": { @@ -202,7 +203,8 @@ Example: "ed25519:anotherthing": "abcdef" } } - } + }, + "version": "42" } ``` From ed945d67444a5c011bd55d6cfbcf39ee3812e8bd Mon Sep 17 00:00:00 2001 From: manuroe Date: Wed, 6 Feb 2019 12:03:44 +0100 Subject: [PATCH 22/36] Key backup: Expose the number of keys stored in the backup so that matrix clients can compare it with the number of keys they have locally. --- proposals/1219-storing-megolm-keys-serverside.md | 1 + 1 file changed, 1 insertion(+) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 74a148a25d3..1f33fb45958 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -172,6 +172,7 @@ On success, returns a JSON object with keys: - `auth_data` (object): Required. Same as in the body parameters for `POST /room_keys/version`. - `version` (string): Required. The backup version. +- `count` (number): Required. The number of keys stored in the backup. Error codes: From 82ff866b5890e1e7132c5bdfc603c79ce92d7964 Mon Sep 17 00:00:00 2001 From: manuroe Date: Wed, 6 Feb 2019 12:30:21 +0100 Subject: [PATCH 23/36] Key backup: Add `hash` to represent stored keys so that a matrix client A can check it is synchronised with the backup. If not, that means that another client B has pushed keys client A does not have locally. Client A should then propose to the end user to retrieve keys from the backup. --- .../1219-storing-megolm-keys-serverside.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 1f33fb45958..22e57968636 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -172,6 +172,7 @@ On success, returns a JSON object with keys: - `auth_data` (object): Required. Same as in the body parameters for `POST /room_keys/version`. - `version` (string): Required. The backup version. +- `hash` (string): Required. A hash value representing stored keys. - `count` (number): Required. The number of keys stored in the backup. Error codes: @@ -240,7 +241,9 @@ Body parameters: `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the definition of this property. -On success, returns the empty JSON object. +On success, returns a JSON object with keys: + +- `hash` (string): Required. The new hash value representing stored keys. Error codes: @@ -267,8 +270,11 @@ Example: Result: ```javascript -{} +{ + "hash": "abcdefghi" +} ``` + ##### `PUT /room_keys/keys/${roomId}?version=$v` Store several keys for the given room, using the given backup version. @@ -308,8 +314,11 @@ Example: Result: ```javascript -{} +{ + "hash": "abcdefghi" +} ``` + ##### `PUT /room_keys/keys?version=$v` Store several keys, using the given backup version. @@ -353,7 +362,9 @@ Example: Result: ```javascript -{} +{ + "hash": "abcdefghi" +} ``` #### Retrieving keys From 7cde3193e535aa3510fc008fb8ce55617815a194 Mon Sep 17 00:00:00 2001 From: manuroe Date: Thu, 7 Feb 2019 10:29:30 +0100 Subject: [PATCH 24/36] Key backup: Explain `hash` better --- proposals/1219-storing-megolm-keys-serverside.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 22e57968636..069021c86c0 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -172,7 +172,10 @@ On success, returns a JSON object with keys: - `auth_data` (object): Required. Same as in the body parameters for `POST /room_keys/version`. - `version` (string): Required. The backup version. -- `hash` (string): Required. A hash value representing stored keys. +- `hash` (string): Required. The hash value which is an opaque string + representing stored keys in the backup. Client can compare it with the `hash` + value they received in the response of their last key storage request. + If not equal, another matrix client pushed new keys to the backup. - `count` (number): Required. The number of keys stored in the backup. Error codes: @@ -243,7 +246,8 @@ Body parameters: On success, returns a JSON object with keys: -- `hash` (string): Required. The new hash value representing stored keys. +- `hash` (string): Required. The new hash value representing stored keys. See +`GET /room_keys/version/{version}` for more details. Error codes: From 0051c6a377ca9fa724f0848b6d2cb58dd1680660 Mon Sep 17 00:00:00 2001 From: manuroe Date: Thu, 7 Feb 2019 22:54:53 +0100 Subject: [PATCH 25/36] Key backup: Return {hash, count} for key upload requests This is this tuple that allows the client to check if it has locally all keys of the backup --- proposals/1219-storing-megolm-keys-serverside.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 069021c86c0..05c68520ed9 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -248,6 +248,7 @@ On success, returns a JSON object with keys: - `hash` (string): Required. The new hash value representing stored keys. See `GET /room_keys/version/{version}` for more details. +- `count` (number): Required. The new count of keys stored in the backup. Error codes: @@ -275,7 +276,8 @@ Result: ```javascript { - "hash": "abcdefghi" + "hash": "abcdefghi", + "count": 10 } ``` @@ -319,7 +321,8 @@ Result: ```javascript { - "hash": "abcdefghi" + "hash": "abcdefghi", + "count": 10 } ``` @@ -367,7 +370,8 @@ Result: ```javascript { - "hash": "abcdefghi" + "hash": "abcdefghi", + "count": 10 } ``` From 87824c1c963f13dfe6072e76fe372eb18c81d65c Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Fri, 15 Mar 2019 13:38:19 -0400 Subject: [PATCH 26/36] Update proposals/1219-storing-megolm-keys-serverside.md Co-Authored-By: uhoreg --- proposals/1219-storing-megolm-keys-serverside.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 05c68520ed9..0863bc8d607 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -4,7 +4,7 @@ Storing megolm keys serverside Background ---------- -A user who uses end-to-end encyrption will usually have many inbound group session +A user who uses end-to-end encryption will usually have many inbound group session keys. Users who log into new devices and want to read old messages will need a convenient way to transfer the session keys from one device to another. While users can currently export their keys from one device and import them to From 1c4262e556108d0b1ada17c722659a6d010e5dff Mon Sep 17 00:00:00 2001 From: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Date: Fri, 15 Mar 2019 13:40:44 -0400 Subject: [PATCH 27/36] Apply suggestions from code review Co-Authored-By: uhoreg --- proposals/1219-storing-megolm-keys-serverside.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 0863bc8d607..09ded1cd640 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -162,7 +162,7 @@ On success, returns a JSON object with keys: ##### `GET /room_keys/version/{version}` -Get information about the given version, or the current version if `{version}` +Get information about the given version, or the current version if `/{version}` is omitted. On success, returns a JSON object with keys: @@ -448,7 +448,7 @@ On success, returns the empty JSON object. The `auth_data` property for the backup versions endpoints for `m.megolm_backup.v1.curve25519-aes-sha2` is a signedjson object with the -followin keys: +following keys: - `public_key` (string): the curve25519 public key used to encrypt the backups - `signatures` (object): signatures of the public key From 825757ffd8c9f38f588b4d6dadf0a826675ff601 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 31 Jul 2019 16:37:54 -0400 Subject: [PATCH 28/36] add information about verifying backup by entering key --- proposals/1219-storing-megolm-keys-serverside.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 09ded1cd640..e1ff0740487 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -493,7 +493,10 @@ key backup. This proposal does not attempt to protect against that. An attacker who gains access to a user's account can create a new backup version using a key that they control. For this reason, clients SHOULD confirm with users before sending keys to a new backup version or verify that it was -created by a trusted device by checking the signature. +created by a trusted device by checking the signature. One way to confirm the +new backup version if the signature cannot be checked is by asking the user to +enter the recovery key, and confirming that the backup's public key matches +what is expected. Other Issues ------------ From 80adbaff4c69ae342f16c94894e0b46006a77597 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 31 Jul 2019 16:38:20 -0400 Subject: [PATCH 29/36] switch to MSC1946 for storing recovery key --- .../1219-storing-megolm-keys-serverside.md | 27 +++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index e1ff0740487..7e074a0d50c 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -102,8 +102,8 @@ settings. ### Recovery key The recovery key can be saved by the user directly, stored encrypted on the -server (as proposed in -[MSC1687](https://github.com/matrix-org/matrix-doc/issues/1687)), or both. If +server (using the method proposed in +[MSC1946](https://github.com/matrix-org/matrix-doc/issues/1946)), or both. If the key is saved directly by the user, then the code is constructed as follows: 1. The 256-bit curve25519 private key is prepended by the bytes `0x8B` and @@ -124,6 +124,29 @@ results in 0, and ensure that the total length of the decoded string is 35 bytes. Clients must then remove the first two bytes and the last byte, and use the resulting string as the private key to decrypt backups. +If MSC1946 is used to store the key on the server, it must be stored using the +`account_data` `type` `m.megolm_backup.v1`. + +As a special case, if the recovery key is the same as the curve25519 key used +for storing the key, then the contents of the `m.megolm_backup.v1` +`account_data` for that key will be the an object with a `passthrough` property +whose value is `true`. For example, if `m.megolm_backup.v1` is set to: + +```json +{ + "encrypted": { + "key_id": { + "passthrough": true + } + } +} +``` + +means that the recovery key for the backup is the same as the private key for +the key with ID `key_id`. (This is mostly intended to provide a migration path +for for backups that were created using an earlier draft that stored the +recovery information in the `auth_data`.) + ### API #### Backup versions From 7ed536751650b2762e370d1fcdbc7ad6aa0410d9 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Sat, 10 Aug 2019 14:14:30 -0700 Subject: [PATCH 30/36] clarifications, fix formatting --- .../1219-storing-megolm-keys-serverside.md | 31 ++++++++++++------- 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 7e074a0d50c..dbcf309196d 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -195,10 +195,10 @@ On success, returns a JSON object with keys: - `auth_data` (object): Required. Same as in the body parameters for `POST /room_keys/version`. - `version` (string): Required. The backup version. -- `hash` (string): Required. The hash value which is an opaque string - representing stored keys in the backup. Client can compare it with the `hash` - value they received in the response of their last key storage request. - If not equal, another matrix client pushed new keys to the backup. +- `hash` (string): Required. The hash value which is an opaque string + representing stored keys in the backup. Client can compare it with the `hash` + value they received in the response of their last key storage request. + If not equal, another matrix client pushed new keys to the backup. - `count` (number): Required. The number of keys stored in the backup. Error codes: @@ -212,7 +212,7 @@ Update information about the given version. Only `auth_data` can be updated. Body parameters: - `algorithm` (string): Required. Must be the same as in the body parameters for `GET - /room_keys/version`. + /room_keys/version`. - `auth_data` (object): Required. algorithm-dependent data. For `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the definition of this property. @@ -270,7 +270,7 @@ Body parameters: On success, returns a JSON object with keys: - `hash` (string): Required. The new hash value representing stored keys. See -`GET /room_keys/version/{version}` for more details. + `GET /room_keys/version/{version}` for more details. - `count` (number): Required. The new count of keys stored in the backup. Error codes: @@ -470,11 +470,18 @@ On success, returns the empty JSON object. ##### `auth_data` for backup versions The `auth_data` property for the backup versions endpoints for -`m.megolm_backup.v1.curve25519-aes-sha2` is a signedjson object with the +`m.megolm_backup.v1.curve25519-aes-sha2` is a [signed +json](https://matrix.org/docs/spec/appendices#signing-json) object with the following keys: - `public_key` (string): the curve25519 public key used to encrypt the backups -- `signatures` (object): signatures of the public key +- `signatures` (object): signatures of the `auth_data`. + +The `auth_data` should be signed by the user's [master cross-signing +key](https://github.com/matrix-org/matrix-doc/pull/1756), and may also be +signed by the user's device key. The allows clients to ensure that the public +key is valid, and prevents an attacker from being able to change the backup to +use a public key that have the private key for. ##### `session_data` for key backups @@ -516,10 +523,10 @@ key backup. This proposal does not attempt to protect against that. An attacker who gains access to a user's account can create a new backup version using a key that they control. For this reason, clients SHOULD confirm with users before sending keys to a new backup version or verify that it was -created by a trusted device by checking the signature. One way to confirm the -new backup version if the signature cannot be checked is by asking the user to -enter the recovery key, and confirming that the backup's public key matches -what is expected. +created by a trusted device by checking the signature. Alternatively, if the +signature cannot be verified, the backup can be validated by prompting the user +to enter the recovery key, and confirming that the backup's public key +corresponds to the recovery key. Other Issues ------------ From cf953c47fd799af7573bc257fce6fd6db47c5d02 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 9 Sep 2019 17:29:32 -0400 Subject: [PATCH 31/36] clarifications, change "hash" to "etag" --- .../1219-storing-megolm-keys-serverside.md | 112 ++++++++++-------- 1 file changed, 63 insertions(+), 49 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index dbcf309196d..f51ed5f941e 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -38,20 +38,26 @@ Proposal -------- This proposal creates new APIs to allow clients to back up room decryption keys -on the server. Decryption keys are encrypted (using public key crypto) before -being sent to the server along with some unencrypted metadata to allow the -server to manage the backups, overwriting backups with "better" versions of the -keys. The user is given a private recovery key to save for recovering the keys -from the backup. - -Clients can create new versions of backups. Aside from the initial backup -creation, a client might start a new version of a backup when, for example, a -user loses a device, and wants to ensure that that device does not get any new -decryption keys. - -Once one client has created a backup version, other clients can fetch the -public key for the backup from the server and add keys to the backup, if they -trust that the backup was not created by a malicious device. +on the server. Room decryption keys are encrypted (using public key crypto) +before being sent to the server along with some unencrypted metadata to allow +the server to manage the backups. If a key for a new megolm session is +uploaded, it is added to the current backup. If a key is uploaded for a megolm +session is that is already present in the backup, the server will use the +metadata to determine which version of the key is "better". The way in which +the server determines which key is "better" is described in the [Storing +Keys](#storing-keys) section. The user is given a private recovery key in +order to recover the keys from the backup in the future. + +Clients can create new key backups (sometimes also referred to in the API as +backup versions) to replace the current backup. Aside from the initial backup +creation, a client might start a new a backup when, for example, a user loses a +device and wants to ensure that that device does not get any new decryption +keys. In this case, the client will then create a new backup using a new key +that the device does not have access to. + +Once one client has created a backup, other clients can fetch the public part +of the recovery key from the server and add keys to the backup, if they trust +that the backup was not created by a malicious device. ### Possible UX for interactive clients @@ -63,31 +69,30 @@ On receipt of encryption keys (1st time): 1. client checks if there is an existing backup: `GET /room_keys/version` 1. if not, ask if the user wants to back up keys 1. if yes: - 1. generate new curve25519 key pair - 2. create new backup version: `POST /room_keys/version` - 3. display private key for user to save (see below for the format) + 1. generate new curve25519 key pair, which will be the recovery key + 2. create new backup: `POST /room_keys/version` + 3. display private key for user to save (see below for the + [format of the recovery key](#recovery-key)) 2. if no, exit and remember decision (user can change their mind later) 3. while prompting, continue to poll `GET /room_keys/versions`, as another device may have created a backup. If so, go to 1.2. - 2. if yes, get public key, prompt user to verify a device that signed the - key¹, or enter recovery key (which can derive the backup key). + 2. if yes, either get the public part of the recovery key and check that it + is signed by the master cross-signing key, or prompt user to enter the + private part of the recovery key (which can derive the public part). 1. User can also decide to create a new backup, in which case, go to 1.1. 2. send key to backup: `PUT /room_keys/keys/${roomId}/${sessionId}?version=$v` 3. continue backing up keys as we receive them (may receive a - `M_WRONG_ROOM_KEYS_VERSION` error if a new backup version has been created: + `M_WRONG_ROOM_KEYS_VERSION` error if a new backup has been created: see below) On `M_WRONG_ROOM_KEYS_VERSION` error when trying to `PUT` keys: 1. get the current version -2. notify the user that there is a new backup version, and display relevant - information +2. notify the user that there is a new backup, and display relevant information 3. confirm with user that they want to use the backup (user may want use the backup, to stop backing up keys, or to create a new backup) -4. verify the device that signed the backup key¹, or enter recovery key - -¹: cross-signing (when that is completed) can be used to verify the device -that signed the key. +4. ensure the backup key is signed by the user's master key, or prompt the user + to enter the recovery key On receipt of undecryptable message: @@ -125,7 +130,7 @@ is 35 bytes. Clients must then remove the first two bytes and the last byte, and use the resulting string as the private key to decrypt backups. If MSC1946 is used to store the key on the server, it must be stored using the -`account_data` `type` `m.megolm_backup.v1`. +`account_data` type `m.megolm_backup.v1`. As a special case, if the recovery key is the same as the curve25519 key used for storing the key, then the contents of the `m.megolm_backup.v1` @@ -160,8 +165,8 @@ Body parameters: - `algorithm` (string): Required. The algorithm used for storing backups. Currently, only `m.megolm_backup.v1.curve25519-aes-sha2` is defined. - `auth_data` (object): Required. algorithm-dependent data. For - `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the definition of - this property. + `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the [definition of + this property](#auth_data-backup-versions). Example: @@ -195,10 +200,10 @@ On success, returns a JSON object with keys: - `auth_data` (object): Required. Same as in the body parameters for `POST /room_keys/version`. - `version` (string): Required. The backup version. -- `hash` (string): Required. The hash value which is an opaque string - representing stored keys in the backup. Client can compare it with the `hash` - value they received in the response of their last key storage request. - If not equal, another matrix client pushed new keys to the backup. +- `etag` (string): Required. The etag value which is an opaque string + representing stored keys in the backup. Clients can compare it with the + `etag` value they received in the response of their last key storage request. + If not equal, another client has pushed new keys to the backup. - `count` (number): Required. The number of keys stored in the backup. Error codes: @@ -214,8 +219,8 @@ Body parameters: - `algorithm` (string): Required. Must be the same as in the body parameters for `GET /room_keys/version`. - `auth_data` (object): Required. algorithm-dependent data. For - `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the definition of - this property. + `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the [definition of + this property](#auth_data-backup-versions). - `version` (string): Required. The backup version. Must be the same as the query parameter or must be the current version. Example: @@ -251,9 +256,15 @@ version. If the server already has a backup in the backup version for the given session and room, then it will keep the "better" one. To determine which one is -"better", key backups are compared first by the `is_verified` flag (`true` is -better than `false`), then by the `first_message_index` (a lower number is better), -and finally by `forwarded_count` (a lower number is better). +"better", keys are compared: + +- first by the `is_verified` flag (`true` is better than `false`), +- then, if `is_verified` is equal, by the `first_message_index` (a lower number is better), +- and finally, is `is_verified` and `first_message_index` are equal, by + `forwarded_count` (a lower number is better). + +If neither key is better than the other (that is, if all three fields are +equal), then the server should keep the existing key. Body parameters: @@ -264,12 +275,12 @@ Body parameters: - `is_verified` (boolean): Required. Whether the device backing up the key has verified the device that the key is from. - `session_data` (object): Required. Algorithm-dependent data. For - `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the definition of - this property. + `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the [definition of + this property](#auth_data-backup-versions). On success, returns a JSON object with keys: -- `hash` (string): Required. The new hash value representing stored keys. See +- `etag` (string): Required. The new etag value representing stored keys. See `GET /room_keys/version/{version}` for more details. - `count` (number): Required. The new count of keys stored in the backup. @@ -299,7 +310,7 @@ Result: ```javascript { - "hash": "abcdefghi", + "etag": "abcdefghi", "count": 10 } ``` @@ -344,7 +355,7 @@ Result: ```javascript { - "hash": "abcdefghi", + "etag": "abcdefghi", "count": 10 } ``` @@ -393,7 +404,7 @@ Result: ```javascript { - "hash": "abcdefghi", + "etag": "abcdefghi", "count": 10 } ``` @@ -401,7 +412,7 @@ Result: #### Retrieving keys When retrieving keys, the `version` parameter is optional, and defaults to -retrieving the latest backup version. +retrieving keys from the latest backup version. ##### `GET /room_keys/keys/${roomId}/${sessionId}?version=$v` @@ -463,7 +474,8 @@ Error codes: Deletes keys from the backup. -On success, returns the empty JSON object. +Returns the same as `PUT +/room_keys/keys/${roomId}/${sessionId}?version=$v`. #### `m.megolm_backup.v1.curve25519-aes-sha2` definitions @@ -479,9 +491,9 @@ following keys: The `auth_data` should be signed by the user's [master cross-signing key](https://github.com/matrix-org/matrix-doc/pull/1756), and may also be -signed by the user's device key. The allows clients to ensure that the public +signed by the user's device key. This allows clients to ensure that the public key is valid, and prevents an attacker from being able to change the backup to -use a public key that have the private key for. +use a public key that they have the private key for. ##### `session_data` for key backups @@ -489,7 +501,9 @@ The `session_data` field in the backups is constructed as follows: 1. Encode the session key to be backed up as a JSON object with the properties: - `algorithm` (string): `m.megolm.v1.aes-sha2` - - `sender_key` (string): base64-encoded device curve25519 key + - `sender_key` (string): base64-encoded device curve25519 key in + [session-sharing + format](https://gitlab.matrix.org/matrix-org/olm/blob/master/docs/megolm.md#session-sharing-format) - `sender_claimed_keys` (object): object containing the identity keys for the sending device - `forwarding_curve25519_key_chain` (array): zero or more curve25519 keys From 8123c4ef0ff499eefbf7ca5becae11511d8835f7 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 9 Sep 2019 22:57:29 -0400 Subject: [PATCH 32/36] additional clarification --- proposals/1219-storing-megolm-keys-serverside.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index f51ed5f941e..1103eecf528 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -91,8 +91,8 @@ On `M_WRONG_ROOM_KEYS_VERSION` error when trying to `PUT` keys: 2. notify the user that there is a new backup, and display relevant information 3. confirm with user that they want to use the backup (user may want use the backup, to stop backing up keys, or to create a new backup) -4. ensure the backup key is signed by the user's master key, or prompt the user - to enter the recovery key +4. ensure the public part of the recovery key is signed by the user's master + key, or prompt the user to enter the private part of the recovery key On receipt of undecryptable message: From 54e73e47291712ed250d8ed3482e789ac0c8b9e5 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Fri, 4 Oct 2019 10:41:28 -0400 Subject: [PATCH 33/36] Apply suggestions from code review Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> --- proposals/1219-storing-megolm-keys-serverside.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 1103eecf528..23f1e0f5a3d 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -113,7 +113,7 @@ the key is saved directly by the user, then the code is constructed as follows: 1. The 256-bit curve25519 private key is prepended by the bytes `0x8B` and `0x01` -2. All the bytes in the string are above are XORed together to form a parity +2. All the bytes in the string above, including the two header bytes, are XORed together to form a parity byte. This parity byte is appended to the byte string. 3. The byte string is encoded using base58, using the same mapping as is used for Bitcoin addresses. @@ -129,12 +129,14 @@ results in 0, and ensure that the total length of the decoded string is 35 bytes. Clients must then remove the first two bytes and the last byte, and use the resulting string as the private key to decrypt backups. +#### Enconding the recovery key for server-side storage via MSC1946 + If MSC1946 is used to store the key on the server, it must be stored using the `account_data` type `m.megolm_backup.v1`. As a special case, if the recovery key is the same as the curve25519 key used for storing the key, then the contents of the `m.megolm_backup.v1` -`account_data` for that key will be the an object with a `passthrough` property +`account_data` for that key will be an object with a `passthrough` property whose value is `true`. For example, if `m.megolm_backup.v1` is set to: ```json @@ -245,7 +247,7 @@ On success, returns the empty JSON object. Error codes: -- `M_NOT_FOUND`: No backup version found. +- `M_NOT_FOUND`: This backup version was not found. #### Storing keys From 576177b579699ec7e07e114d338e60db56c2f1a7 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Wed, 9 Oct 2019 17:52:53 -0400 Subject: [PATCH 34/36] make version optional in versions update --- proposals/1219-storing-megolm-keys-serverside.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 23f1e0f5a3d..13f16bc99b2 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -223,7 +223,7 @@ Body parameters: - `auth_data` (object): Required. algorithm-dependent data. For `m.megolm_backup.v1.curve25519-aes-sha2`, see below for the [definition of this property](#auth_data-backup-versions). -- `version` (string): Required. The backup version. Must be the same as the query parameter or must be the current version. +- `version` (string): Optional. The backup version. If present, must be the same as the path parameter. Example: From 5799c433fe992154f1e37921c41debd6f4313319 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Thu, 10 Oct 2019 15:03:10 -0400 Subject: [PATCH 35/36] add HTTP status codes for errors and move key format to the right spot --- .../1219-storing-megolm-keys-serverside.md | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index 13f16bc99b2..ff74eb16984 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -210,7 +210,7 @@ On success, returns a JSON object with keys: Error codes: -- `M_NOT_FOUND`: No backup version has been created. +- `M_NOT_FOUND`: No backup version has been created. (with HTTP status code 404) ##### `PUT /room_keys/version/{version}` @@ -247,7 +247,7 @@ On success, returns the empty JSON object. Error codes: -- `M_NOT_FOUND`: This backup version was not found. +- `M_NOT_FOUND`: This backup version was not found. (with HTTP status code 404) #### Storing keys @@ -289,7 +289,8 @@ On success, returns a JSON object with keys: Error codes: - `M_WRONG_ROOM_KEYS_VERSION`: the version specified does not match the current - backup version + backup version (with HTTP status code 403). The current backup version will + be included in the `current_version` field of the HTTP result. Example: @@ -426,7 +427,7 @@ On success, returns a JSON object in the same form as the request body of `PUT Error codes: - M_NOT_FOUND: The session is not present in the backup, or the requested - backup version does not exist. + backup version does not exist. (with HTTP status code 404) ##### `GET /room_keys/keys/${roomId}?version=$v` @@ -446,7 +447,7 @@ a successful response with body: Error codes: -- `M_NOT_FOUND`: The requested backup version does not exist. +- `M_NOT_FOUND`: The requested backup version does not exist. (with HTTP status code 404) ##### `GET /room_keys/keys?version=$v` @@ -466,7 +467,7 @@ a successful response with body: Error codes: -- `M_NOT_FOUND`: The requested backup version does not exist. +- `M_NOT_FOUND`: The requested backup version does not exist. (with HTTP status code 404) #### Deleting keys @@ -503,14 +504,14 @@ The `session_data` field in the backups is constructed as follows: 1. Encode the session key to be backed up as a JSON object with the properties: - `algorithm` (string): `m.megolm.v1.aes-sha2` - - `sender_key` (string): base64-encoded device curve25519 key in - [session-sharing - format](https://gitlab.matrix.org/matrix-org/olm/blob/master/docs/megolm.md#session-sharing-format) + - `sender_key` (string): base64-encoded device curve25519 key - `sender_claimed_keys` (object): object containing the identity keys for the sending device - `forwarding_curve25519_key_chain` (array): zero or more curve25519 keys for devices who forwarded the session key - - `session_key` (string): base64-encoded (unpadded) session key + - `session_key` (string): base64-encoded (unpadded) session key in + [session-sharing + format](https://gitlab.matrix.org/matrix-org/olm/blob/master/docs/megolm.md#session-sharing-format) 2. Generate an ephemeral curve25519 key, and perform an ECDH with the ephemeral key and the backup's public key to generate a shared secret. The public half of the ephemeral key, encoded using base64, becomes the `ephemeral` From a6977f19c53e6ccd366b972d95e7ce4bef5eb5f8 Mon Sep 17 00:00:00 2001 From: Hubert Chathi Date: Mon, 28 Oct 2019 12:58:27 -0400 Subject: [PATCH 36/36] Update proposals/1219-storing-megolm-keys-serverside.md Co-Authored-By: aditsachde <23707194+aditsachde@users.noreply.github.com> --- proposals/1219-storing-megolm-keys-serverside.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/1219-storing-megolm-keys-serverside.md b/proposals/1219-storing-megolm-keys-serverside.md index ff74eb16984..95430fe21f8 100644 --- a/proposals/1219-storing-megolm-keys-serverside.md +++ b/proposals/1219-storing-megolm-keys-serverside.md @@ -129,7 +129,7 @@ results in 0, and ensure that the total length of the decoded string is 35 bytes. Clients must then remove the first two bytes and the last byte, and use the resulting string as the private key to decrypt backups. -#### Enconding the recovery key for server-side storage via MSC1946 +#### Encoding the recovery key for server-side storage via MSC1946 If MSC1946 is used to store the key on the server, it must be stored using the `account_data` type `m.megolm_backup.v1`.