Declaring End-to-end Encryption stable and turning it on by default for private rooms.

[ara4n]

Somehow we seem to be missing a high-level tracking bug for the endgame of E2E. (The starting point was matrix-org/matrix-spec-proposals#501, but that's a spec bug and feels a bit weird to hijack it for this).

In order to declare E2E stable (and by extension Matrix), we need to:

• Cross-sign devices at login to eliminate verification warnings (We could cross-sign devices to aid trust when new ones join a room #2714)
• Optionally incrementally backup encrypted E2E keys to your HS so if you lose all your devices you can restore your history via a recovery key that you own. (Improving megolm key storage (meta) #5675)
• Improve verification UX (comparing per-verification mnemonic passphrases rather than public keys) (Implement E2E key verification process #2142)
• Hook up key sharing to let history be visible from before you joined a room (but after you were invited to it), history visibility allowing. (Let megolm session keys be available to devices added by invited users since the point they are invited #2713) - mainly fixed, other than for edge case where users add devices after being invited to a E2E room
• Have less than 0.0x% unresolved unable-to-decrypt errors reported through telemetry ("Unable to decrypt: The sender's device has not sent us the keys for this message." (The UISI bug) #2996, Telemetry to report incidence of UISIs #6390)
• Fix device lists getting out of sync over federation (E2E device lists can get out of sync over federation when users delete their devices. #6974)
• Support full-text search within E2E rooms (Client-side search for E2E #2548, solved by github.com/matrix-org/seshat)
• Support push and in-app notifications in E2E rooms at least by locally calculating them. We could also support setting metadata on the messages to bing users, especially if we had pseudonymous MXIDs.
• Support NotifPanel in E2E rooms (Notif panel needs to know how to deal with encrypted room #6874)
• Turn it on by default for private rooms. All rooms created with the intent of use as a 1:1 or a private group chat must be E2E (and homeservers would enforce that, as best they can, c.f. Support only E2E or public rooms on my homeserver matrix-org/synapse#3269), although obviously a broken client could still try emitting unencrypted events into such a room). The code exists for this already over at https://github.com/matrix-org/synapse/pull/3426/files.

Ideally:

• Provide some kind of E2E daemon/proxy to stop non-E2E clients/bots/bridges/curl-users being unceremoniously dumped out in the cold. (#6778)
• Improve the UX for managing the trust of the devices which are in a given room (E2E device/key management dashboard to replace UnknownDeviceDialog #4522). In practice if verification is working nicely this might not be so important.
• Optionally hook up key sharing to let history be visible from before you were invited to a room (history visibility allowing) (#2286)
• Get an end-to-end audit of crypto spanning a set of reference implementations - e.g. matrix-{web,ios,android}-sdk + synapse.

[aaronraimist]

#6959 should probably be in here at least under ideally

[dbkr]

I think '* Optionally hook up key sharing to let history be visible from before you were invited to a room' needs to be non-optional, or if not then change the history visibility settings for e2e rooms to line up with what's actually possible, ie. remove 'anyone' and 'Members only (since the point in time of selecting this option)')

[ilmaisin]

It was originally promised that e2e encryption would be enabled by default when it was out of beta. That didn't happen.

But anyway, as I understand, there are basically three things to be done before e2ee can be default: cross-signing, local search and notifications. Are there any on-going efforts to achieve those?

[ara4n]

yes, of course. cross-signing is in the final stages; notifications just got largely fixed on riot/web and already worked on mobile; local search development is kicking off again as of tomorrow. we also want to fix element-hq/element-meta#80 before turning it on by default.

[ilmaisin]

Well, cross-signing has been "in the final stages" for half a year now. A honest, detailed explaination on what is really happening would be nice.

[ara4n]

a quick but honest detailed explanation is:

• @uhoreg has been working solidly on it for ages
• he shipped working SDK & server code about 2-3 months ago, as per https://scitech.video/videos/watch/1c8a0262-dee9-4927-b9e9-9856a7f4d8bf and https://scitech.video/videos/watch/d1ef04a8-397a-4570-a9a9-cef143ea637c
• we then made an architectural decision to switch to using 1:1s rather than toDevice messages for verification, so you get history of verifications persisted nicely in your DM with the person who is verifying you (or vice versa).
• this work has now been implemented too (as of last week or so)
• meanwhile, much of the Riot/Web team got pulled onto dealing with Privacy stuff, thanks to https://matrix.org/blog/2019/06/30/tightening-up-privacy-in-matrix
• this work is now finished and shipping next week (hopefully); meanwhile, work has commenced again on hooking up the E2E UX for cross-signing in Riot.
• The current proposed UX changes are: https://matrix.org/_matrix/media/r0/download/matrix.org/iYotsQUzFQTgsZxaiNyBEfzF (warning; 7MB JPEG)
• There is a lot of work to land this on Riot across all 3 platforms, as they share no code, so it will have to be implemented 3 times over. We'll do Web first, and probably have to iterate a bit, and then mobile will follow.

Sorry it's taken a while; turns out that this stuff is hard, and we've been juggling a lot of stuff thanks to privacy dramas etc.

[ara4n]

meanwhile, e2e search has been progressing first via https://github.com/matrix-org/pantalaimon (and works well; i use it daily, but it's not integrated tightly with Riot), and now by https://github.com/matrix-org/seshat, which I believe works with Riot/Desktop although I haven't tested it yet.

[ara4n]

turning it on by default is formally proposed as an MSC now, as part of the Canonical DMs proposal: (point 5 of https://github.com/matrix-org/matrix-doc/blob/travis/msc/immutable-dms/proposals/2199-canonical-dms.md#creating-dms)

[ara4n]

...and UISI errors (aka UTDs) are almost unheard of, in my experience. We fixed a major remaining cause of them back in July (matrix-org/synapse#5693) and have built out a whole new project for adding OpenTracing to Synapse so that whenever find further ones, we can pull up a full visualisation of precisely what went wrong to diagnose it and so snuff them out conclusively one by one if/when we see them: https://github.com/matrix-org/synapse/pulls?q=is%3Apr+opentracing+is%3Aclosed.

I think those were the only pending points.

[ilmaisin]

#11125 is supposed to implement the e2e search, but it seems to have been sitting idle for two weeks now. By the way, Conversations has been doing e2ee search for ages.

The progress still looks like being really slow to me, though I absolutely may have missed something as I am not an expert on this stuff.

[ara4n]

You’ve missed that the e2e search is 3 layers deep: riot-web, matrix-react-sdk and seshat itself (which unlike Conversations is designed to be crossplatform, written in Rust). The react-sdk layer was active less than a week ago: https://github.com/matrix-org/matrix-react-sdk/tree/poljar/seshat-rebase and the seshat layer was active a few hours ago: https://github.com/matrix-org/seshat/tree/sqlcipher. The feature works great; i’ve been using it for a few weeks. The only issue is that it stores its data currently in plaintext on disk, which is not ideal - so we are reencrypting it when it sits on disk.

[ilmaisin]

Is there any way to donate specifically to e2ee work? Looking at those "referenced" notices, it seems that I am not the only one who would like to have this as a much higher priority. I don't want my donations to be spent to the fiddling with emojis or any other such bells and whistles.

Should I open a bountysource entry for this issue or something?

[turt2live]

As of about 3 minutes ago the team has been given the directive to implement this as fast as safely possible. What this means is that in the coming months we should have a release that implements cross-signing and generally better e2e UX as a result.

We are aware that this statement has been made several times now, but hopefully this time we're not lying given we're about to have code to back up our statement.

[ddobrev]

Hello, would you have any news?

[turt2live]

The statement above regarding our new directive is still accurate, though this time we are in fact not lying. We're extremely close to being able to show off the work we've done, though there's still a large chunk of work outstanding before we're comfortable putting it out for release.

Much of it can be tried on riot.im/develop if you're willing to risk the chances of us mixing it up :)

[jryans]

Cross-signing and E2E by default for DM rooms will be enabled on the develop channel (https://riot.im/develop) in the next day or so to collect feedback from early adopters.

We're hoping to release to the stable channel a few weeks after that, but as this is a huge milestone, we want to be absolutely sure it's ready before releasing to everyone.

[ilmaisin]

Any update on this? #13212 means that it cannot be done today, I think?

[jryans]

We've identified a few more release blocking issues to resolve and will have another RC later today for more testing. Updated release target is now next Monday (2020-04-20).

[ilmaisin]

The Android and iOS clients also need to have this implemented before it can really be the default, right? How far from completion are they from?

[jryans]

RiotX Android and Riot iOS are targeting the same release date as Riot Web for this work.

[rgpublic]

Um, just a quick question while we are all anxiously waiting for this very cool new feature to drop on us: What does "turning it on my default" exactly mean? What happens to my existing 1:1 chats in Riot? Will they magically be encrypted as soon as I access them with the new version? Or will I have to leave/reenter them somehow? Should I recommend our users to check anything so that they won't suddenly lose access to their chat history? Perhaps making absolutely sure they have key backup enabled and?/or? having exported their E2E room keys in the user profile dialog?

[t3chguy]

They will not, for existing rooms the onus is on room admins to enable encryption. 1:1s are still rooms where everyone is an admin.

[rgpublic]

Understood. Thanks a lot @t3chguy for the clarification!

[Ekleog]

Maybe I'm missing something, but… as far as I could find in the UIs, cross-signing isn't supported yet in a released version of riot. Would it not make sense to first release cross-signing, wait for at least a few weeks for things and bugs to stabilize, and then turn on encryption by default, once it's confirmed that it actually works? Otherwise, I'd expect a lot of angry shouting at every bug that may be remaining in the cross-signing code but not yet apparent due to the low intersection between people using the development branch and people using e2e rooms

[jryans]

We've identified a few new performance and behaviour issues to resolve before release thanks to everyone's feedback and testing on staging. We believe there may be around 1 week of work to resolve at this point, but we'll keep evaluating every day.

[jryans]

We've published 1.6.0-rc.4 with some additional cross-signing fixes, so we're getting very close thanks to everyone's testing and feedback! Please do test and file issues for any feedback or problems you may see.

[QEDeD]

It's good to see that you are taking the time needed to get the launch right, even though it must be rather tempting to go ahead when you are this close.

[rgpublic]

Hooray. Party. I just got an update on my (Android) phone with a new RiotX version and E2EE. But: There is no matching update (PPA) update available for the Desktop (Linux). What should I do? Just wait?

[jryans]

There is no matching update (PPA) update available for the Desktop (Linux). What should I do? Just wait?

For issues and questions like this, please follow updates and ask questions in #riot-web:matrix.org.

[jryans]

Riot Web and Desktop v1.6.0 are now available with cross-signing and E2EE by default for DMs and private rooms enabled! 🥳 Thanks to everyone who helped test and provide feedback along the way. 😄

That wraps up the main focus of this issue. For any related concerns that seem unresolved, please file new issues.

[DarwinPorras]

Hi jryans. Today I updated all my devices to Riot 1.6.0. I was able to verify all my sessions in different accounts but I can not make chats to be encrypted I also do not see the padlock closed on my app. I don't know if I forgot to setup something but I have this message on any user account

SECURITY
Messages in this room are not end-to-end encrypted.

Verify
- Riot Desktop on Windows
Hide sessions

Do you know what can I do to enable E2EE on direct messages?

[turt2live]

@DarwinPorras you'll have to go into the room settings. Please visit #riot-web:matrix.org for support.

[DarwinPorras]

@turt2live Thanks!

[Bun-Bun]

While I understand the necessity for this feature for many people, it is actually a detriment to my server.

For my purposes my server should never allow encryption ever. How do I disable encryption full stop?

[t3chguy]

For my purposes my server should never allow encryption ever. How do I disable encryption full stop?

That would be a query for your server of choice, probably https://github.com/matrix-org/synapse/

Synapse can disable both cross-signing & encryption altogether.

[Bun-Bun]

Forgive my ignorance, but the last time I check (only a couple weeks ago) there was no official way of disabling encryption on synapse.

Having a config option on my self hosted riot to disable the encryption would be helpful. Though that only helps the web users. You're right it needs to be at the server side and all riot clients need to respect that by transparently disabling encryption and not throwing an error or other weirdness that makes users ask questions.

[ptman]

Indeed, matrix-org/synapse#4401 makes it sound like disabling encryption isn't yet possible.

[t3chguy]

Right.

For my purposes my server should never allow encryption ever.

But this cannot be done by a riot-web instance unless you force everyone to use that, people use things like riot android/ios too.
Ergo it needs to be done on the server.

It could be done using Nginx as a reverse proxy to intercept and fail m.room.encryption state events and also to return the right flags to claim the server doesn't support cross-signing.

[turt2live]

Please let's not use closed issues for support - instead, #synapse:matrix.org, #riot-web:matrix.org, or a new/already open issue's comment section is best.