Unable To Decrypt meta issue

[BillCarsonFr]

Unable to decrypt Epic Issue

Meta issue relating to all "Unable to Decrypt" problems, a.k.a "Waiting for this message, this may take a while".

[A decryption failure normally manifests when the recipient doesn't receive the keys for a particular encryption session; hence the acronym UISI: "Unknown inbound session ID". Nowadays the acronym UTD (unable to decrypt) is generally preferred.]

Needed information to resolve a reported issue

Reports from users are hugely helpful in identifying and prioritising the causes of Unable to Decrypt errors.

In order to properly debug an Unable To Decrypt error, we need logs from the receiver of the message (the one seeing the issue) and those from the sender. We can't debug issues without logs from both sides. Instructions on sending problem reports (aka "rageshakes") from within various clients are below.

In your problem report, please use the word "decrypt" or "UTD", and identify which event you can't read. Either include the event ID, or something like message from @dave:example.com at 10:10. For example, you might write UTD $173031096849423vSpOu:matrix.org.

On the sender side, the description is less important: just write "Bob can't decrypt me" or something.

If you'd like us to feed back to you on why you can't decrypt a message, open an issue in the Github project for your client, and mention the issue in your problem report.

Sending a rageshake from Element Web:

• Click on your avatar in top left part of the screen.
• Select "All Settings".
• Select "Help & About".
• Fill in the description as above.
• Then click on the Send logs button.

(Protip: although the dialog tells you you need to create a github issue, that is only necessary if you want us to feed back to you about what went wrong.)

Alternatively:

• From the composer in any room, type /rageshake <description>, where <description> is as above.
• A dialog box opens. Click on the Send logs button.

Sending a rageshake from Element X:

• Click on your avatar in the top left part of the screen.
• Select "Report a problem".
• Enter a description, as above.
• Ensure that "Allow logs" is enabled.
• Click "Send".

How to send rageshake from (classic) Element Android:

• Tap on the top right button on the home screen.
• Select Report Bug.

How to send rageshake from (classic) element iOS:

• Open the top left drawer.
• Select the Feedback button at the bottom.

---

Causes of Unable to Decrypt errors

🟢: We believe that this will be fixed with Element R
🕛: Affects a feature which is not yet supported in Element R

We categorize the main sources of UISI errors as follow:

Client Issues

Sender Side

• 🟢 The sender failed to send the keys. There’s no consistent retry logic on network error, so key sending is fragile. Queue and retry to-device msgs, so E2EE works in bad networks or with unreliable HS #673. Test: TestClientRetriesSendToDevice
• Memberlist out of sync in fresh DM element-web#20962
• 🟢 The keys may be sent slowly:
  • batching of m.room_key to-device messages is suboptimal element-web#24680 Test: complement-crypto#34
  • m.room_key_request responses are prioritised above regular m.room_key messages element-web#24681
• 🕛 Sharing historical room keys on invite is slow and should prioritise recent messages Keyshare-on-invite should start with most-recently-used sessions element-web#23778
• We abandon claiming OTKs from remote servers after 10s, and never retry, permenantly breaking E2EE to users on those servers. element-web#24138
• Failures when sharing outbound session keys are swallowed element-web#23792
• Fallback keys are cycled too quickly matrix-org/matrix-rust-sdk#3127
• NSE process can encrypt events which can cause UTDs due to key reuse element-ios#7751
• Lost OTK, leading to "OneTime key already exists" error and later UTDs matrix-org/matrix-rust-sdk#1415
• Element-web makes invalid assumptions about room membership changes with lazy-loading element-web#27285
• After reloading, my own failed-to-send message is UTD element-web#27334
• Olm sessions are vulnerable to wedging on OS crash or power failure, leading to UTDs matrix-org/matrix-rust-sdk#3354
• Message encrypted with different session from what is sent to recipients #2425
• Delayed membership responses in /sync cause UTDs:
  • Delayed membership responses in /sync cause UTDs matrix-org/matrix-rust-sdk#3622
  • Delayed membership responses in /sync cause UTDs  matrix-org/matrix-js-sdk#4291
• Encrypted room messages are sent even if to-device messages could not be sent element-android#8873
• Sliding sync: Clear the device list cache when the connection is reset. matrix-org/matrix-rust-sdk#3959
• Failed to encrypt message to a user in a room element-ios#7844

Receiver Side

• 🟢 incoming encrypted to-device messages can be lost when the application is restarted #762 Test: complement-crypto#37
  • If you restart (e.g. upgrade) Element while it's waiting to process a m.room_key, it'll drop it and you'll get UISIs element-web#23113
• 🟢 Rolling "Waiting for that message" bug (Last message always in UTD) element-android#5905 (No test since no specific testcase is known - closed because Rust is sufficiently different)
• Olm session seems to change in storage element-web#24450
• 🟢 To-device messages may be lost if /keys/query fails element-web#24682 Test: complement-crypto#38
• Device not receiving to-device messages element-x-android#2520
• Broken olm: Olm sessions sometimes get out of sync, resulting in undecryptable messages.
  • 🟢 Failed to decrypt Olm message with InvalidMAC(MacError) element-ios#7479 (Cannot reproduce, so no test)
  • Decryption failure caused one-time-key to be discarded element-ios#7480
  • 🟢 to-device messages can be processed out-of-order, causing dropped keys and decryption errors element-web#25723 Test: complement-crypto#35
  • Element X iOS: Olm sessions can wedge when using multiple Clients (e.g with NSE process) matrix-org/matrix-rust-sdk#3110
  • Clients may throw away one-time keys which are still published on the server, or have messages in flight #2356
  • Olm sessions getting wedged #2310
• Decrypting messages shows IndexedDB transaction failure element-web#14174
• Unable to decrypt cause: new device, and there were no active devices to receive keys #2421
• Unwedging doesn't work if wedged session is new, and first one created for device matrix-org/matrix-rust-sdk#3427
• Expected UTDs: Threaded replies to messages sent before you joined an encrypted room are incorrectly shown as UTD element-web#27577
• Historical messages decrypt on cache reloading, but not before matrix-org/matrix-rust-sdk#3768
• room key was not requested from backup element-x-android#3471
• Switching from SSS back to SS proxy causes to-device messages to be lost, causing UTDs matrix-org/matrix-rust-sdk#3993
• "Download attachment" incorrectly reports "Error decrypting attachment" for network errors element-web#28016
• Video thumbnails incorrectly report "Error decrypting video" for network errors element-web#28060
• UI is not updated when we receive message keys via the NotificationClient matrix-org/matrix-rust-sdk#4033
• UTD event was not decrypted after keys were received matrix-org/matrix-rust-sdk#4189
• Unexplained Decryption failure due to "Unable to decrypt attachment: Mismatched SHA-256 digest" element-web#28312
• replied-to UTD messages are not updated when the key is received matrix-org/matrix-rust-sdk#4196
• Expected UTDs after resetting your identity #2610

Server Issues

• To-device messages can take a long time to get sent over federation
  • To-device messages delayed by _DestinationWakeupQueue matrix-org/synapse#15161
  • Synapse does not attempt to send events in the device outbox at startup synapse#16680
  • Federation catchup doesn't send to_device EDUs until the remote end has caught up synapse#8691
• To-device messages may get lost on the server
  • to_device message didn't get delivered matrix-org/synapse#9533
  • to-device message went missing matrix-org/synapse#15335
  • Running under sqlite, Synapse incorrectly populates the to-device messages current stream ID matrix-org/synapse#16681
• bugfix: correctly tell clients when the fallback key has been used matrix-org/sliding-sync#390
• Synapse drops received federated to-device messages if it cannot talk to worker processes synapse#17117
• E2E Device lists can get out of sync with the devices actually present in a room. #2411: E2E Device lists can get out of sync with the devices actually present in a room, causing keys not to be sent. This can happen for various reasons: see the linked issue.
• Rolling a homeserver's database back via backup could cause duplicate OTKs and hence UISIs #2155
• /sync and /members do not return "current state" synapse#16940
• /sync incorrectly calculates state changes for non-gappy syncs with lazy-loading synapse#17050

Key Backups

• Successful download from key backup, but 0 keys returned matrix-org/matrix-rust-sdk#3197
• Key backups: some backups have incorrect MAC causing UTDs on login #2338
• Key Backup | Dart SDK is creating invalid payload for session data sender_clencaimed_keys famedly/matrix-dart-sdk#1739
• Desynchronized Secret Storage #2322
• Failures to deserialize/decrypt backed-up room keys are not shown in the UI matrix-org/matrix-rust-sdk#3875
• Key backup not signed by MSK, leading to UTDs on new devices #2595
• Some clients do not set up key backup automatically, leading to UTDs #2596

Protocol Issues

• A client has been offline for too long and the senders have run out of one time keys. This will be addressed by the proposal to maintain fallback keys.
• /sync API does not tell clients when the server's view of state changes outside the timeline matrix-org/matrix-spec#1209
• Room events arrive faster than to-device messages during federation connectivity problems, causing decryption failures matrix-org/matrix-spec#1123
• One-time-key upload/claim is racy matrix-org/matrix-spec#1124
• Important/urgent to-device messages (eg room keys) can get blocked behind big queues of less important ones matrix-org/matrix-spec#1659
• Users whose servers were unreachable will receive undecryptable messages due to failed OTK claim #2154
• Users who join an encrypted room at the same time as a message is sent may receive a UTD (join/send race) #2268
• Rotating the current device keys will break encryption and cause UTDs #2374

Missing features

• Users joining a private room within a space via "restricted" join rule have no way of accessing (encrypted) history #646
• If all devices are logged out there will be no end to encrypt to. This will be addressed by the proposal for dehydrated devices. Device dehydration v2 #922

UX

Expected UTD

• Some room history (pre-invite for example), should never be decryptable by you. This kind of history should probably be hidden or displayed differently.
• Expected UTDs: Hide device-relative historical UTDs #2313: New devices cannot decrypt existing history until they have access to key backup

User Config

• Message keys withheld due to "Never send encrypted messages to unverified sessions from this session" are not shown as such #2450

[BillCarsonFr]

Interesting related blog post https://blog.neko.dev/posts/unable-to-decrypt-matrix.html

[jittygitty]

@BillCarsonFr Wow what a great write-up you found at https://blog.neko.dev/posts/unable-to-decrypt-matrix.html

Please correct me if I'm wrong, but having looked at that, it seems that just about all of those errors could be resolved via wizard/options provided to the user by the client which could try to resend encryption key with help of server or to even create a new one for the channel/room. I don't see any 'technical' issue/blocker other than currently missing functionality client-side to prompt user for permission to, with the help of in-between server, do what's necessary to fix room encryption/decryption.

If there's a 'security' issue with properly re-identifying the correct user, for example to re-provide keys to the other user while they've gone offline so they can pick them up from server when they are online again, can't we have a simple "challenge/response" ie question/answer to "pick-up" the keys? (I'm trying to avoid creation of a "new" room/channel and instead fix one already created but giving errors of unable to decrypt. The sender's device has not sent us the keys for this message.)

It seems the reason certain steps aren't by default taken to automatically fix such issues, is that it could be a security risk in certain situations. But if the "creator" of a room gives needed permission, it seems in a sense trivial to resolve such decryption/key errors etc. Or am I wrong and missed something?

And to the author of:
https://blog.neko.dev/posts/unable-to-decrypt-matrix.html
Many thanks!

[BillCarsonFr]

Please correct me if I'm wrong, but having looked at that, it seems that just about all of those errors could be resolved via wizard/options provided to the user by the client which could try to resend encryption key

FTR, there used to be such UI, but it was prone to social attack and annoying in the UI.
It's something explored, but it's a bit hard to have a fix all encryption problems button. We are trying to go to the bottom of the root cause for distribution failure by moving to the rust sdk

[jittygitty]

@BillCarsonFr Ah ok guess I'm relatively new so didn't know of the older UI. Personally, I'd be ok with the "social attack" risks and UI annoyance versus the embarrassment of inviting users to my new chat and getting decryption errors.

My concern was that it may be impossible to fix the root for all cases of decryption issues, especially if some are due to security measures which may need to be over-ridden by user consent in order to fix, which brings us back to that UI annoyance.

But regardless, it's great to hear the underlying sdk is being improved to hopefully eliminate or greatly reduce these issues. I had heard of rust in conduit (I run go-dendrite) but didn't know Kotlin SDK is being all redone in rust, is that right?

Anyway, thanks again to everyone working on this! (I look forward to some beta-testing with the new sdk when its ready for that.)

[anon8675309]

The link to send debug logs does not appear on Element Web on my server. Did this UI change, or is there some option that an admin needs to enable to get that to show up?

[t3chguy]

@anon8675309 your config.json must have the URL to send debug logs to, like the example https://github.com/vector-im/element-web/blob/develop/config.sample.json#L25

[anon8675309]

Can you confirm that this is still the case? I have the bug_report_endpoint_url entry from the sample and the link to send debug logs does not appear. If it's working as expected for you, I'll set up a new server and open a new ticket with the minimal steps to reproduce. (I searched for a report of this issue and didn't find anything, but I'll do so again before opening a new issue).

[hieronymousch]

Was able to send feedback from Element desktop... and have this issue with one single user on my home server, both users on the same server. Upgraded the room to version 10 and got the error again after not even 10 messages

[Ezwen]

Hi there, I have been encountering this problem a lot recently. I've already sent logs from element-web and element-android as the person who received encrypted messages that cannot be decrypted, but I could not yet send logs as somehow sending such problematic messages.

Question: when this situation happens in a matrix room, where a given user ends up only sending messages that cannot be decrypted by other room members, is there any (even intricate) known workaround? The only "workaround" I used so far was to upgrade the room to a newer room version, which solves the issue by creating a new room, but I can't really call this a proper solution… and right now I have this problem on a room that is already using version 10…

[lousando]

@Ezwen I've found that asking sender of messages to run /discardsession usually fixes any messages moving forward. Though it does not solve the messages with the encryption issue.

[penyuan]

Updates on recent progress here:

...

Just want to say a bit thank you to @richvdh for this progress update. ❤️ There is still MUCH that needs to be fixed in terms of bugs and UI/UX when it comes to encryption/decryption weirdness, and so many people I introduce to Element still get bounced off because of these problems. But at least getting these progress updates is helpful!

[kegsay]

Recently I've been giving updates for this on This Week in Matrix. If you fail to decrypt a message please:

• send a bug report to us and mention:
  • the event ID which failed to decrypt,
  • whether it was 1 or many events which failed to decrypt,
  • if many events, are they all from different people?
• ask the sender to also send a bug report mentioning the event ID

We often need both sides of the conversation to fix the issue.

It would also be helpful for us if you can opt-in to analytics, as that feeds into our graphs which plot UTDs in aggregate. The general trend of the past few months has thankfully been fewer UTDs across clients that opt-in, but there is more work to be done here.

[yennor]

I usually got that kind of problem when me or the peer beeing in an area with bad mobile phone connection. With bad I mean really bad. You can get disconnected from the network for several minutes all the time, randomly get connected again for a few seconds, or beeing connected, but almost no data gets through.
I haven't been there (rural area of Colombia) since last year and won't for a few months. So So I can't tell if the situation improved with the new clients.
But maybe for your test-suite, simulate random tcp package drops (very high percentage) with high RTT (Several seconds, sometimes I measured up to 20 seconds. around 5-8 seconds is normal). And sent a few thousand messages there and back.

[jittygitty]

Does matrix-org/matrix-js-sdk#454 already "implement" solution for: #647 ?
And if so how would my friend request keys from me (on Desktop Element) on a new fresh install of his?

Are we able now to send decryption keys to whomever we wish?

[t3chguy]

@jittygitty yes but Rust Crypto does not support that and that is what Element uses, so the js-sdk PR is unrelated

[jittygitty]

@t3chguy That's good news, so I guess if I wanted to make use of this feature, I would have to install the "Web" version client of element chat on my webserver? (I currently run Dendrite already, but with mobile and Desktop clients only.)

Or is there another client app that uses js-sdk? (Element Desktop does not use it?) Or will rust crypto gain it soon?

I use another account on 'matrix.org' using desktop+mobile, for that acct. we can simply login at chat.element.io and if I'm logged in via my other devices, I can send keys to my web chat.element.io and resend to friend who lost his? thx

[t3chguy]

Element Desktop = Element Web + Electron. Element Web only supports Rust Crypto at this time. As for other matrix-js-sdk consumers I suggest finding a place discussing Matrix rather than Element specifically.

[jittygitty]

Apologies, indeed, seems I'm confused as to what matrix SDK is used by what "client", and I had not noticed the matrix-org/matrix-js-sdk#454 was outside of element-hq repositories. So, I guess you are saying that likely none of the clients under https://github.com/element-hq are able to leverage that matrix-js-sdk pull 454?

Is there a place I can find all such pertinent information on the various clients available? If anyone can point me in the right direction with a link would be appreciated. Otherwise, I'll try some search engine lookups to dig for such info. thx

[mpeter50]

It seems Cinny uses the matrix-js-sdk, and maybe there are more, but I havent found another client that does.
Then, its an other question if Cinny makes use of this feature of the SDK. You could ask about that in Cinny's support room.

[richvdh]

Another round-up of recent updates. (Most of these have already been reported in TWIM, but I think it's handy to have a record here too.)

• We found and fixed a cause of broken Olm sessions on Element iOS. As with the previous fix to EX iOS: you don't have to be using Element iOS yourself to notice UTDs in this scenario: it causes problems on the sender and recipient side alike.
• We also have more fixes to both Element X iOS (#2944) and Element X Android (#3050) which could cause broken Olm Sessions.
• We found and fixed a bug in matrix-rust-sdk affecting Element X.
• We added a workaround to the sliding-sync proxy which would guard against messages being lost after the database was dropped, potentially causing UTDs for Element X clients.
• We rolled out a change to Synapse and all Element clients, which will identify messages sent before you joined a room, and let you know you shouldn't expect to decrypt them.

A reminder that we're still at war with this issue, and it's incredibly helpful for people to send debug logs when they come across UTD errors.

That said, our analytics show that we are starting to make progress here:

[richvdh]

I realised we're overdue for an update of recent work done in this area.

First, some graphs of UTD rates, according to our metrics:

(blue=android, purple=ios, green=web/desktop)

We changed how the metrics were calculated back in August, to allow breakdown between EX and classic mobile clients. As you can see, there was a big drop back in June, and rates have been relatively static at 1-5% of users (depending on client) since then.

Focus recently has been on analysing reports to see where the most important areas for fixes are. #2356 is the most frequent root cause, so we're going to prioritise a fix for that.

Meanwhile we did fix a few problems:

• A bug in Element X Android which could cause broken Olm sessions.
• A bug in Element X which meant it didn't update replies when the original message was decrypted.

There are a few new causes in the list above, but I won't enumerate them here, as they are still under investigation and haven't been seen in many cases.