Add Sliding Sync /sync/e2ee endpoint for To-Device messages

[MadLittleMods]

Add Sliding Sync /sync/e2ee endpoint for To-Device messages.

This is being introduced as part of Sliding Sync but doesn't have any sliding window component. It's just a way to get E2EE events without having to sit through a big initial sync (/sync v2). And we can avoid encryption events being backed up by the main sync response or vice-versa.

Splitting To-Device messages out to its own endpoint also helps when clients need to have 2 or more sync streams open at a time, e.g a push notification process and a main process. This can cause the two processes to race to fetch the To-Device events, resulting in the need for complex synchronisation rules to ensure the token is correctly and atomically exchanged between processes. (based on the words from MSC3885)

Part of the Sliding Sync simplification. See this discussion below for why it may not be as useful as we thought before implementing.

Based on:

• MSC3575: Sliding Sync (aka Sync v3) matrix-org/matrix-spec-proposals#3575
• MSC3885: Sliding Sync Extension: To-Device messages matrix-org/matrix-spec-proposals#3885
• MSC3884: Sliding Sync Extension: E2EE matrix-org/matrix-spec-proposals#3884

Dev notes

Running relevant tests:

$ SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.rest.client.test_sliding_sync

# Or without running with poetry but in the poetry environment
$ poetry shell
SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.rest.client.test_sync tests.rest.client.test_sliding_sync tests.rest.client.test_sendtodevice

$ SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.rest.client.test_sliding_sync

---

Notes on sharing/inheriting Twisted trial unit tests: #17167 (comment)

---

Putting quotes around a type is called a forward reference. (mypy)

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Code style is correct
  (run the linters)

[CLAassistant]

All committers have signed the CLA.

[MadLittleMods]

Since I'm not familiar exactly with how /sync should operate or exactly what we care about, I need some advice on whether this part matters.

Does /sync/e2ee need to care about the device changes that would come from doing all of that extra work with rooms?

Perhaps device_lists (device changes) should be in their own endpoint? It seems like a lot of work but I haven't profiled or seen how fast all of things go.

[erikjohnston]

Yeah, it's important that device_lists get told about those things, which sucks a bit. I think you're right that in future we might want to reconsider bundling device_lists and e2ee, but I think for now we should keep it bundled (as that is what the Rust SDK expects).

We can always change it later once we can try it out in practice.

[MadLittleMods]

Updated to include all of the room derived information ✅

As far as I can tell, we don't have tests for the scenarios that this important information is necessary for.

---

It feels like all the benefits of this endpoint have fallen away though.

We're doing all the same work for this new endpoint as the old sync v2 so the performance is bound to be about the same. Which also means To-Device events will probably get backed up by other events above the same. I did add a filter to only work with membership events which might help.

Work	Sync v2	/sync/e2ee
_generate_sync_entry_for_account_data()	✅	❌
_generate_sync_entry_for_rooms()	✅	✅ (sets up derived data for device_lists)
_generate_sync_entry_for_presence()	✅ (not run for matrix.org anyway)	❌
_generate_sync_entry_for_device_list()	✅	✅
_generate_sync_entry_for_to_device()	✅	✅
device_one_time_keys_count	✅	✅
device_unused_fallback_key_types	✅	✅

And it doesn't seem to relieve the Sliding Sync proxy if we're going to use the same device for sync v2 and this new endpoint. This point is being tracked in this discussion -> #17167 (comment)

[erikjohnston]

Hmm, if we can split up _generate_sync_entry_for_rooms so that we can get just the membership changes then I think that would help a lot. Though equally happy to land it as is and do follow up in a separate PR

[MadLittleMods]

They're pretty gnarled and coupled at the moment. In order to decouple, instead of trying to re-use the data we pull out in _generate_sync_entry_for_rooms(), it might be easier just to do our own dedicated lookup to figure out these membership changes in our own new function.

I would definitely prefer to land this and update in a follow-up 👍

[MadLittleMods]

@erikjohnston Before trying to optimize /sync/e2ee, it would be good to see just how fast/slow it goes. Traces from matrix.org on good size account would probably give us the best indication. Not sure how hard it may be to get enough to_device/device_lists traffic for a good sample

[MadLittleMods]

Consolidated all of these device_lists tests spread between test_devices.py and test_sync.py to here

[MadLittleMods]

Added tests for device_one_time_keys_count in /sync because they didn't exist before

[erikjohnston]

Otherwise I think looks good

[MadLittleMods]

Thanks for the review @erikjohnston and tips/context @anoadragon453 @kegsay @Hywan @MatMaul 🐧

[Hywan]

🎉 thanks for your work!