Implement MSC2477: User-Defined EDUs

[matrixbot]

This issue has been migrated from #12413.

---

User-Defined EDUs are detailed in MSC2477. This issue details what needs to be modified in Synapse in order to implement it.

Below are the broad changes that need to happen, broken up into sensible chunks that can land individually (aka, these are design to potentially land in different versions of Synapse and not break anything):

Chunk 1

• Create an experimental config option, default disabled. See synapse/config/experimental.py
• Create a new room version. They are defined in synapse/api/room_versions.py.
• Update event auth to check the current power levels in the room and allow/disallow EDUs to be sent
  • EDUs live outside of the DAG, and are simply auth'd against the latest state of the room as the homeserver knows it.

Chunk 2

• Add the PUT /_matrix/client/v1/rooms/{roomId}/ephemeral/{eventType}/{txnId} endpoint.
  • Remember unstable prefixes!
  • Look at RoomTypingRestServlet for an example.
• Rate-limits for sending EDUs.
  • Concern of users spamming EDUs into a room
  • A level that still allows users to update their location frequently enough, or play/pause music quickly.
• Store and aggregate, assign each EDU its own stream ID using a new stream ID sequence for user-defined EDUs
  • Make a new table for user-defined EDUs
    • Requires a new migration file in the latest schema directory
  • We hold EDUs and throw them away after a period of time. We can’t determine based on EDU type as the EDU type may just be m.room.encrypted (see MSC3673 (encrypted user-defined EDUs))
  • Extend sync token definition with our new user-defined EDU sequence number.
  • Purging EDUs from the database after a configurable amount of time
    • The MSC currently leaves delivery guarantee up to implementation, aka. how long does the server hold on to an EDU while the client is offline before it is purged. For now we may want to simply clear these after a hardcoded period of time. A subsequent MSC could better determine how this should work (allowing the sender to specify how long an EDU should be held before it is purged?).
  • Store timestamp with the EDU in a table, so we can later delete based on time.
  • Have an index on timestamp (for purging based on sent time) and user ID / stream ID (for pulling out new events for a user when they /sync).
• Sync
  • Add a new entry to the sync token
  • Possibly create a new EDUSource (see e.g. PresenceEventSource) which pulls out the latest events for a user given the stream ID they’ve caught up to from their sync token
    • This is what pulls EDUs from the database for clients
  • Adding new events to the sync response
  • Ensure that clients can properly filter custom EDU types, noting that the server may see all encrypted EDUs as type m.room.encrypted.
  • Users that join the room should not receive EDUs that were sent from before they joined.
• Ensure methods are appropriately cached
• Ensure this all works in Synapse's worker mode.
  • Look at MultiWriterIdGenerator.
  • Cache invalidations should be streamed across workers.

Chunk 3

• Support sending and receiving user-defined EDUs over federation
  • Adding extra fields to the EDU schema as defined in the MSC
  • Ignoring custom EDUs if they’re not in a room version that supports them.
    • Note that Synapse currently drops EDUs it doesn't recognise the type of on the floor.
  • Adding to the ephemeral array in outgoing federation txns

Chunk 4

To really call this feature complete, we should forward user-defined EDUs to application services, much as we do for existing EDUs today. However, this first needs to be designed and added to the MSC.