Skip to content

Commit

Permalink
Link to MSC3051 and separate alternative version to 3382
Browse files Browse the repository at this point in the history 
Option two is moved to matrix-org#3382
Also added link to matrix-org#3051 with array implementation of relations.
  • Loading branch information
@MurzNN
MurzNN authored Sep 7, 2021
1 parent ae52584 commit 7d4542f
Showing 1 changed file with 6 additions and 124 deletions.
130 changes: 6 additions & 124 deletions proposals/2881-message-attachments.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,15 @@ On the display side, when the user sends multiple images, the problem is that ea

Messages with multiple attachments now already implemented in many messengers, for example - in Skype, Slack, VK Messenger. And Matrix, because lack of support, now have problems with bridging those messages to Matrix rooms.

This is a "best fallback" version of multiple attachments implementation, more optimal implementation is moved to separate [MSC3382: Inline message Attachments](https://github.com/matrix-org/matrix-doc/pull/3382).

## Proposal

### Option one "Best fallback"

For solve described problem, I propose to add `m.attachment` relation type to current events, that will point to other media events in room, which must be shown as attachment to current event, and `is_attachment: true` marker field to all media, that was send to be an attachment for some message.

With having this feature, Matrix client should allow users to attach one or multiple media (images, video, files) to message on client side, without instant sending of them to room, and send them together with text message.

When user press "Send" button, Matrix client do the upload of all media, that user attached to message, as separate events to room (how it is done now), before sending message with typed text. And after sending of all attachments is finished, client send message with aggregating event, using `m.relates_to` field (from the [MSC2674: Event relationships](https://github.com/matrix-org/matrix-doc/pull/2674)), that points to all previously sent events with media, to group them into one gallery.
When user press "Send" button, Matrix client do the upload of all media, that user attached to message, as separate events to room (how it is done now), before sending message with typed text. And after sending of all attachments is finished, client send message with aggregating event, using `m.relates_to` field (from the [MSC2674: Event relationships](https://github.com/matrix-org/matrix-doc/pull/2674)), that points to all previously sent events with media, to group them into one gallery. Currently the `m.relates_to` is object that is bad for extensability, but here is [MSC3051: A scalable relation format](https://github.com/matrix-org/matrix-doc/pull/3051) that fixes this.

For exclude showing those events in modern clients before grouping event added, I propose extend separate media events via adding "marker" field `is_attachment: true`, if clients got this value - they must exclude showing this media in timeline, and shows them only in gallery with grouping event.

Expand Down Expand Up @@ -79,122 +78,7 @@ For edits of "message with attachments" we can reuse same "m.relates_to" array v

For delete (redact action) message with attachments, we must also apply `redact` action to each message attachment event too.

#### Fallback:

I see no serious problems with fallback display of attachments. For Matrix clients, that don't yet support this feature, the attachments will be represented as separate media events, like the user upload each attachment separately, before sending main message.

### Option two "Best implementation"

As lite alternative to option one, we can send only one event with direct links to all attached media, instead of sending separate event for each attachment, because it will give less "spam" for room. Eg when user is sending message with 20 attachments - it will send only one event to room, instead of 21 like in option one implementation. But the main problem with this option is *fallback*.

This can be done together with [MSC1767: Extensible events in Matrix](https://github.com/matrix-org/matrix-doc/pull/1767) with adding new type `m.attachments`, which will contain the group of attached elements.

Each element of `m.attachments` array has a structure like a message with media item (`m.image`, `m.video`, etc), here is example of the message with this field:

```json
{
"type": "m.room.message",
"content": {
"msgtype": "m.text",
"body": "Here is my photos and videos from yesterday event",
"m.attachments": [
{
"msgtype": "m.image",
"url": "mxc://example.com/KUAQOesGECkQTgdtedkftISg",
"body": "Image 1.jpg",
"info": {
"mimetype": "image/jpg",
"size": 1153501,
"w": 963,
"h": 734,
}
},
{
"msgtype": "m.video",
"url": "mxc://example.com/KUAQOe1GECk2TgdtedkftISg",
"body": "Video 2.mp4",
"info": {
"mimetype": "video/mp4",
"size": 6615304,
"w": 1280,
"h": 720,
}
}
]
}
}
```

#### Fallback

For fallback display of attachments in old Matrix clients, we can attach them directly to `formatted_body` of message, here is HTML representation:
```html
<p>Here is my photos and videos from yesterday event</p>
<div class="mx-attachments">
<p>Attachments:</p>
<ul>
<li><a href="https://example.com/_matrix/media/r0/download/example.com/KUAQOesGECkQTgdtedkftISg">Image 1.jpg</a></li>
<li><a href="https://example.com/_matrix/media/r0/download/example.com/0f4f88120bfc9183d122ca8f9f11faacfc93cd18">Video 2.mp4</a></li>
</ul>
</div>
```
and JSON of `content` field:
```json
"content": {
"msgtype": "m.text",
"body": "Here is my photos and videos from yesterday event\nAttachments:\nhttps://example.com/_matrix/media/r0/download/example.com//KUAQOesGECkQTgdtedkftISg\nhttps://example.com/_matrix/media/r0/download/example.com/0f4f88120bfc9183d122ca8f9f11faacfc93cd18",
"format": "org.matrix.custom.html",
"formatted_body": "<p>Here is my photos and videos from yesterday event</p>\n<div class=\"mx-attachments\"><p>Attachments:</p>\n<ul>\n<li><a href=\"https://example.com/_matrix/media/r0/download/example.com//KUAQOesGECkQTgdtedkftISg\">Image 1.jpg</a></li>\n<li><a href=\"https://example.com/_matrix/media/r0/download/example.com/0f4f88120bfc9183d122ca8f9f11faacfc93cd18\">Video 2.mp4</a></li>\n</ul></div>"
}
```
And modern clients, that have support of the attachments feature, will cut the `div.mx-attachment` tag and replace it to rich gallery block with thumbnails of attachments.

If [MSC2398: proposal to allow mxc:// in the "a" tag within messages](https://github.com/matrix-org/matrix-doc/pull/2398) will be merged before this, we can replace `http` urls to direct `mxc://` urls, for support servers, that don't allow downloads without authentication and have other restrictions.

**If we will come up with better fallback display, maybe bring this option as main suggestion?**


## Client support

### Compose recommendations:

In the message composer, on "upload file" or "paste file from clipboard" event, the Matrix client must not instantly upload the file to the server, but the client must show its thumbnail in the special area, with the ability to remove it and to add more media. *Alternatively, it can start uploading instantly to improve the speed of the following message sending process, but there is no way to delete media in Matrix API yet ([MSC2278: Deleting attachments for expired and redacted messages](https://github.com/matrix-org/matrix-doc/blob/matthew/msc2278/proposals/2278-deleting-content.md), so server will store each file, even if it is not attached to the message.*

On "message send" action, Matrix client must upload each attached media to server, get `mxc` of it, post an event to room, and attach its `event_id` to current message contents in `m.relates_to` array (option one); or collect all `mxc` urls in `m.attachments` array on option two.

If the user uploads only one media and leaves the message text empty, media can be sent as regular `m.image` or similar message, like in current implementation.

Editing interface can be represented exactly like the composer interface, where user have the textarea for input message text, and area with all current attachments as tiny thumbnails, in which he can rearrange attachments, remove one of current attachments (that will remove its line from array of `m.relates_to` and do the `redact` action on corresponding event with media in option one, or remove item from `m.attachments` in option two; and delete media file using [MSC2278](https://github.com/matrix-org/matrix-doc/blob/matthew/msc2278/proposals/2278-deleting-content.md)), add new attachment (that will upload it as new event with refer to it in edited message `m.relates_to` array in option one / added to `m.attachments` in option two).


### Display recommendations:

On the client side, attachments can be displayed as grid of clickable thumbnails, like the current `m.image` events, but with a smaller size, having fixed height, like a regular image gallery. On click, Matrix client must display media in full size, and, if possible, as a gallery with "next-previous" buttons. Also clients can implement collapse/expand action on gallery grid.

If the message contains only one attachment, it can be displayed as full-width thumbnail in timeline, like current `m.image` and `m.video` messages.

Example of composer interface implementation with multiple attachments we can lookup in [Slack](https://slack.com/), [VK Messenger](https://vk.com/messenger), [Skype](https://skype.com).

For prevent showing of attachments as regular media in timeline before main aggregating event will be added to room, clients should visually hide media events, that have `"is_attachment": true` value, to display them later in gallery, but can already start downloading of attachments thumbnails, for speed-up display of them in gallery.

Together with [MSC2675: Serverside aggregations of message relationships](https://github.com/matrix-org/matrix-doc/pull/2675) all attachments will can be even aggregated on server side.

## Server support

This MSC does not need any changes on server side.


## Potential issues

1. On bad connection to server Matrix client can send attachments as events with `"is_attachment": true` but not send final `m.message` event, this will lead to posting invisible media to room. This can be solved on client side via caching unsent group of events, and repeat sending when connection will be recovered.

2. In option one - individual media event, to which `m.message` refers, can be deleted (redacted) after. As result, `m.message` will contain relation to redacted event. In this situation Matrix clients can exclude this item from display.

3. In option one - there are no restrictions, that message with attachments can refer only to other events, that have `"is_attachment": true`, because this is not too easy to control, and in theory user can post message, that can refer to other media, owned by other users, and `redact` event will try to delete them. But the API should restrict regular user to redact events of other users (if he isn't moderator), so those `redact` actions should already be successfully ignored by server.

4. If client attach too much media to one message, he can got rate limiting problem on server side. This can be solved via splitting and delaying send of attachments, to match server rate limits.

## Alternatives
## Alternatives

1. An alternative can be embedding images (and other media types) directrly into message body via html tags to "body" field (`<a href>` or `<img src>`), but this will make composing, extracting and stylizing of the attachments harder.

Expand Down Expand Up @@ -226,11 +110,9 @@ We still can implement both things together (my *MSC2881* as one main text with
```
But this way will give harder way to render of main message event, because Matrix clients must do the search of all attached events manually in timeline, and server will be unable to aggregate them to main message.

#### Fallback:

## Future considerations

In future, we may extend the `m.attachments` field with new types to allow attaching external URL as cards with URL preview, oEmbed entities, and other events (for example, to forward the list of several events to other room with the user comment).

I see no serious problems with fallback display of attachments. For Matrix clients, that don't yet support this feature, the attachments will be represented as separate media events, like the user upload each attachment separately, before sending main message.

## Unstable prefix
Clients should use `org.matrix.msc2881.m.attachments`, `org.matrix.msc2881.m.attachment` and `org-matrix-msc2881-mx-attachments` strings instead of proposed, while this MSC has not been included in a spec release.
Clients should use `org.matrix.msc2881.m.attachment` string instead of proposed, while this MSC has not been included in a spec release.

0 comments on commit 7d4542f

Please sign in to comment.