Filter out flood of member & hidden event spam when we detect the scrollback is full of it

[MadLittleMods]

Your use case

Why would you like to do it?

Rooms can be overwhelmed by bulk spam users joining rooms (thousands and thousands). Each one of those joins and leaves creates an event in the timeline.

Currently in rooms like this and trying to scrollback, you just get stuck on the thousands of member events that we only paginate 20 at a time. Each request is so slow and it doesn't even get me further back in actual results I want to see.

The goal of this change is to make the room scrollback usable again and be able to view the history of the room. Otherwise, when these spam incidents occur, that whole time period in the room is essentially a black hole.

What would you like to do? / How would you like to achieve it?

When we detect that the whole /messages response is filled with m.room.member join, leave, and invite events, we can ask the user whether they want to continue scrolling back without them. If they accept, we should add a filter to /messages to not include them.

Here is a mockup of what the user prompt could look like: "It looks like you're paginating through a lot of member events, would you like to scrollback without them?"

Have you considered any alternatives?

It's possible to hide all join/leave messages in the timeline with Settings -> Preferences -> Timeline section -> toggle the Show join/leave messages (invites/removes/bans unaffected) (showJoinLeaves) setting. But this just affects the display of the event. It doesn't help with filtering them out of the /messages pagination requests to being with to speed things up and get to the results we care about.

Additional context

• Hide member events
• Filter out member events when paginating /messages
• Scrollback should filter member events when there is too many
• Scrollback is slow and filled with member events
• Flood of member state spam
• Filter out bulk spam member events when we detect the scrollback is full of them

[t3chguy]

When we detect that the whole /messages response is filled with m.room.member join, leave, and invite events, we can ask the user whether they want to continue scrolling back without them. If they accept, we should add a filter to /messages to not include them.

This will cause the profiles we show on messages to be incorrect, so is not possible until (if) we remove historical profiles.

[MadLittleMods]

This will cause the profiles we show on messages to be incorrect, so is not possible until (if) we remove historical profiles.

Good caveat 👍

As a user, I'd be totally fine with using the current profile state for the scrollback messages (this is even a Labs option matrix-org/matrix-react-sdk#7815). Or just not have the profile information. Certainly a lot better than not being able to access the content. It's also probable, that not everyone who joined has a profile or even chatted for us to worry about this. In the spam incident, those users didn't have profiles that mattered and the majority never chatted.

[t3chguy]

Also I'm not sure its technically valid in Matrix to change filters mid-stream

[t3chguy]

As a user, I'd be totally fine with using the current profile state for the scrollback messages. Or just not have the profile information. Certainly a lot better than not being able to access the content.

This caveat has more issues than you'd expect though, as your push rules need historical profiles to evaluate if a given message pings you or not, so whilst scrolling back you may not have messages highlighted which should be highlighted.

[MadLittleMods]

Also I'm not sure its technically valid in Matrix to change filters mid-stream

Judging by Synapse, it works just fine. I didn't spot anything in the spec mentioning it either (did you see something?)

This caveat has more issues than you'd expect though, as your push rules need historical profiles to evaluate if a given message pings you or not, so whilst scrolling back you may not have messages highlighted which should be highlighted.

These all sound like minor problems compared to not being able to read the content. The content is still digestible without profiles. These differences can also be explained in a little hint under a ? icon next to the button/link or if we want to go extreme, explain it in the room status bar (like the lost connection bar) while scrolling through part of the timeline that has this restriction.

[t3chguy]

Missing notifications is major, not minor

[catscratchedme]

This seems to be the same problem in principle as this previous issue I created, in the sense that the client is spending a lot of time loading messages that the user probably doesn't want to see. (In this case, it's the member join messages; in my older issue, it's the redacted messages).

Even if the member join messages / redacted messages are hidden, the client will still be loading the hidden messages/redactions, which takes a lot of time.

#19086

[MadLittleMods]

Missing notifications is major, not minor

This has to be put in context. The first step is making the messages available to read in any capacity. While arguably readable by waiting hours for the scrollback to get to the mention, I'd much rather be able to get to the message right away where it maybe hasn't highlighted and figure it out because I'm human.

1. But highlights can still be on the table:
   • While someone's profile could have changed, in a lot of cases, it hasn't if you're catching up on recent messages. For example, my profile has never changed. Others change it more regularly with the their status. Some change it very infrequent (months, years). Missing highlight is only a problem where your profile has changed since where you scrolled back to which is a much smaller subset of people. And it's also not a problem for people who lurk and don't participate in the room.
   • While scrollback can be for new messages you're just catching up on, it's also for looking through the history where mention highlights aren't as important because you've already seen the message before.
   • In any case, the necessary information is available in the events via Add proper support for Mentions (through matching MXID on both formatted_body and body) element-web#19833

[turt2live]

Related: #486

[turt2live]

Duplicate of #486

[MadLittleMods]

Moved back to issue: https://github.com/vector-im/element-web/issues/24420