Cache and gradually fill Explore rooms

[LecrisUT]

Is your suggestion related to a problem? Please describe.

An example is in the Matrix community space or any of its subspaces. When you hit explore rooms it take ages for it to load, and when you switch back and forth these have to be fetched again.

Describe the solution you'd like.

• When the initial list of rooms/subspaces is queried and received from matrix, first populate a list of dummy rooms
• When checking if the rooms are valid and getting metadata check against a local cache and possibly schedule to renew it in the background
• Gradually update the list of dummy rooms with the proper room details

[dkasak]

This is still pretty bad on a recent-ish Synapse (1.51.0) and Element Desktop 1.10.6. Searching the Element Corp space for elementx on my homeserver takes 23 seconds to show the first result and about 60 seconds for the search to complete.

I noticed that each /hierarchy response returns 20 results, which seems low. There isn't really much data in those 20 items, so perhaps bumping this to something more reasonable like 50 or even 100 would speed stuff up by reducing the number of requests?

In any case, I think the priority of this should be bumped; a first time user waiting for over a minute for some search results is not a very good user experience if we want spaces to be a serious organizational tool.

[t3chguy]

The reason the pages requested are relatively small are for the UI to be snappier when scrolling, the mechanism for scrolling vs loading for filtering is identical so differentiating the intent is non-trivial. Ideally there'd be an MSC for the filtering to be done by the server, like for the /publicRooms API.

would speed stuff up by reducing the number of requests?

I doubt the requests are the overhead here, its that the server computing this requires a lot of work and thus is slow, with the way the pagination works the server would be caching things so that the request overhead would be negligible.

[dkasak]

The reason the pages requested are relatively small are for the UI to be snappier when scrolling

Would increasing the number of results from 20 to 100 really make a significant difference in response times? Running a few manual measurements from the command line with curl doesn't show much difference.

I doubt the requests are the overhead here, its that the server computing this requires a lot of work and thus is slow, with the way the pagination works the server would be caching things so that the request overhead would be negligible.

Empirically, from my testing, this looks to be wrong. Each request takes about 3-5 seconds to complete even when completed in quick succession and response times do not vary significantly with the number of results requested.

Note that this is on Synapse 1.51.0. Reportedly this was improved a bit in 1.54.0, which I've yet to test, so the point may be moot.

Ideally there'd be an MSC for the filtering to be done by the server, like for the /publicRooms API.

I'm reluctant to rely on the server too much since it will make it harder or prevent us from ever encrypting more information, such as room names. If it can be done, it's far preferable to fine-tune requests and implement caching so that this can be done efficiently in a completely client-side manner.

[t3chguy]

I'm reluctant to rely on the server too much since it will make it harder or prevent us from ever encrypting more information, such as room names

This is already an issue with /publicRooms.

[dkasak]

Indeed, but:

1. I argue that we should not double down on this by piling on additional problems and instead aim to fix existing ones.
2. /publicRooms is strictly related to public rooms where encryption is inherently less useful for confidentiality.

[lampholder]

Just an extra data point - it took me a few seconds shy of three minutes to find a room in the Element space over federation.

I:

• clicked on the Element space
• selected "Explore Rooms"
• typed the name of the room I wanted to find into the search box straight away
• waited three minutes
• the link eventually turned up

I don't usually use the feature because it's so slow - on this occasion I needed to find the room through searching, so I waited the wait.