Joining rooms over federation can be very slow (SYN-293)

[matrixbot]

Edit: Currently tracking related work in the milestones:

• https://github.com/matrix-org/synapse/milestone/6
• https://github.com/matrix-org/synapse/milestone/8
• https://github.com/matrix-org/synapse/milestone/10

For an overview of the technical changes, see MSC3902.

---

Original Post:

I have Synapse/0.7.1-r2 installed from pip/github.
I am running nginx on a public server in front of it:

• https://matrix.ytnoc.net:443/ -> proxied to http 8009
• https://matrix.ytnoc.net:8448/ -> proxied to https 8448
  I have copied my key and crt in use on the nginx server down to my homeserver install and restarted (had to do this before federation worked).

I can initiate 1:1 chats or join rooms on matrix.org from matrix.ytnoc.net. If I join an active room like #matrix-dev:matrix.org, it takes well over a minute to synchronize. I only had one instance where I saw all the user avatars and the channel name show up. The rest of the time, the channel name on the left is replace my my local id (@​kb3dfz:matrix.ytnoc.net). See screenshot.

#test:matrix.org seems to be working just fine with federation.

I have attached homeserver.log

(Imported from https://matrix.org/jira/browse/SYN-293)

(Reported by John Hogemiller)

Attachments:

https://matrix.org/jira/secure/attachment/10115/icognito-1.png
https://matrix.org/jira/secure/attachment/10118/noalias.png
https://matrix.org/jira/secure/attachment/10116/redacted_homeserver.log
https://matrix.org/jira/secure/attachment/10113/Screen+Shot+2015-02-28+at+11.24.53+AM.png
https://matrix.org/jira/secure/attachment/10117/usernames+only.png

[matrixbot]

Jira watchers: @erikjohnston

[matrixbot]

Once I open an incognito browser (for different cache) I did get more, mainly avatars and images. The channel ID showed up, not the alias.

So I feel there's still some disconnect, but not a big one.

-- John Hogemiller

[matrixbot]

After 24 hours, the channel alias still hasn't shown up, just the ID (as seen in the screenshot). I can click on the cog and see the alias. I just submitting it as is, changing it to be an alias on my server, and listing both: ["#matrix-dev:matrix.org", "#matrix-dev:matrix.ytnoc.net"]. In all cases, it reverts back to the list propogated from matrix.org.

-- John Hogemiller

[matrixbot]

Can a project admin edit the description? I think some of my initial issues (no avatars) were browser cache related, and the key issue remaining is that the channel alias won't show up.

-- John Hogemiller

[matrixbot]

Did you have multiple clients open? Were the ones that did not get the complete state the ones that you didn't use to join the room?

-- @erikjohnston

[matrixbot]

I initially had one client open. It did not get the room member's list/complete state (first screenshot). It showed the room as my username. Later, upon opening an incognito client, I got the member list, and it showed the room id and member list. Refreshing my main client got the room id and member list to show. m.room.alias can be viewed by clicking on the gear, but does not appear on the left channel list.

This finicky behavior seems to be based on room size.
Joining #test:matrix.org worked just fine.
Joining #matrix:matrix.org (today) shows me usernames on the left side (screenshot "usernames only"). Hitting the gear, no m.room.alias appears. I see the same in chrome, chrome incognito, firefox.

-- John Hogemiller

[richvdh]

Having just done a join from a test server to matrix HQ, the join eventually completed with:

2018-04-18 13:55:23,582 - synapse.access.https.8447 - 93 - INFO - POST-726- None - 8447 - {@richvdh:matrixtest.sw1v.org} Processed request: 467006ms (53781ms, 4376ms) (12344977ms/16508ms/4567) 65B 200 "POST /_matrix/client/r0/join/%23matrix%3Amatrix.org? HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36"

- that's 12345 seconds (3.5 hours) waiting for db connections. given the whole request completed in "only" 467 seconds, presumably it is doing a bunch of that waiting in parallel.

I suspect we're handling each server separately and in parallel from the point of view of fetching server keys and persisting said keys to the db. We should apply more intelligence here.

[richvdh]

Also related: #3120

[richvdh]

Some things which are related here:

• sometimes we use an out-of-date server for send_joins #2418
• get_keys_from_store should do one big lookup, not hundreds of tiny ones #3818
• store_server_verify_keys shouldn't need to lock the table #3819
• server-key fetching logic is slow and queue-bound #3825
• matrix.org had the wrong value for a signing key #3807
• joining #neb:matrix.org often fails #2956
• presence processing pins the cpu for hours after joining a large room #5248
• Joining a large room timed out, and subsequent attempts ISE #5705

[enter-the-voiddddd]

I am experiencing the same lag when federating to any room that has more than 100 users. Running synapse in docker with open file limits set to 20000 and running postgres database. Using nginx as a reverse proxy following the instructions here.

[Janhouse]

Having similar experience without being able to join any larger rooms (~1000 users) over federated connection.
Should it actually be possible?

[FSG-Cat]

Yes its completely possible. A lot of the matrix devs and enthusiaists run their own homeservers and quite a few are in rooms like HQ and HQ is currently at 25k members so ye joining 1000+ user rooms over Fed is fully possible and works.

[nickian]

I'm experiencing this as well. Can't seem to join most matrix.org rooms. The Raspberry Pi one finally loaded randomly after errors in Element popping up saying "Can't join room." Is this a function of my server's resources, or something else? I'm only running on a virtual machine with 4GB memory and 2 CPUs, using SQLite.

[aaronraimist]

@nickian you definitely should be using Postgres for any chance of joining a room to take a reasonable amount of time https://matrix-org.github.io/synapse/latest/setup/installation.html#using-postgresql

[nickian]

It randomly joined finally when I logged back in a half hour later, despite the error.

[BBaoVanC]

It randomly joined finally when I logged back in a half hour later, despite the error.

Yeah it's a bit of a UI problem in Element. It shouldn't say there was an error, because the server is probably still trying to join anyways.

[deepbluev7]

If you people missed it, there is work being done on this: https://matrix.org/blog/2022/01/28/this-week-in-matrix-2022-01-28#synapse-website

[tucnak]

@richvdh First of all, thank you so much for putting in the work to battle this awful issue that plagued Matrix since forever! Could you please be kind to provide some simple up-to-date note/overview on the ongoing efforts so those of us eagerly awaiting, and possibly looking to contribute— could understand this better? I, for one, am very interested in resolving this, and I'm sure many home server maintainers, too would be happy to chip in, but the codebase and the intrinsics seem rather intimidating given there's limited granularity in the issuing for SYN-293.

[richvdh]

I've put some notes on progress up at https://hackmd.io/@richvdh/SkhcKnjKY https://hackmd.io/R20t3MTaSO6NvfPQh978QA.

[neoromantique]

+1 This is very annoying and breaks new user UX immensely.

Even if it is not possible to speed it up, at least providing some feedback that it is happening in the background would be great

[tucnak]

@richvdh 404 Not Found!

[mind-overflow]

Hey, is there any progress on this? Joining big rooms still takes an annoyingly long time on homeservers.

[deepbluev7]

Please stop spamming this issue with "+1"s or "this affects me too". It's not helpful. If it affects you, add a thumbs-up to the issue.

As about progress, there is no writeup for it, but you can follow along by reading the changelogs and looking for lines containing "faster" and "join". I.e. https://github.com/matrix-org/synapse/releases/tag/v1.60.0 mentions 2 things done as preparatory work.

You can also take a look at https://github.com/matrix-org/synapse/milestone/6 which lists some of the work done on this. But some pull requests like #12877 are also done as part of the faster joins work, but not explicitly listed.

So in general, you can find the current progress by using the search on this repository, since most PRs and issues mention "faster joins". As you can see it is actively being worked on, but also quite technical, so for most people the updates or explanations won't make sense.

Also, don't ask when it will be done. It will take longer every time you ask :3

[richvdh]

@richvdh 404 Not Found!

Sorry about that; I've updated the link to my notepad. However, the milestones (https://github.com/matrix-org/synapse/milestone/6, https://github.com/matrix-org/synapse/milestone/8) are probably a better place to follow along.

[gorbehnare]

Hello, I'm just trying to figure out if this is the problem I am seeing, or if something else is broken. I set up my first synapse server last week, and within two days I had everything working (including VoIP and video calls). By this point, I had already joined the public synapse-admins room and was able to chat, ask and answer questions there. As of this week, I keep getting "no known server" and "failed to join" errors after very long wait times when trying to join (seemingly) any public room.
In frustration, I have scrapped the VM, and installed a new VM with more RAM, a new domain, postgresSQL, etc... all from scratch, and I am still facing this. What are the test cases for this? how can I find out if the issue is with my homeserver or just a general matrix issue here?

[richvdh]

how can I find out if the issue is with my homeserver or just a general matrix issue here?

Find the /join request from your client to your server in your homeserver logs, then look at the other lines logged for that request (see https://matrix-org.github.io/synapse/latest/usage/administration/admin_faq.html#how-can-i-find-the-lines-corresponding-to-a-given-http-request-in-my-homeserver-log). They will give you a clue about why your join is failing.

[clokep]

It looks like the various bits of this have been implemented. All the milestones are empty. I'm sure there's more that could be done, but overall this seems good to file as separate, specific follow-ups.