Timeout joining big remote rooms

[matrixbot]

This issue has been migrated from #16683.

---

Sorry for filing a support request but I can't join #synapse because of this problem.

Tried looking at other issues. Some are old with fixes (which obviously does not apply here since the fixes are included) and others don't really have an answer or are closed without resolve.

When joining (preview works) for example #synapse:matrix.org after a while this appears in the logs,

2023-11-24 12:08:48,322 - synapse.replication.http.membership - 89 - INFO - POST-19608 - remote_join: @zib:xmr.se into room: !ehXvUhWNASUkSLvAGP:matrix.org
2023-11-24 12:08:48,767 - synapse.http.matrixfederationclient - 348 - INFO - POST-19608 - {GET-O-1935} [matrix.org] Completed request: 200 OK in 0.45 secs, got 514 bytes - GET matrix-federation://matrix.org/_matrix/federation/v1/make_join/%21ehXvUhWNASUkSLvAGP%3Amatrix.org/%40zib%3Axmr.se?ver=1&ver=2&ver=3&ver=4&ver=5&ver=6&ver=7&ver=8&ver=9&ver=10&ver=11
2023-11-24 12:09:26,428 - synapse.http.matrixfederationclient - 348 - INFO - POST-19608 - {PUT-O-1937} [matrix.org] Completed request: 200 OK in 37.66 secs, got 49098167 bytes - PUT matrix-federation://matrix.org/_matrix/federation/v2/send_join/%21ehXvUhWNASUkSLvAGP%3Amatrix.org/%24RTRSjwHh4xBZVO5gche9Ri3sj_yLKESeifa1C4CYrEs?omit_members=false
2023-11-24 12:13:52,344 - synapse.http.client - 932 - INFO - POST-9058 - Error sending request to  POST synapse-replication://master/_synapse/replication/remote_join/%21ehXvUhWNASUkSLvAGP%3Amatrix.org/%40zib%3Axmr.se/XGkCQlkQMB: RequestTimedOutError 504: Timeout waiting for response from remote server
2023-11-24 12:14:07,398 - synapse.http.server - 124 - INFO - POST-21356 - <SynapseRequest at 0x7fce7d068210 method='POST' uri='/_synapse/replication/remote_join/%21ehXvUhWNASUkSLvAGP%3Amatrix.org/%40zib%3Axmr.se/XGkCQlkQMB' clientproto='HTTP/1.1' site='9093'> SynapseError: 500 - Internal server error
2023-11-24 12:14:07,398 - synapse.http.server - 776 - WARNING - POST-21356 - Not sending response to request <SynapseRequest at 0x7fce7d068210 method='POST' uri='/_synapse/replication/remote_join/%21ehXvUhWNASUkSLvAGP%3Amatrix.org/%40zib%3Axmr.se/XGkCQlkQMB' clientproto='HTTP/1.1' site='9093'>, already disconnected.
2023-11-24 12:14:07,411 - synapse.http.client - 923 - INFO - POST-9058 - Received response to POST synapse-replication://master/_synapse/replication/remote_join/%21ehXvUhWNASUkSLvAGP%3Amatrix.org/%40zib%3Axmr.se/XGkCQlkQMB: 500
2023-11-24 12:14:07,423 - synapse.access.http.8086 - 465 - INFO - POST-9058 - ... - 8086 - {@zib:xmr.se} Processed request: 319.213sec/0.000sec (0.007sec, 0.000sec) (0.003sec/0.050sec/10) 55B 500 "POST /_matrix/client/v3/join/%23synapse%3Amatrix.org?server_name=matrix.org HTTP/1.0" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.11.50 Chrome/118.0.5993.89 Electron/27.0.2 Safari/537.36" [0 dbevts]

Client error : MatrixError: [500] Internal server error (https://xmr.se/_matrix/client/v3/join/%23synapse%3Amatrix.org?server_name=matrix.org)

Increasing client_timeout under federation does not seem to help.

Thank you

[shyrwall]

Seems it's fixed by deleting the room with forge-purge and then rejoining. So I guess an incomplete join causes some problems in the db that later gives an internal server error.