groups APIs are incredibly slow to query

[ara4n]

sierra:~ matthew$ time curl 'https://matrix.org/_matrix/client/r0/groups/%2Blinear-supporters%3Amatrix.org/users?' -H 'Authorization: Bearer secret' -H 'Origin: https://riot.im' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en,en-US;q=0.9,fr;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36' -H 'Accept: */*' -H 'Referer: https://riot.im/develop/' -H 'Connection: keep-alive' --compressed
{"chunk":[{"displayname":"Tamas Herman","is_privileged":false,"attestation":{"valid_until_ms":1520874516227,"signatures":{"matrix.org":{"ed25519:auto":"KX4uLoD39O4uothjPSnQmjk+eR8hwAPPK0qRVM+bQbLcbS5g4Zww+OWHzrn2rUh5DhjSiRJqy63\/7+cKSeD5CA"}},"group_id":"+linear-supporters:matrix.org","user_id":"@onetom:matrix.org"},"avatar_url":"mxc:\/\/matrix.org\/FdkukGfAFeEQIImdCOSPSgAA","is_public":true,"user_id":"@onetom:matrix.org"},{"displayname":"JonTheNiceGuy","is_privileged":false,"attestation":{"valid_until_ms":1520859432317,"signatures":{"matrix.org":{"ed25519:auto":"ZXJuP7kYFVoUssNHu98Ee91qMH7KmF\/wH\/Y5Gct4T9zL6ktttjVWp8wS879r8J7EF1Z0DsO6E2QLT2Yv6w0wDw"}},"group_id":"+linear-supporters:matrix.org","user_id":"@JonTheNiceGuy:matrix.org"},"avatar_url":"mxc:\/\/matrix.org\/YPtCDgcBZizMPMakFHzTEDnG","is_public":true,"user_id":"@JonTheNiceGuy:matrix.org"},

...


real    0m17.808s
user    0m0.013s
sys 0m0.010s

17 seconds just to get the list of users in +linear-supporters:matrix.org?!

[BACtaki]

@ara4n , I'd like to work on this issue to familiarize myself with the synapse codebase, hope that's okay?

[svmhdvn]

Is this still an issue? I just tried to repro the command and it took roughly 2 seconds:

time curl 'https://matrix.org/_matrix/client/r0/groups/%2Blinear-supporters%3Amatrix.org/users?' -H 'Authorization: Bearer <SECRET>' -H 'Origin: https://riot.im' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: en,en-US;q=0.9,fr;q=0.8' -H 'User-Ag
ent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36' -H 'Accept: */*' -H 'Referer: https://riot.im/develop/' -H 'Connection: keep-alive' --compressed

...

2.38 real
0.02 user
0.00 sys

[ara4n]

if you get lucky it takes about 2s; if you don't (e.g. if your riot is doing a lot of simultaneous groups requests around the same time, or if the server is a bit slow) it can be as much as 17s. as per the original bug. I don't think anything has been significantly improved on it, and there's a misdesign in the implementation on synapse (e.g. lack of caching). However, there is a chance this is going to be changed in the relatively near future by significantly improving the groups API.

If anyone wants to dig into the synapse code and profile it, though, it'd still be appreciated.

[clokep]

Communities/groups are being removed, see #11584.