Shadow Ban unit tests are flaky due to random sleep times

[anoadragon453]

Description

The ShadowBanTestCase is flaky:

• https://buildkite.com/matrix-dot-org/synapse/builds/11211#9f35c864-1cd1-4e2f-8322-b3b693fea6d5/2675-2747
• https://buildkite.com/matrix-dot-org/synapse/builds/11211#91d9b873-8574-41b1-8cb4-394ec97a99ed/85-1850
• https://buildkite.com/matrix-dot-org/synapse/builds/11195#cfddfa55-f5f4-4d57-bd91-6b453f44862b/54-1805

Likely due to requests sometimes being artificially slowed down to thwart malicious users:

synapse/synapse/handlers/room_member.py

Lines 316 to 317 in e259d63

	# We randomly sleep a bit just to annoy the requester.
	await self.clock.sleep(random.randint(1, 10))

These tests seem to be hitting this timeout, which the containing function has an arbitrary timeout value of 100 (which I believe means 10.0s). That means if a request takes >1s and we had a random sleep of 9s, the request will timeout.

Solutions here I can think of:

1. Upping this limit to something slightly higher (not sure what else relies on this though)
2. Weaving the timeout value through a bunch of functions all the way up to self.render such that these tests can explicitly say they expect to take slightly longer
3. Having a decorator, @override_timeout(150) similar to @override_config. This would be nice, though I'm not sure how exactly to pull it off

[clokep]

Thanks for investigating this a bit! I'll take this since I'm still working on this code anyway. 😄

[clokep]

Solutions here I can think of:

1. Upping this limit to something slightly higher (not sure what else relies on this though)
2. Weaving the timeout value through a bunch of functions all the way up to self.render such that these tests can explicitly say they expect to take slightly longer
3. Having a decorator, @override_timeout(150) similar to @override_config. This would be nice, though I'm not sure how exactly to pull it off

Another option would be to patch random.randint to always return 0 for those tests or something. I think that's my running contender right now, unless we'd benefit from having the timeout be configurable anyway.

[anoadragon453]

@clokep Perhaps instead we should make a function that generates timeouts for ShadowBanned users and mock that during tests? Something may rely on random.randint now or in future.

Edit: We're only mocking across a single TestCase instead of all tests, so there is no concern.