P-150 feat: unit test for threadpool behaviour

[felixfaisal]

Resolves #2165

Some notes:

1. I had to implement our own AuthorApiMock, so that watch_top sends the trusted operation via a channel and we can use this channel to assert the order of operations.
2. I've introduced a hardcoded sleep into assertion A1 under the test feature to create artificial lag during testing so that I can demonstrate the expected behaviour of threadpool
3. This can be made even more modular to perform more tests but it could be a different issue

[Kailai-Wang]

Can you try to link the Linear issue too :)

[linear]

P-150 Create artificial lag for testing threadpool behavior

Context

This was mentioned in #2159
We need to introduce artificial lag onto the system and see if behaves correctly based on threadpools, One way would be having a ThrottledRequestType where the handling logic simply sleeps for X seconds before it returns.

[Kailai-Wang]

Thanks!

The logic looks fine, I've got two questions:

• can we simply implement watch_top in AuthorApiMock? We are free to change it :) - to avoid repeating the code
• can we have a test case where we have more than n_workers tasks (can be delayed A1) and see them returning in batch? Like the n_workers + 1th task should be processed only after we have vacant thread

[kziemianek]

Looks good in general. I left two comments. I think the one regarding feature toggle needs to be addresed at least.

[Kailai-Wang]

We need to resolve the conflict and then we are good to go I guess?

[BillyWooo]

Let's solved the conflicts and merge.