perf(worker.ts): lowering the threshold at which blocking redis operation is used to fetch next job

[urossmolnik]

With constant delayed jobs in highly used queue (mostly processing at least one job all the time), it can happen calculated blockTimeout is mostly smaller then current hardcoded threshold of 50ms.

If this happens, worker run loop ends up looping with no awaits, causing high CPU usage on Redis and runtime itself.

Lowering default to 10ms solved the problem for our use case. This PR also exposes this setting as advanced setting if further tweaks are needed.

[manast]

Thanks for the PR. Before merging I will make some analysis to why this improves your case. Blocking for 10ms seems like it should not have a lot of effect considering the overhead of issuing a blocking command for such a little amount of time, but as mentioned, I will try to fully understand why it helps in your case and see if there is a better solution.

[urossmolnik]

@manast thanks. Just do add, this did significantly improve our use case (about half less calls to Redis).
We do still have quite high number of calls to Redis and it does seem like only every 10th fetch call actually returns the job. So I am sure a better solution exists 🤔

[manast]

@urossmolnik did you try with a blocking value less than 10ms? or even blocking as long as it is a positive number?

[urossmolnik]

@urossmolnik did you try with a blocking value less than 10ms? or even blocking as long as it is a positive number?

@manast we did try with 5ms. But that did not make any difference. We can try today going down to 1ms and see.

Could the cause just be that every worker gets same timeout. So just one worker will get the job and the rest will just end up fetching empty job?

[manast]

If you have several instances of workers, then it is unavoidable that if you do not have the "pipe" of jobs waiting to be processed full, that some workers will try to fetch and get empty. But for 1 worker, independently of the concurrency it should not fetch empty more than once every drain delay or something like that.

[urossmolnik]

Going down and executing blocking query as soon as blockTimeout > 0, did get us another 15% less commands executed on Redis.
We did also followed up today and round delay's to seconds. Wondering if this can help with blockUntil calculations? 🤔