allow configuring controller's concurrency

[aryan9600]

Motivation

Its often desirable to control the max no of reconciliations that the controller can run at a given moment to see more predictable behavior or to better utilize the host machine's resources.

Fixes #1248

Solution

Add concurrency to controller::Config which defines a limit on the number of concurrent reconciliations that the controller can execute at any given moment. Its default by 0, which lets the controller run with unbounded concurrency.

[codecov]

Codecov Report

Merging #1277 (9e3334b) into main (0a5fb72) will increase coverage by 0.22%.
The diff coverage is 87.96%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1277      +/-   ##
==========================================
+ Coverage   72.21%   72.43%   +0.22%     
==========================================
  Files          75       75              
  Lines        6258     6337      +79     
==========================================
+ Hits         4519     4590      +71     
- Misses       1739     1747       +8     

Files Changed	Coverage Δ
kube-runtime/src/controller/mod.rs	34.19% <67.74%> (-0.02%)	⬇️
kube-runtime/src/scheduler.rs	97.16% <86.66%> (-0.68%)	⬇️
kube-runtime/src/controller/runner.rs	94.94% <98.33%> (+1.28%)	⬆️
kube-runtime/src/controller/future_hash_map.rs	95.45% <100.00%> (+0.21%)	⬆️

... and 1 file with indirect coverage changes

[clux]

hey again! i like the general approach here. only had time to put some quick comments on the public interface (which only has a few nits) as i'm not super online this week. hoping to leave the small internal extension there to natalie.

[clux]

Finally gotten around to really look at this, and even tested this out a little bit in a busy controller. Functionally, it looks great 👍. It seems to be behaving the desired way by flattening the curve (compressing the reconciliation rate graph down).

The only points i have to make here are about the 1. complexity of the implementation and 2. if backpressure will become a problem here.

1. This is already a very complicated part of the codebase, and would like to have the helpers as contained as possible. Have put some questions/comments in various places.
2. If users set concurrency: 1 (controller runtime's default btw) in a highly parallel controller you might end up in a bad failure mode; throttling the controller, but continuing to fill the scheduler's pending set (afaikt). Maybe we need a configurable max limit on the amount of pending reconciliations and just return a new error type after this? Or maybe we let it grow for now and defer to backpressure mechanisms in the applier or issues at a later time (this is an opt-in thing after all, and we already have deduping to help us out). I am not sure, but I am curious to see what does controller-runtime does here - if they do anything.

[nightkr]

@clux:

If users set concurrency: 1 (controller runtime's default btw) in a highly parallel controller you might end up in a bad failure mode; throttling the controller, but continuing to fill the scheduler's pending set (afaikt). Maybe we need a configurable max limit on the amount of pending reconciliations and just return a new error type after this?

pending is largely the best place waiting lot for that backpressure, IMO. If we propagate the backpressure beyond the scheduler then we end up with 1) stale caches (because the reflector would also be backpressured), 2) less deduplication, 3) more memory usage (for modified objects we'd store both the old version in the reflector cache and the new version in the queue).

The memory cost of the pending queue (a few strings per object) is also trivial compared to the reflector cache itself (a full copy of each object, regardless of whether it's even queued at all).

[clux]

Thanks @nightkr, that is reassuring. I'll resolve the comments related to congestion or backpressure.

[clux]

Thank you very much. This is looking good to go from my end 👍