Joining of machines fails when max # of (worker) reconciliation is reached.

[prashanth26]

What happened:
Newly created machines fail to join which hinders (takes longer) for rollingUpdates/Scale up. This problem and leads to hanging of machineSet/machineDeployments.

What you expected to happen:
Already created machines should be able to join/re-join even even if there are several machine operations in parallel.

How to reproduce it (as minimally and precisely as possible):
Create a machineSet with a higher desired value. Deploy pods with PDBs that prohibit draining of machines. Now scale down the machine set to a much smaller number (more than half it original desired). You will see that the machineSet takes too long to reach the desired state (over drain timeout period).

Anything else we need to know:

Environment:

[prashanth26]

/priority critical
/area operations
/status in-progress
/size xs

/assign

[prashanth26]

Restarting the MCM temporarily solves this issue. However, we need to look into this issue

[prashanth26]

I checked through the code path. Looks like joining/re-joining of nodes is possible (and isn't blocked by this). However, I think this issue is due to the maximum parallel workers allowed. Currently, MCM only allows 5 workers (per resource kind) to process in parallel.

Code for the same. In below links

• https://github.com/gardener/machine-controller-manager/blob/master/cmd/machine-controller-manager/app/options/options.go#L57
• https://github.com/gardener/machine-controller-manager/blob/master/pkg/controller/controller.go#L492-L507

My guess is due to this as restarting usually fixes this issue. And I had observed more than 5 machines stuck in draining machines hence causing this. Also checked Prometheus to find this graph to find the average processing time and worker (machine) queue length to increase causing this starvation. The peeks keep dropping to zero on restarts and keeps increasing as machines stuck on draining.

[prashanth26]

There are two ways solutions i see.

1. Increase the max allowed concurrent workers.
   a. Set this statically to a higher number. Say - 10.
   b. We could set this value based on the number of nodes in the cluster.
2. Set drain call timeouts to smaller intervals (~10mins intervals). So drainTimeout of 2hours and setting this to 10mins would mean. 12 different drain calls instead of 1 long 120mins call. Which is currently causing this block.

I prefer approach (2). However, maybe there is better way to solve this.

cc: @amshuman-kr

[amshuman-kr]

@prashanth26 Thanks a lot for analysis. My preference, actually is different from option 1 and 2.

3. Decouple drain (at least the waiting part) from reconciliation path. Ideally, we should fire whatever delete/evict calls that are possible and simply ReconcileAfter and pick from where the last reconciliation left off in the next reconciliation. The current waiting and retries in a loop inside the reconciliation is a bad idea generally.

[prashanth26]

(3) is also a good idea. It skipped my mind.

Let's increase the workers to 10 for now. And later we can decouple the drain logic as you mentioned in (3).

[prashanth26]

/reopen to keep track of this issue and implement Amshu's suggestion of part (3) eventually.

[prashanth26]

closing this issue in favor of - #48.