Avoid the deletion of the machines in CrashLoopBackoff state by the safety controller

[AxiomSamarth]

What this PR does / why we need it:
This PR will avoid the deletion of the machines in CrashLoopBackoff state by the safety controller

Which issue(s) this PR fixes:
Fixes #583

Special notes for your reviewer:
More UTs shall be added to test this scenario in the near future. This one is an immediate solution to mitigate the issue.

Release note:

Avoid the deletion of the machines in CrashLoopBackoff state by the safety controller

[prashanth26]

Hi @AxiomSamarth ,

We would also need this fix and test on the OOT code path here - https://github.com/gardener/machine-controller-manager/blob/master/pkg/util/provider/machinecontroller/machine_safety.go#L222.

[AxiomSamarth]

Sure, @prashanth26 I will make the changes and update the PR. Thank you :)

[AxiomSamarth]

@amshuman-kr @prashanth26 For writing this UT, I had to introduce a new method Add() in the driver interface. Should it be retained or now that we understand how to handle the scenario, should we remove the Add() method from the interface and of course driver implementations?

I also see a similar thing to do in the out of tree code as well. I will have to add a new method in the interface. Am I doing it correctly or is there a different approach?

[amshuman-kr]

@AxiomSamarth Thanks for the unit test!

@amshuman-kr @prashanth26 For writing this UT, I had to introduce a new method Add() in the driver interface. Should it be retained or now that we understand how to handle the scenario, should we remove the Add() method from the interface and of course driver implementations?

I don't think it is a good idea to enhance the Driver interface for a functionality that is only used in the unit tests. That leads to nop implementations in proper drivers which is not desirable.

You can enhance just the FakeDriver to have an additional add function field and also make it implement an Add function which delegates to the add function field (but without adding the Add function to the Driver interface). You can also enhance the NewFakeDriver to take the additional function parameter. Then in the unit tests you can typecast the Driver instance returned by the NewFakeDriver to the FakeDriver struct type and call the Add function. This way you get the feature needed by the unit tests without disturbing other Driver implementations.

I also see a similar thing to do in the out of tree code as well. I will have to add a new method in the interface. Am I doing it correctly or is there a different approach?

You can use pretty much the same approach as I mentioned above for OOT as well. Or you can chose to even generate mocks for the OOT interface and simulate the responses that way. I am fine either way.

[amshuman-kr]

You can use pretty much the same approach as I mentioned above for OOT as well. Or you can chose to even generate mocks for the OOT interface and simulate the responses that way. I am fine either way.

I think the mock approach is more suitable for OOT.

[AxiomSamarth]

The recommended changes are made and pushed to this PR branch.

• The Driver Interface is restored to as it was before.
• FakeDriver struct implements the necessary methods for the unit tests for both in-tree and out-of-tree.
• The proposed logic by @prashanth26 to avoid the machines from being deleted by the safety controller is also included in this for both in-tree and out-of-tree safety controller code.

Looking forward for review comments/suggestions.

[amshuman-kr]

LGTM except for a small suggestion to move the fakeVMs as a private field in the FakeDriver struct for the OoT case.

[amshuman-kr]

Please move the instance construction inside the NewFakeDriver func. The func can take what parameters are required.

[amshuman-kr]

@AxiomSamarth Thanks for the changes. I am not sure there is a good mapping between the in-tree and the out-of-tree the way VMs are added and verified. The in-tree test case is clear to read. The way the fake driver in OoT stores, nodeName etcd in the struct, is the problem I think. I would suggest to improve the fake driver struct and the implementation to be more general (handle multiple VMs/nodes etc.)

[AxiomSamarth]

Hi Amshu,

I have not really made changes to the structure of the FakeDriver struct. Probably, because I could not think of any better way. However, I have modified the AddMachine method and also employed this AddMachine method in the NewFakeDriver struct to support handling multiple fakeVMs.

Also, with this change, I could add multiple fakeVMs and see through the test results that the machine in crashloopBackOff state remains from deletion while the other test machine in running state gets deleted.

[amshuman-kr]

Thanks for the changes. LGTM :-)

[prashanth26]

Overall looks good. Just a minor change. Also, squash all the commits once you are done.

[prashanth26]

/lgtm

[prashanth26]

This fix should also resolve https://github.com/gardener/machine-controller-manager/issues/544 according to colleagues from MS.