Support for external machine controller (driver) #460

prashanth26 · 2020-05-04T17:58:49Z

What this PR does / why we need it:
With this PR, new providers can be added out of tree (OOT) without sending in PRs to MCM repository. This OOT follows controller patterns where the provider is expected to implement their own machine controller. However, they can make use of the template machine controller provided as a library.

A sample machine controller for AWS can be found here - https://github.com/prashanth26/machine-controller-aws/commits/spliting-controllers2

So the basic idea is that - The existing path of MCM code flow remains as it is for now. So existing clusters running the MCM would work as is today --> AWS machineClaass --> machine ---> machineSet ---> machineDeployment all managed by the same code path at MCM repo.

Anyone trying to build new providers will make use of some new libraries which is already used by the sampleProvider repo and just implement the 5 basic methods following the documentation guide.

So basically a new provider will have to follow this guide - https://github.com/prashanth26/machine-controller-manager/blob/spliting-controllers2/docs/development/cp_support_new.md. In turn they will build his own MachineController and the MachineSet and MachineDeployment controller will be used from the existing MCM code.

And the new providers will make use of the generic MachineClass instead of ProviderSpecficiMachineClass.
MachineClass ---> machine --> Managed by their MC code
MachineSet ---> MachineDeployment ---> Managed by the core current MCM code (edited)

In the future we will migrate all provider code in MCM into OOT providers and get rid of in-tree code.

Which issue(s) this PR fixes:
Fixes #178

Special notes for your reviewer:

Release note:

Support for external / OOT (Out Of Tree) machine controller. A new provider can be maintained out of the core MCM repository.

amshuman-kr

Thanks a lot for the PR! The overall structure looks good and also the tests. I have asked a few questions and some suggestions too.

Also, why not move the default implementation in the AWS controller into somewhere under pkg/util/provider except for the construction of the driver.Driver implementation struct?

amshuman-kr · 2020-05-12T07:19:22Z

cmd/machine-controller-manager/app/options/options.go

+			KubeAPIBurst:              30,
+			LeaderElection:            leaderelectionconfig.DefaultLeaderElectionConfiguration(),
+			ControllerStartInterval:   metav1.Duration{Duration: 0 * time.Second},
+			ExternalMachineController: false,


Is there no way to do it dynamically by filtering machines, machinesets and machinesets if they refer to older typed machine classes? I.e. machines, machinesets and machinesets that refer to the generic machineclasses are filtered out because they will anyway be reconciled by the external machine controller.

Yes, we could do this. However, this would partly depend on how our migration logic would look like. For example if we continue working with machine objects the way they are today and ignore the machineClassType field to fetch the machine classes, we might need some logic like this.

But yes, I could revert this change for now and consider reconciling based on machineClassType ref in machine objects and reintroduce this change if necessary for migration. Reverting this commit.

Actually, this has less to do with migration logic and more to do with backward compatibility. For backward compatibility, you anyway need to check the machineclass kind. Just extend it to filter out the generic machineclass.

Yes Amshu, this check kind of already exists today - https://github.com/prashanth26/machine-controller-manager/blob/spliting-controllers2/pkg/controller/machine_util.go#L259-L260. Just reworded the log message a little.

So the current MCM will not process the genericMachineClass ref objects with the current code as well. I forgot to mention this earlier.

A minor improvement, currently the filtering is happening during reconciliation. It might be better to do the filtering before enquing, no?

Yes, I think that should be better. Shall make that change.

I have made the changes here - 61da081.

cmd/machine-controller-manager/app/options/options.go

pkg/controller/deployment.go

pkg/controller/machine.go

pkg/controller/machineset.go

pkg/util/provider/driver/driver.go

pkg/util/provider/machinecontroller/machineclass.go

prashanth26 · 2020-05-12T10:53:09Z

Also, why not move the default implementation in the AWS controller into somewhere under pkg/util/provider except for the construction of the driver.Driver implementation struct?

I was wondering wouldn't something like the ability of the provider to configure their own flags? - the https://github.com/prashanth26/machine-controller-aws/blob/spliting-controllers2/cmd/machine-controller/app/options/options.go#L82-L113. Wouldn't it be better for providers to be able to control this feature?

amshuman-kr · 2020-05-12T11:21:33Z

I was wondering wouldn't something like the ability of the provider to configure their own flags? - the https://github.com/prashanth26/machine-controller-aws/blob/spliting-controllers2/cmd/machine-controller/app/options/options.go#L82-L113. Wouldn't it be better for providers to be able to control this feature?

They could always do it like you have done right now, if they really what to support custom flags. WDYT?

prashanth26 · 2020-05-12T15:57:05Z

I was wondering wouldn't something like the ability of the provider to configure their own flags? - the https://github.com/prashanth26/machine-controller-aws/blob/spliting-controllers2/cmd/machine-controller/app/options/options.go#L82-L113. Wouldn't it be better for providers to be able to control this feature?

They could always do it like you have done right now, if they really what to support custom flags. WDYT?

I guess that is true. I shall move the controller initialization also into a library package like you suggested - f074b63.

amshuman-kr · 2020-05-18T14:29:27Z

@prashanth26 Thanks a lot for the changes. I just had another look. The PR looks almost ready!

Only real open point is the external controller flag.

prashanth26 · 2020-05-18T14:47:14Z

@prashanth26 Thanks a lot for the changes. I just had another look. The PR looks almost ready!

Okay, thank you.

Only real open point is the external controller flag.

Yes, currently I have reverted the changes and kept it to work as it was earlier. It ignores processing machine objects which have reference to any unknown machineClass kind here. This should work for now, however might need more thought once we plan to migrate objects like AWSMachineClass to MachineClass

ggaurav10

some minor suggestions.

pkg/util/provider/machinecodes/codes/code_string.go

pkg/util/provider/machinecodes/codes/codes.go

pkg/util/provider/machinecodes/status/status.go

pkg/util/provider/machinecontroller/machineclass.go

prashanth26 · 2020-06-04T06:20:45Z

Hi All,

I am done with all my changes for OOT. I hope i have addressed all your concerns. Please raise any major concerns you may have, minor concerns regarding documentation could probably be addressed with a follow-up PR. We plan to positively merge this PR by EOD and make a release.

Regards,
Prashanth

- Added new field to MachineStatus - lastKnownState - Relocated client builders for core & machine APIs - Added drain as package - Driver interface for external machine controllers - Updated CRDs to add generic MachineClass - Removed machineClass specific checks on MS/MD

Co-authored-by: Amshuman K R <amshuman.rao.karaya@sap.com>

Co-Authored by: Gaurav Gupta <ggaurav10>

Porting changes similar to on commit - gardener@fa7fa31 Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com> Porting changes to OOT implementation

prashanth26 · 2020-06-07T13:57:28Z

Hi All,

I have just squashed a few commits for better readability. I have also merged the master with the current branch.

prashanth26 added the kind/epic Large multi-story topic label May 4, 2020

prashanth26 requested review from ggaurav10 and a team as code owners May 4, 2020 17:58

prashanth26 added the do-not-merge/work-in-progress label May 4, 2020

prashanth26 mentioned this pull request May 4, 2020

[WIP]Support for out-of-tree/external cloud-provider specific drivers #209

Closed

gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels May 10, 2020

amshuman-kr suggested changes May 12, 2020

View reviewed changes

gardener-robot-ci-3 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels May 12, 2020

gardener-robot-ci-1 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label May 12, 2020

gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label May 12, 2020

prashanth26 force-pushed the spliting-controllers2 branch from 42c76aa to 655c6dd Compare May 12, 2020 16:01

gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label May 12, 2020

gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label May 12, 2020

prashanth26 force-pushed the spliting-controllers2 branch from 655c6dd to 793b105 Compare May 12, 2020 16:03

gardener-robot-ci-2 added reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) and removed reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) labels May 12, 2020

prashanth26 mentioned this pull request May 13, 2020

Align AWS implementation with controller based implementation gardener/machine-controller-manager-provider-aws#10

Merged

ggaurav10 reviewed May 20, 2020

View reviewed changes

gardener-robot-ci-2 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label May 26, 2020

gardener-robot-ci-1 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 4, 2020

prashanth26 force-pushed the spliting-controllers2 branch from 40af2b0 to a8f6783 Compare June 4, 2020 06:13

gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 4, 2020

gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 4, 2020

ggaurav10 approved these changes Jun 4, 2020

View reviewed changes

prashanth26 and others added 11 commits June 7, 2020 17:51

Apply suggestions from code review

3171a80

Co-authored-by: Amshuman K R <amshuman.rao.karaya@sap.com>

Library to initialize machine controller

71cc972

Updating docs

47946ea

Minor code readability changes

7b20dc5

Taking in code change suggestions

0c6f46b

Co-Authored by: Gaurav Gupta <ggaurav10>

Bugfix: Issues with secrets/finalizers/lastKnownState

106b397

Removed retry field from driver API responses

74f02c3

Updated documentation for OOT provider impl

c6514fa

Skip drain for machines that are NotReady

9fd3513

Porting changes similar to on commit - gardener@fa7fa31 Signed-off-by: Andrey Klimentyev <andrey.klimentyev@flant.com> Porting changes to OOT implementation

Merge branch 'master' into spliting-controllers2

f13e506

prashanth26 force-pushed the spliting-controllers2 branch from a8f6783 to f13e506 Compare June 7, 2020 12:23

gardener-robot-ci-2 added the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 7, 2020

gardener-robot-ci-3 removed the reviewed/ok-to-test Has approval for testing (check PR in detail before setting this label because PR is run on CI/CD) label Jun 7, 2020

prashanth26 added status/accepted Issue was accepted as something we need to work on and removed do-not-merge/work-in-progress labels Jun 7, 2020

prashanth26 merged commit 9a35595 into gardener:master Jun 8, 2020

This was referenced Aug 13, 2020

Improvements for MCM #287

Closed

Improve deletion logic of machines #350

Closed

This was referenced Sep 15, 2020

Move from in-tree to out-of-tree MCM gardener/gardener-extension-provider-gcp#174

Closed

Migration from in-tree to out-of-tree MCM controllers gardener/gardener-extension-provider-gcp#175

Merged

prashanth26 deleted the spliting-controllers2 branch May 2, 2021 07:45

prashanth26 mentioned this pull request Jul 21, 2021

Fix any cache/server inconsistencies #437

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for external machine controller (driver) #460

Support for external machine controller (driver) #460

prashanth26 commented May 4, 2020 •

edited

Loading

amshuman-kr left a comment

amshuman-kr May 12, 2020

prashanth26 May 12, 2020

amshuman-kr May 18, 2020

prashanth26 May 18, 2020

amshuman-kr May 18, 2020 •

edited

Loading

prashanth26 May 18, 2020

prashanth26 May 27, 2020

prashanth26 commented May 12, 2020

amshuman-kr commented May 12, 2020

prashanth26 commented May 12, 2020 •

edited

Loading

amshuman-kr commented May 18, 2020

prashanth26 commented May 18, 2020 •

edited

Loading

ggaurav10 left a comment

prashanth26 commented Jun 4, 2020

prashanth26 commented Jun 7, 2020

Support for external machine controller (driver) #460

Support for external machine controller (driver) #460

Conversation

prashanth26 commented May 4, 2020 • edited Loading

amshuman-kr left a comment

Choose a reason for hiding this comment

amshuman-kr May 12, 2020

Choose a reason for hiding this comment

prashanth26 May 12, 2020

Choose a reason for hiding this comment

amshuman-kr May 18, 2020

Choose a reason for hiding this comment

prashanth26 May 18, 2020

Choose a reason for hiding this comment

amshuman-kr May 18, 2020 • edited Loading

Choose a reason for hiding this comment

prashanth26 May 18, 2020

Choose a reason for hiding this comment

prashanth26 May 27, 2020

Choose a reason for hiding this comment

prashanth26 commented May 12, 2020

amshuman-kr commented May 12, 2020

prashanth26 commented May 12, 2020 • edited Loading

amshuman-kr commented May 18, 2020

prashanth26 commented May 18, 2020 • edited Loading

ggaurav10 left a comment

Choose a reason for hiding this comment

prashanth26 commented Jun 4, 2020

prashanth26 commented Jun 7, 2020

prashanth26 commented May 4, 2020 •

edited

Loading

amshuman-kr May 18, 2020 •

edited

Loading

prashanth26 commented May 12, 2020 •

edited

Loading

prashanth26 commented May 18, 2020 •

edited

Loading