KEP-3902: Decouple TaintManager from NodeLifeCycleController #3901

yuanchen8911 · 2023-03-07T18:23:09Z

One-line PR description: Code reorganization: decouple the TaintManager from the NodeLifecycleController to enable more flexible extension and enhancement in node taint-based eviction.

Issue link: Decouple TaintManager from NodeLifecycleController #3902

Other comments: Decouple Taint-based Pod Eviction from Node LifeCycle Controller kubernetes#115779

The proposal were presented to sig-apps and sig-node. An implementation is WIP.

yuanchen8911 · 2023-03-07T18:56:24Z

sig/node
sig/scheduling

yuanchen8911 · 2023-03-07T18:56:43Z

/cc @ddebroy @atosatto @ravisantoshgudimetla

k8s-ci-robot · 2023-03-07T18:56:46Z

@yuanchen8911: GitHub didn't allow me to request PR reviews from the following users: atosatto.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @ddebroy @atosatto @ravisantoshgudimetla

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

yuanchen8911 · 2023-03-07T19:03:20Z

/cc @atiratree @Huang-Wei

yuanchen8911 · 2023-03-07T19:03:37Z

/sig node scheduling

kerthcet · 2023-03-07T23:02:03Z

FYI: may this kubernetes/kubernetes#115840 related? I didn't read the KEP.

atosatto · 2023-03-07T23:19:07Z

FYI: may this kubernetes/kubernetes#115840 related? I didn't read the KEP.

@kerthcet we see this as a preliminary step aiming to remove deprecated features in NodeLifecycleController to enable the work proposed in this KEP.

yuanchen8911 · 2023-03-20T17:06:17Z

@soltysh @kow3ns, @alculquicondor, thanks a lot for the discussions on the proposal in today's sig-apps meeting. Your review on the KEP will be very appreciated. @Huang-Wei will provide feedback from the sig-scheduling perspective. Thanks again!

alculquicondor · 2023-03-20T17:47:35Z

ownership discussion kubernetes/kubernetes#116771

keps/sig-scheduling/3902-decoupled-taint-manager/README.md

SergeyKanzhelev · 2023-06-05T23:42:05Z

keps/sig-scheduling/3902-decoupled-taint-manager/README.md

+
+* Fix all reported bugs if any.
+
+### Upgrade / Downgrade Strategy


if somebody had the controller replaced, after the split the second one will stay? SO they will need to disable one more?

For downgrade, disabling the new feature using the feature gate will automatically disable a custom controller and use the current and default combined controller instead.

BTW, Aldo and other reviewers strongly suggested custom controller should not be the focus of the proposal.

yuanchen8911 · 2023-06-06T00:54:41Z

@SergeyKanzhelev , thanks for the comments! Can you take another look?

wojtek-t · 2023-06-06T07:45:19Z

And after that, we will pursue PRR review from @wojtek-t .

Queued - will take a look, but most probably Wed/Thu.

wojtek-t

The overall proposal makes sense to me (including going directly to Beta), but I have a bunch of comments about the PRR itself - PTAL

keps/sig-scheduling/3902-decoupled-taint-manager/README.md

wojtek-t · 2023-06-09T18:12:17Z

keps/sig-scheduling/3902-decoupled-taint-manager/README.md

+
+*This section must be completed when targeting alpha to a release.*
+
+* **How can this feature be enabled / disabled in a live cluster?**


Please update to the new KEP template (as mentioned above) - the feature-gate is then required explicitly :)

wojtek-t · 2023-06-09T18:16:12Z

keps/sig-scheduling/3902-decoupled-taint-manager/README.md

+    * The feature will work as usual.
+* **Are there any tests for feature enablement/disablement?**
+    * Planned for Beta release
+    * Appropriate tests have been added in the integration tests. See [Integration tests](#integration-tests) for more details.


These tests haven't been added, rather they are planned, right?

That being said, if I'm reading correctly, those tests are rather checking if things work fine when FG is enabled. They are not trying to test the enablement/disablement - being "run k8s with FG enabled, restart controller-manager with FG disabled and check if things still work as expected (and vice-versa)"

Which BTW is probably fine here, given this is in-memory feature (so no state is really stored here).
So I would be ok with saying sth like:
"No enablement/disablement tests are needed since this is in-memory feature and just regular tests with the feature being enabled/disabled do the job."

updated as suggested.

wojtek-t · 2023-06-09T18:16:47Z

keps/sig-scheduling/3902-decoupled-taint-manager/README.md

+*This section must be completed when targeting beta graduation to a release.*
+
+* **How can a rollout or rollback fail? Can it impact already running workloads?**
+  A feature gate `SeparateTaintManager` controls whether to use the split `TaintManager` or the old combined `NodeLifecycleController`.


Please answer the question. Can it impact already running workloads?

Does this make sense?

"This is an opt-in feature, and it does not change any default behavior. Unless there is a bug in the implementation, a rollout can not fail. If a rollout does fail, running workloads will not be evicted propertly on tainted nodes. We don't expect a rollback can fail."

wojtek-t · 2023-06-09T18:17:51Z

keps/sig-scheduling/3902-decoupled-taint-manager/README.md

+  A feature gate `SeparateTaintManager` controls whether to use the split `TaintManager` or the old combined `NodeLifecycleController`.
+
+* **What specific metrics should inform a rollback?**
+  N/A


This isn't N/A for sure.

Some huge number of pod evictions? [Please add appropriate metric]

How about "A significantly changing number of pod evictions and/or a substantial increase in pod eviction latency"?

wojtek-t · 2023-06-09T18:18:45Z

keps/sig-scheduling/3902-decoupled-taint-manager/README.md

+  N/A
+
+* **Were upgrade and rollback tested? Was the upgrade→downgrade→upgrade path tested?**
+  They will be covered in unit and e2e tests.


Will they? We don't really have downgrade e2e tests running anywhere.

Please run them manually and describe the test case
[https://github.com//pull/3658 is a nice example]

Added to test plan "manually verify the rollback and upgrade-downgrade-upgrade path will pass the e2e testing"

The overall proposal makes sense to me (including going directly to Beta), but I have a bunch of comments about the PRR itself - PTAL

Thanks for the review, Wojciech! I've updated the doc to (hopefully) address the comments :-). Can you please take another look when you get a chance? Thanks again!

wojtek-t · 2023-06-09T18:19:58Z

keps/sig-scheduling/3902-decoupled-taint-manager/README.md

+* **How can an operator determine if the feature is in use by workloads?**
+    * It can be determined by if the feature gate `SeparateTaintManager` is used.
+* **How can someone using this feature know that it is working for their instance?**
+    * Node taints and taint-based pod eviction work as usual. Admins and users should not see any different behavior if everything works as planned.


How can I determine it as an operator running gazillions clusters?

Please specify a metric (number of pod evictions is probably a best one).

Changed it to "Node taints and taint-based pod eviction should work as usual and there is no significant change in the number pod evictions (taint_manager_pod_evictions_total) and/or pod eviction latency (taint_manager_pod_eviction_latency)."

wojtek-t · 2023-06-09T18:20:17Z

keps/sig-scheduling/3902-decoupled-taint-manager/README.md

+* **What are the reasonable SLOs (Service Level Objectives) for the enhancement?**
+    * The performance of node taint-based eviction should remain the same level as before.
+* **What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?**
+    * The metrics for both `NodeLifecycleController` and `TaintManager`'s queues should stay the same levels as before.


What metrics? Please be specific.

How about "the number of pod evictions and pod eviction latency"?

taint_manager_pod_evictions_total taint_manager_pod_eviction_latency

wojtek-t · 2023-06-09T18:20:49Z

keps/sig-scheduling/3902-decoupled-taint-manager/README.md

+* **What are other known failure modes?**
+    * No
+* **What steps should be taken if SLOs are not being met to determine the problem?**
+    * If the default or custom taint-manager is working or not. 


I don't understand it - can you please clarify?

Updated to

If the pod eviction latency increases significantly, validate if the communication between NodeLifecycleController and TaintManager works. If the number of pod evictions is abnormal, run tests to verify the TaintManager works properly.

Add diagrams Update diagrams Update diagrams Update diagrams Resize diagram images Resize diagram images Resize diagram images Update image size Resize diagram images Fix a format issue Update images Update README.md Update README.md Update README.md Update README.md Update README.md Update README.md Add issue id Update toc Fix typos Fix a format issue Fix format issues Update decoupled-taint-manager KEP Fix a typo in kep.yaml Fix errors Revise README.md Update alternatives Update alternatives Update alterntives Update alternatives Update README Update keps/sig-scheduling/3902-decoupled-taint-manager/README.md Co-authored-by: Wei Huang <weih@hey.com> Address Wei Huang's comments Add Story 3 Remove backup files Address Aldo and Ravi's comments Update Goals Reformat code Reformat code Focus on code split Remove the backup file Address Aldo's comments Fix typos Fix an format issue Fix toc Update README.md Another revison to address comments Fix diagram error Update status in README.md Address sig-app comments Address Wei's comments Update Address final comments from sig-app/sig-scheduling Address sig-node comments Update risk and mitigation Update risk and mitigations Adress PRR review comments Fix typos Update status

SergeyKanzhelev

/lgtm

lgtm from SIG Node.

wojtek-t · 2023-06-12T08:14:00Z

This seems fine from PRR perspective - thanks!

/lgtm
/approve PRR

k8s-ci-robot · 2023-06-12T08:14:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Huang-Wei, SergeyKanzhelev, wojtek-t, yuanchen8911

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/prod-readiness/OWNERS~~ [wojtek-t]
~~keps/sig-scheduling/OWNERS~~ [Huang-Wei]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/apps Categorizes an issue or PR as relevant to SIG Apps. labels Mar 7, 2023

k8s-ci-robot requested review from kow3ns and soltysh March 7, 2023 18:23

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 7, 2023

yuanchen8911 changed the title ~~Add KEP on a new taint-manager~~ KEP: Decouple taint-manager from NodeLifeCycleController Mar 7, 2023

yuanchen8911 mentioned this pull request Mar 7, 2023

Decouple TaintManager from NodeLifecycleController #3902

Open

10 tasks

yuanchen8911 changed the title ~~KEP: Decouple taint-manager from NodeLifeCycleController~~ KEP-3902: Decouple taint-manager from NodeLifeCycleController Mar 7, 2023

yuanchen8911 force-pushed the master branch 2 times, most recently from ac1a89f to 91ba8c8 Compare March 7, 2023 18:46

k8s-ci-robot requested a review from ravisantoshgudimetla March 7, 2023 18:56

k8s-ci-robot requested a review from ddebroy March 7, 2023 18:56

k8s-ci-robot requested review from atiratree and Huang-Wei March 7, 2023 19:03

k8s-ci-robot added sig/node Categorizes an issue or PR as relevant to SIG Node. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. labels Mar 7, 2023

yuanchen8911 force-pushed the master branch 3 times, most recently from 18f3967 to f21c474 Compare March 20, 2023 15:52

SergeyKanzhelev reviewed Jun 5, 2023

View reviewed changes

keps/sig-scheduling/3902-decoupled-taint-manager/README.md Show resolved Hide resolved

SergeyKanzhelev reviewed Jun 5, 2023

View reviewed changes

yuanchen8911 force-pushed the master branch from e268340 to 33a2268 Compare June 6, 2023 00:53

yuanchen8911 requested a review from SergeyKanzhelev June 6, 2023 00:53

wojtek-t self-assigned this Jun 6, 2023

yuanchen8911 force-pushed the master branch 2 times, most recently from 4b5ff1e to 89f2ac7 Compare June 6, 2023 20:32

wojtek-t reviewed Jun 9, 2023

View reviewed changes

yuanchen8911 force-pushed the master branch 2 times, most recently from 17cd65c to 86dd1f5 Compare June 9, 2023 20:36

yuanchen8911 force-pushed the master branch from 86dd1f5 to 4438354 Compare June 9, 2023 20:37

yuanchen8911 requested a review from wojtek-t June 9, 2023 20:40

SergeyKanzhelev approved these changes Jun 9, 2023

View reviewed changes

k8s-ci-robot assigned SergeyKanzhelev Jun 9, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 9, 2023

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 12, 2023

k8s-ci-robot merged commit bdccc9c into kubernetes:master Jun 12, 2023

k8s-ci-robot added this to the v1.28 milestone Jun 12, 2023

soltysh mentioned this pull request Jun 15, 2023

KEP-4040: Controller name #4073

Closed

This was referenced Jul 10, 2023

[WIP] Docs for Decoupled TaintManager from NodeLifeCycleController (KEP-3902) kubernetes/website#41970

Closed

Decouple TaintManager from NodeLifeCycleController (KEP-3902) kubernetes/kubernetes#119208

Merged

atosatto mentioned this pull request Oct 18, 2023

Docs for Decoupled TaintManager from NodeLifeCycleController (KEP-3902) kubernetes/website#43555

Merged

yuanchen8911 mentioned this pull request Oct 24, 2023

Add a blog post about decoupled taint eviction controller kubernetes/website#43676

Merged

atosatto mentioned this pull request Mar 7, 2024

REQUEST: New membership for atosatto kubernetes/org#4804

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-3902: Decouple TaintManager from NodeLifeCycleController #3901

KEP-3902: Decouple TaintManager from NodeLifeCycleController #3901

yuanchen8911 commented Mar 7, 2023 •

edited

Loading

yuanchen8911 commented Mar 7, 2023

yuanchen8911 commented Mar 7, 2023

k8s-ci-robot commented Mar 7, 2023

yuanchen8911 commented Mar 7, 2023

yuanchen8911 commented Mar 7, 2023

kerthcet commented Mar 7, 2023

atosatto commented Mar 7, 2023 •

edited

Loading

yuanchen8911 commented Mar 20, 2023 •

edited

Loading

alculquicondor commented Mar 20, 2023

SergeyKanzhelev Jun 5, 2023

yuanchen8911 Jun 6, 2023 •

edited

Loading

yuanchen8911 commented Jun 6, 2023

wojtek-t commented Jun 6, 2023

wojtek-t left a comment

wojtek-t Jun 9, 2023

wojtek-t Jun 9, 2023

yuanchen8911 Jun 9, 2023

wojtek-t Jun 9, 2023

yuanchen8911 Jun 9, 2023 •

edited

Loading

wojtek-t Jun 9, 2023

yuanchen8911 Jun 9, 2023

wojtek-t Jun 9, 2023

yuanchen8911 Jun 9, 2023 •

edited

Loading

yuanchen8911 Jun 9, 2023

wojtek-t Jun 9, 2023

yuanchen8911 Jun 9, 2023 •

edited

Loading

wojtek-t Jun 9, 2023

yuanchen8911 Jun 9, 2023 •

edited

Loading

wojtek-t Jun 9, 2023

yuanchen8911 Jun 9, 2023 •

edited

Loading

SergeyKanzhelev left a comment

wojtek-t commented Jun 12, 2023

k8s-ci-robot commented Jun 12, 2023


		* Fix all reported bugs if any.

		### Upgrade / Downgrade Strategy


		This section must be completed when targeting alpha to a release.

		* How can this feature be enabled / disabled in a live cluster?

KEP-3902: Decouple TaintManager from NodeLifeCycleController #3901

KEP-3902: Decouple TaintManager from NodeLifeCycleController #3901

Conversation

yuanchen8911 commented Mar 7, 2023 • edited Loading

yuanchen8911 commented Mar 7, 2023

yuanchen8911 commented Mar 7, 2023

k8s-ci-robot commented Mar 7, 2023

yuanchen8911 commented Mar 7, 2023

yuanchen8911 commented Mar 7, 2023

kerthcet commented Mar 7, 2023

atosatto commented Mar 7, 2023 • edited Loading

yuanchen8911 commented Mar 20, 2023 • edited Loading

alculquicondor commented Mar 20, 2023

Choose a reason for hiding this comment

yuanchen8911 Jun 6, 2023 • edited Loading

Choose a reason for hiding this comment

yuanchen8911 commented Jun 6, 2023

wojtek-t commented Jun 6, 2023

wojtek-t left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuanchen8911 Jun 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuanchen8911 Jun 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuanchen8911 Jun 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuanchen8911 Jun 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuanchen8911 Jun 9, 2023 • edited Loading

Choose a reason for hiding this comment

SergeyKanzhelev left a comment

Choose a reason for hiding this comment

wojtek-t commented Jun 12, 2023

k8s-ci-robot commented Jun 12, 2023

yuanchen8911 commented Mar 7, 2023 •

edited

Loading

atosatto commented Mar 7, 2023 •

edited

Loading

yuanchen8911 commented Mar 20, 2023 •

edited

Loading

yuanchen8911 Jun 6, 2023 •

edited

Loading

yuanchen8911 Jun 9, 2023 •

edited

Loading

yuanchen8911 Jun 9, 2023 •

edited

Loading

yuanchen8911 Jun 9, 2023 •

edited

Loading

yuanchen8911 Jun 9, 2023 •

edited

Loading

yuanchen8911 Jun 9, 2023 •

edited

Loading