NodeController should add NoSchedule taints and we should get rid of getNodeConditionPredicate()

[davidopp]

getNodeConditionPredicate() in plugin/pkg/scheduler/factory/factory.go makes our code hard to understand because it hides the node-condition-based filtering in the node lister, which is totally non-obvious.

We should get rid of this function and have NodeController add NoSchedule taints for these situations instead. (Alas, we might not actually be able to get rid of getNodeConditionPredicate() completely until we get rid of the Unschedulable field of PodSpec, but at least we can get rid of all the other code in this function.) If there are pods that should still be able to schedule in any of these situations (e.g. DaemonSet pods?) we should add tolerations in admission control for them (e.g. see pkg/controller/daemon/daemoncontroller.go).

cc/ @kubernetes/sig-scheduling-misc @kubernetes/sig-cluster-lifecycle-misc
cc/ @gmarek @kevin-wangzefeng

Sub-tasks according to the design doc:

• Add node taints label and feature flag (Task 0: Added node taints labels and feature flags #49547)
• In the node controller, taint Nodes according to the Node Condition (Task 1: Tainted node by condition. #49257)
• In DaemonSet, update DaemonSetController to Tolerant new Taints (Task 2: Added toleration to DaemonSet pods for node condition taints #50186)
• In admissionController, add MemoryPressure/DiskPressure toleration for no BestEffort pod (Task 3: Add MemoryPressure toleration for no BestEffort pod. #50180)
• In scheduler, disable the current behaviour of filtering out Nodes. Instead, pods will not be scheduled to tainted nodes if a toleration does not exist in its PodSpec (Task 4: Ignored node condition predicates if TaintsByCondition enabled. #50185)
• Add e2e test for this feature (Apply algorithm in scheduler by feature gates. #52723)
• Update doc for this feature (Add documentation for TaintNodesByCondition website#5352)

[resouer]

until we get rid of the Unschedulable field of PodSpec

I think you are referring Unschedulable field of NodeSpec?

[davidopp]

I think you are referring Unschedulable field of NodeSpec?

Yes sorry, that was a typo.

[gmarek]

@resouer - please CC me to all PRs

[bgrant0607]

See also #29178

[davidopp]

Note that memory and disk pressure are checked in separate predicates, not in the NodeConditionPredicate. These should be represented as NoSchedule taints too.

kubernetes/plugin/pkg/scheduler/algorithm/predicates/predicates.go

Line 1239 in e7dfdd5

func CheckNodeMemoryPressurePredicate(pod *v1.Pod, meta interface{}, nodeInfo *schedulercache.NodeInfo) (bool, []algorithm.PredicateFailureReason, error) {

[davidopp]

Setting NoSchedule taints corresponding to the node conditions that today cause pods not to schedule onto a node, should also allow us to use tolerations rather than special-case code in the DaemonSet controller (and thereby make it easier to have DaemonSet pods be scheduled by the default scheduler.)

[davidopp]

ref/ #42002

[k82cn]

Sub-tasks according to the design doc, I'll create PR for them accordingly:

The task list was moved to the issue description here.

[wanghaoran1988]

/cc @mdshuai

[k82cn]

/reopen

[k8s-ci-robot]

@k82cn: you can't re-open an issue/PR unless you authored it or you are assigned to it.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k82cn]

/assign

[gmarek]

We won't, but it'll take time for us to get there. We'll roll out Tainting as alpha (obviously), later move it to beta and GA in 1.10 at the earliest. We can't remove Condition checking logic before that.

dsc is a daemon set right? New taints are NoSchedule only, so running Pods won't be affected (at least on the control plane level). Kubelet can still decide to evict them based on problem it observes.

[yastij]

@gmarek - seems logic for the timeline.

From @k82cn's sub-tasks

In DaemonSet, update DaemonSetController to Tolerant new Taints

Since new Taints are NoSchedule Only, Daemon set shouldn't have toleration for these ?

It should instead respect the new taints when applied and not schedule on these nodes, or I'm I missing something ? @lukaszo

[k82cn]

The task for DS in my mind:

1. For all DS pods, it should tolerant MemoryPressure & DiskPressure
2. For critical DS pods, it should tolerant OutOfDisk

And if the taint by condition feature is enabled, we should check OutOfDisk by taints instead of predicates; the refactor is trying to make this simpler :).

[yastij]

@k82cn - You're right, I'll send a PR after yours get merged :)

[jamiehannaford]

@k82cn Do we have a to do list so folks can help out with kubernetes/community#819? I'd like to help when I get some time.

@gmarek I have a question about timeline. If alpha makes it into v1.8, will I be able to schedule to NotReady nodes (no CNI) if I reference the alpha tolerations? Or do you mean it won't be functional at all until v1.10?

[gmarek]

It means you'll need to flip feature gate flag for it to work.

[k82cn]

@jamiehannaford , here is the task list: #42001 (comment)

I'm working on related PRs for them; please ping me when you got time :).

[derekwaynecarr]

@k82cn - i anticipate we will also add a CPUPressure condition which will also mean that no BestEffort pod can be scheduled to that node if the kubelet is running with a static cpu assignment policy.

/cc @sjenning

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[gmarek]

I think it's done (as alpha). @davidopp - do we want to keep this issue open?

[bsalamat]

We can close this.