Descheduler not evicting pods with node selectors

[gaurav1086]

What version of descheduler are you using?

descheduler version: v0.20.0

Does this issue reproduce with the latest release?
Yes

Which descheduler CLI options are you using?
N/A

Please provide a copy of your descheduler policy config file

  policy.yaml: |
    apiVersion: "descheduler/v1alpha1"
    kind: "DeschedulerPolicy"
    strategies:
      RemovePodsViolatingInterPodAntiAffinity:
         enabled: true
      RemovePodsViolatingTopologySpreadConstraint:
         enabled: true

What k8s version are you using (kubectl version)?

kubectl version Output

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6", GitCommit:"fbf646b339dc52336b55d8ec85c181981b86331a", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:30Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.8", GitCommit:"fd5d41537aee486160ad9b5356a9d82363273721", GitTreeState:"clean", BuildDate:"2021-02-17T12:33:08Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

What did you do?

We tried to rebalance pods on a group of nodes with the same (and only 1) Node-Selector.

In our case, we have 3 eligible nodes with the node selector say x=y where we are running a total of 15 pods. The current pod distribution is 7,5,3 .

What did you expect to see?

The pods should be balanced. Expected pod distribution should be: 5,5,5 .

What did you see instead?

Descheduler refuses to evict pods from nodes with the following error in the logs:
{"level":"INFO","message":"\"Ignoring pod for eviction due to node selector/affinity\" pod=\"mypod\""}

[damemi]

Hi @gaurav1086,
This message is logged by the TopologySpreadConstraint strategy when a pod that has been chosen for eviction has a hard selector/affinity requirement that only matches the current node that the pod is on.

The reasoning is that if the pod's affinity only matches the current node, evicting it will not help any balancing because it will just be scheduled back onto the same node. See the code here:

descheduler/pkg/descheduler/strategies/topologyspreadconstraint.go

Lines 287 to 297 in 6b8d4cd

	// if the pod has a hard nodeAffinity or nodeSelector that only matches this node,
	// don't bother evicting it as it will just end up back on the same node
	// however we still account for it "being evicted" so the algorithm can complete
	// TODO(@damemi): Since we don't order pods wrt their affinities, we should refactor this to skip the current pod
	// but still try to get the required # of movePods (instead of just chopping that value off the slice above)
	if aboveToEvict[k].Spec.NodeSelector != nil ||
	(aboveToEvict[k].Spec.Affinity != nil &&
	aboveToEvict[k].Spec.Affinity.NodeAffinity != nil &&
	aboveToEvict[k].Spec.Affinity.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution != nil &&
	nodesPodFitsOnBesidesCurrent(aboveToEvict[k], nodeMap) == 0) {
	klog.V(2).InfoS("Ignoring pod for eviction due to node selector/affinity", "pod", klog.KObj(aboveToEvict[k]))

[gaurav1086]

@damemi Thanks for the response. In our case, we have 3 eligible nodes with that (one) node selector say x=y where we are running a total of 15 pods. The current distribution is 7,5,3 . Expected output is: 5,5,5 . (Updated description above as well)

Shouldn't the first condition in the if ( aboveToEvict[k].Spec.NodeSelector != nil ) not also check if there are any available eligible nodes with that node selector (x=y) where it can (possibly) move the current pod to ?

[damemi]

Shouldn't the first condition in the if ( aboveToEvict[k].Spec.NodeSelector != nil ) not also check if there are any available eligible nodes with that node selector (x=y) where it can (possibly) move the current pod to ?

Ah, you are right. I think that is a bug. I will open a fix for this (Edit: PR opened here: #525)
/assign

Could you also share an example pod yaml of one of the pods that should be evicted?