cluster autoscaler deleted non-empty node as ScaleDownEmpty

[infa-ddeore]

Which component are you using?:
cluster-autoscaler

What version of the component are you using?:
1.23

Component version:

What k8s version are you using (kubectl version)?: 1.23

kubectl version Output

$ kubectl version

What environment is this in?:
EKS

What did you expect to happen?:
cluster autoscaler should re-check node is empty or not before deleting

What happened instead?:
cluster autoscaler deleted non-empty node

How to reproduce it (as minimally and precisely as possible):
It is difficult to reproduce the issue, this behaviour is shows from the logs,

1. 06:46:13.400298 --> a pod is scheduled as per k8 api logs

I0322 06:46:13.400298      11 scheduler.go:675] "Successfully bound pod to node" pod="namespace-name/pod-name-6744f894fb-2nl44" node="ip-10-10-10-100.us-west-2.compute.internal" evaluatedNodes=16 feasibleNodes=4

2. 06:46:13.772090 --> after around ~371 ms, CA thinks node is empty even though above k8 api logs above show a pod is scheduled on the node

I0322 06:46:13.772090       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system" Name:"cluster-autoscaler-status", UID:"e02fdcd5-21d0-48df-8dba-4bbddd8ee247", APIVersion:"v1", ResourceVersion:"503217222", FieldPath:""}): type: 'Normal' reason: 'ScaleDownEmpty' Scale-down: removing empty node ip-10-10-10-100.us-west-2.compute.internal

3. I0322 06:46:13.786697 --> CA adds ToBeDeletedTaint taint to the node

I0322 06:46:13.786697       1 delete.go:103] Successfully added ToBeDeletedTaint on node ip-10-10-10-100.us-west-2.compute.internal

4. 06:46:14.056277 --> CA terminates the node directly thinking it is empty

I0322 06:46:14.056277       1 auto_scaling_groups.go:277] Terminating EC2 instance: i-xxxxxxxxxxxxxx

Anything else we need to know?:

[vadasambar]

Seems like CA's knowledge of the node is stale and since the ToBeDeleted taint is added after the pod is scheduled it's possible that the pod was scheduled after CA obtained node info. Something like this:

1. CA gets info about the node (including pods running on the node)
2. Scheduler schedules a pod on the node
3. CA adds ToBeDeleted taint (thinking the node has no running pods) and deletes the node

[infa-ddeore]

Seems like CA's knowledge of the node is stale and since the ToBeDeleted taint is added after the pod is scheduled it's possible that the pod was scheduled after CA obtained node info. Something like this:

1. CA gets info about the node (including pods running on the node)
2. Scheduler schedules a pod on the node
3. CA adds ToBeDeleted taint (thinking the node has no running pods) and deletes the node

yeah, this is what it looks like, is there a way to enforce CA to drain node in this scenario or any way to improve on this?

[infa-ddeore]

is there any flag or workaround to tackle this issue?

[vadasambar]

@infa-ddeore sorry I don't have enough bandwidth to look at this. If this is important, please bring it up in the sig-autoscaling meeting so that someone else can take a look. 🙏

[leoryu]

I think this issue is not just related ScaleDownEmpty case, but also related to DrainNode case. From

https://github.com/kubernetes/autoscaler/blob/702e9685d6c1d002f4a448f2d88007141dfa6d56/cluster-autoscaler/core/scaledown/actuation/drain.go#L84C26-L84C26

I think the nodeinfo might be stale, and ca will not know some new pod is scheduled to the node.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.