Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve LowNodeUtilization's consideration of evictable pods #529

Closed
damemi opened this issue Mar 17, 2021 · 19 comments · Fixed by #790
Closed

Improve LowNodeUtilization's consideration of evictable pods #529

damemi opened this issue Mar 17, 2021 · 19 comments · Fixed by #790
Assignees
Labels
kind/feature Categorizes issue or PR as related to a new feature.

Comments

@damemi
Copy link
Contributor

damemi commented Mar 17, 2021

Similar to how we check that a pod fits on other nodes before evicting in strategies like Affinity/TopologySpread, it would be an improvement if LowNodeUtilization checked that a pod from an overutilized node can fit on the intended underutilized node before evicting it (ie, check for taints/affinity). This could improve the performance of the strategy and prevent unnecessary infinite evictions each run.

@damemi damemi added the kind/feature Categorizes issue or PR as related to a new feature. label Mar 17, 2021
@RyanDevlin
Copy link
Contributor

I can try this one.

@RyanDevlin
Copy link
Contributor

/assign

@RyanDevlin
Copy link
Contributor

@damemi I've noticed that the Affinity and TopologySpread strategies check for affinity on other nodes, but don't necessarily check for taints. Is checking for taints on other nodes still something that should be included in this feature?

@rustrial
Copy link

@RyanDevlin I guess eviction only makes sense if the Pod can be scheduled on another node, therefore it seems reasonable that all scheduling constraints (Affinity, TopologySpread and Taints) should be checked. Otherwise we would just evict a Pod to see it being scheduled to the same node again, which just increases the risk of service outages caused by Pod eviction (restart).

@RyanDevlin
Copy link
Contributor

@rustrial At the bottom of #551 @ingvagabund commented about how, for the purposes of this feature, it would be overkill to take into account every filtering plugin. I'm currently implementing this feature with checks for NodeAffinity, Taints, and NodeSelector. The optimization isn't perfect, but statistically it should improve performance.

@ingvagabund
Copy link
Contributor

Overkill on the code level view (duplicating the code since we can't import kubernetes/kubernetes code).

@ingvagabund
Copy link
Contributor

Otherwise we would just evict a Pod to see it being scheduled to the same node again, which just increases the risk of service outages caused by Pod eviction (restart).

That's responsibility of PDB to take care of disruptions.

@damemi
Copy link
Contributor Author

damemi commented Apr 21, 2021

@RyanDevlin I guess eviction only makes sense if the Pod can be scheduled on another node

I mentioned this in the other thread (#551 (comment)), sorry for not bringing it up here... but there are use cases where a pod could be evicted regardless of if it fits on another node. For strategies like PodLifetime, you might actually want to evict a pod that will only be recreated on the same node

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jul 20, 2021
@ingvagabund
Copy link
Contributor

/remove-lifecycle stale

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 19, 2021
@pravarag
Copy link
Contributor

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 19, 2021
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 17, 2022
@ingvagabund
Copy link
Contributor

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 17, 2022
@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue or PR as fresh with /remove-lifecycle stale
  • Mark this issue or PR as rotten with /lifecycle rotten
  • Close this issue or PR with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 17, 2022
@jecnua
Copy link

jecnua commented Apr 17, 2022

Still valid

@damemi
Copy link
Contributor Author

damemi commented Apr 18, 2022

Bumped the NodeFit PR to get that moving again
/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 18, 2022
@ingvagabund
Copy link
Contributor

Fixed in #790
/close

@k8s-ci-robot
Copy link
Contributor

@ingvagabund: Closing this issue.

In response to this:

Fixed in #790
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment