Kubernetes executor running slots leak fix #36240
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:providers
provider:cncf-kubernetes
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py
Show resolved
Hide resolved
airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py
Outdated
Show resolved
Hide resolved
airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py
Outdated
Show resolved
Hide resolved
dirrao
deleted the
35675-leak_in_kubernetes_executor_running_task_slots
branch
84 tasks
2 tasks
ephraimbuddy
pushed a commit
that referenced
this pull request
Jan 11, 2024
--------- Co-authored-by: gopal <gopal_dirisala@apple.com> (cherry picked from commit 49108e1)
68 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What happened
Schedulers are racing for pod adoption when there is a delay in schedulers' heartbeats. However, the schedulers are alive but not dead their heartbeat is delayed due to network timeout or heavy processing, etc. This leads to a leak in the executor.running_tasks slots. Eventually, the schedulers are not able to launch the pods due to executor.running_tasks reaches maximum parallelism.
What you think should happen instead
We should remove the entry from the Kubernetes executor running queue when worker pod deleted / moved to another scheduler.
Closes: #35675
Fix: #32928
Fix: #35426
Fix: #36478