diff --git a/docs/runbooks/HCOMisconfiguredDescheduler.md b/docs/runbooks/HCOMisconfiguredDescheduler.md new file mode 100644 index 0000000..947e978 --- /dev/null +++ b/docs/runbooks/HCOMisconfiguredDescheduler.md @@ -0,0 +1,73 @@ +# HCOMisconfiguredDescheduler + +## Meaning + +A Descheduler is a Kubernetes application that causes the control plane to +re-arrange the workloads in a better way. + +The descheduler uses the Kubernetes eviction API to evict pods, and receives +feedback from `kube-api` whether the eviction request was granted or not. +On the other side, in order to keep VM live and trigger live-migration, +KubeVirt handles eviction requests in a custom way and unfortunately +a live migration takes time. +So from the descheduler's point of view, `virt-launcher` pods fail to be +evicted, but they actually migrating to another node in background. +So the way KubeVirt handles eviction requests causes the descheduler to +make wrong decisions and take wrong actions that could destabilize the cluster. + + +To correctly handle the special case of `VM` pod evicted triggering a live +migration to another node, the `Kube Descheduler Operator` introduced +a `profileCustomizations` named `devEnableEvictionsInBackground` +which is currently considered an `alpha` feature +on `Kube Descheduler Operator` side. + +## Impact + +Using the descheduler operator for KubeVirt VMs without the +`devEnableEvictionsInBackground` profile customization can lead to +unstable or oscillatory behavior, undermining cluster stability. + +## Diagnosis + +1. Check the CR for `Kube Descheduler Operator`: + + ```bash + $ kubectl get -n openshift-kube-descheduler-operator KubeDescheduler cluster -o yaml + ``` + +looking for: + + ```yaml + spec: + profileCustomizations: + devEnableEvictionsInBackground: true + ``` + +If not there, the `Kube Descheduler Operator` is not correctly configured +to work alongside KubeVirt. + +## Mitigation + +Set: + ```yaml + spec: + profileCustomizations: + devEnableEvictionsInBackground: true + ``` +on the CR for the `Kube Descheduler Operator` or +remove the `Kube Descheduler Operator`. + +Please notice that `EvictionsInBackground` is an alpha feature, +and it's subject to change and currently provided as an +experimental feature. + + + +If you cannot resolve the issue, see the following resources: + +- [OKD Help](https://okd.io/docs/community/help) +- [#virtualization Slack channel](https://kubernetes.slack.com/channels/virtualization) +