Add HCOMisconfiguredDescheduler runbook

[tiraboschi]

What this PR does / why we need it:
Add a runbook for HCOMisconfiguredDescheduler

A Descheduler is a Kubernetes application that causes the control plane to re-arrange the workloads in a better way.
It operates every pre-defined period and goes back to sleep after it had performed its job.

The descheduler uses the Kubernetes eviction API to evict pods, and receives feedback from kube-api whether the eviction request was granted or not.
On the other side, in order to keep VM live and trigger live-migration, KubeVirt handles eviction requests in a custom way and unfortunately a live migration takes time.
So from the descheduler's point of view, virt-launcher pods fail to be evicted, but they actually migrating to another node in background.
The descheduler notes the failure to evict the virt-launcher pod and keeps trying to evict other pods, typically resulting in it attempting to evict substantially all virt-launcher pods from the node triggering a migration storm.
In other words, the way KubeVirt handles eviction requests causes the descheduler to make wrong decisions and take wrong actions that could destabilize the cluster.
Using the descheduler operator with the LowNodeUtilization strategy results in unstable/oscillatory behavior if the descheduler is used in this way to migrate VMs.
To correctly handle the special case of VM pod evicted triggering a live migration to another node, the Kube Descheduler Operator introduced a profileCustomizations named devEnableEvictionsInBackground
which is currently considered an alpha feature on Kube Descheduler Operator side.
to prevent unexpected behaviours, if the Kube Descheduler Operator is installed and configured alongside HCO, HCO will check its configuration looking for the presence of devEnableEvictionsInBackground profileCustomizations eventually
suggesting to the cluster admin to fix the configuration of the Kube Descheduler Operator via an alert and its linked runbook.

In order to fix the configuration of the Kube Descheduler Operator to be suitable also for the KubeVirt use case,
something like:

apiVersion: operator.openshift.io/v1
kind: KubeDescheduler
metadata:
  name: cluster
  namespace: openshift-kube-descheduler-operator
spec:
  profileCustomizations:
    devEnableEvictionsInBackground: true

should be merged in its configuration.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes https://issues.redhat.com/browse/CNV-48734

Special notes for your reviewer:
It's a runbook for kubevirt/hyperconverged-cluster-operator#3100

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

• Design: A design document was considered and is present (link) or not required
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
• Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
• Community: Announcement to kubevirt-dev was considered

Release note:

Add HCOMisconfiguredDescheduler runbook

[avlitman]

page not found

[tiraboschi]

oh, it moved to: https://okd.io/docs/community/help.

Fixing only here now.
But we have to fix it (in another PR) on many other runbooks:
docs/deprecated_runbooks/KubeMacPoolDown.md:- OKD Help
docs/deprecated_runbooks/KubevirtHyperconvergedClusterOperatorNMOInUseAlert.md:- OKD Help
docs/deprecated_runbooks/VirtualMachineCRCErrors.md:- OKD Help
docs/runbooks/CDIDataImportCronOutdated.md:- OKD Help
docs/runbooks/CDIDataVolumeUnusualRestartCount.md:- OKD Help
docs/runbooks/CDIDefaultStorageClassDegraded.md:- OKD Help
docs/runbooks/CDIMultipleDefaultVirtStorageClasses.md:- OKD Help
docs/runbooks/CDINoDefaultStorageClass.md:- OKD Help
docs/runbooks/CDINotReady.md:- OKD Help
docs/runbooks/CDIOperatorDown.md:- OKD Help
docs/runbooks/CDIStorageProfilesIncomplete.md:- OKD Help
docs/runbooks/CnaoDown.md:- OKD Help
docs/runbooks/HPPNotReady.md:- OKD Help
docs/runbooks/HPPOperatorDown.md:- OKD Help
docs/runbooks/HPPSharingPoolPathWithOS.md:- OKD Help
docs/runbooks/KubeVirtDeprecatedAPIRequested.md:- OKD Help
docs/runbooks/KubeVirtNoAvailableNodesToRunVMs.md:- OKD Help
docs/runbooks/KubeVirtVMIExcessiveMigrations.md:- OKD Help
docs/runbooks/KubemacpoolDown.md:- OKD Help
docs/runbooks/LowReadyVirtControllersCount.md:- OKD Help
docs/runbooks/LowReadyVirtOperatorsCount.md:- OKD Help
docs/runbooks/LowVirtAPICount.md:- OKD Help
docs/runbooks/LowVirtControllersCount.md:- OKD Help
docs/runbooks/LowVirtOperatorCount.md:- OKD Help
docs/runbooks/NetworkAddonsConfigNotReady.md:- OKD Help
docs/runbooks/NoLeadingVirtOperator.md:- OKD Help
docs/runbooks/NoReadyVirtController.md:- OKD Help
docs/runbooks/NoReadyVirtOperator.md:- OKD Help
docs/runbooks/OrphanedVirtualMachineInstances.md:- OKD Help
docs/runbooks/OutdatedVirtualMachineInstanceWorkloads.md:- OKD Help
docs/runbooks/SSPDown.md:- OKD Help
docs/runbooks/SSPFailingToReconcile.md:- OKD Help
docs/runbooks/SSPHighRateRejectedVms.md:- OKD Help
docs/runbooks/SSPOperatorDown.md:- OKD Help
docs/runbooks/SSPTemplateValidatorDown.md:- OKD Help
docs/runbooks/VMStorageClassWarning.md:- OKD Help
docs/runbooks/VirtAPIDown.md:- OKD Help
docs/runbooks/VirtApiRESTErrorsBurst.md:- OKD Help
docs/runbooks/VirtApiRESTErrorsHigh.md:- OKD Help
docs/runbooks/VirtControllerDown.md:- OKD Help
docs/runbooks/VirtControllerRESTErrorsBurst.md:- OKD Help
docs/runbooks/VirtControllerRESTErrorsHigh.md:- OKD Help
docs/runbooks/VirtHandlerRESTErrorsBurst.md:- OKD Help
docs/runbooks/VirtHandlerRESTErrorsHigh.md:- OKD Help
docs/runbooks/VirtOperatorDown.md:- OKD Help
docs/runbooks/VirtOperatorRESTErrorsBurst.md:- OKD Help
docs/runbooks/VirtOperatorRESTErrorsHigh.md:- OKD Help

[sradco]

@tiraboschi is it possible to add a redirect?

[tiraboschi]

Not really sure, we should try asking on https://github.com/okd-project/okd-web/