Use PriorityClass

[Bezbran]

Overview

Kubernetes offers a feature called PriorityClass which empowers cluster operators to determine the relative priority of pods. For our setup, pods such as the node-agent daemonset and kube-vuln should inherently possess a higher priority compared to other application pods.

Issue

Currently, the absence of a defined PriorityClass for our pods results in certain pods (like node-agent, kube-vuln, etc.) lingering in the "Pending" state. This persists until a cluster operator intervenes by manually deleting pods from nodes.

Proposed Solution

We should leverage PriorityClasses to address this:

• system-node-critical: This should be assigned to node-agent pods, ensuring they run on every node without exception.

• system-cluster-critical: This is apt for the remaining pods (operator, kubescape, kube-vuln, etc.) as they are essential for maintaining the cluster's health.

[tchatzig]

It would be nice to be able to set up the priorityclass on the helm chart values.

like this chart for example
https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml#L890