fix: promote system addons to system-cluster-critical

[jackfrancis]

Reason for Change:

This PR updates blobfuse-flexvolume and keyvault-flexvolume daemonsets to the system-cluster-critical PriorityClass to ensure that they are properly prioritized during pod scheduling.

For the nmi daemonset in the aad-pod-identity addon, we migrated the namespace to kube-system, as system-cluster-critical isn't allowed in the default namespace.

Specifically, during scale up events in response to more pods than available scheduleable nodes (e.g., cluster-autoscaler), we want to ensure that system daemonsets are scheduled first of all on new nodes, so that nodes don't reach pod capacity w/ non-system-critical pods before these daemonsets have a chance to be scheduled.

Issue Fixed:

Azure/aad-pod-identity#471
Fixes #2532
Fixes #2531

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version

Notes:

[codecov]

Codecov Report

Merging #2533 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #2533   +/-   ##
=======================================
  Coverage   72.11%   72.11%           
=======================================
  Files         130      130           
  Lines       24460    24460           
=======================================
  Hits        17639    17639           
  Misses       5799     5799           
  Partials     1022     1022

[mboersma]

/lgtm

[acs-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jackfrancis, mboersma

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jackfrancis,mboersma]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jackfrancis]

@palma21 @yangl900 FYI, we observed that ensuring all system addons/components have a DaemonSet PriorityClass of system-cluster-critical or system-node-critical is important during cluster-autoscaler scale up scenarios. E.g., if cluster-autoscaler adds more nodes to ensure that Pending pods have schedulable nodes, we want to ensure that system daemonsets get scheduling priority when those new nodes online.

[palma21]

Thanks for the heads up we do have that on our backlog to add to a few that are missing.

FYI, a PDB to non DS system-cluster-critical or system-node-critical pods is required or things like core DNS can block an otherwise empty node to scale down.