Add support for the PodNodeSelector admission controller

[xmudrii]

What this PR does / why we need it:

Add support for the PodNodeSelector admission controller.

The admission controller is enabled by enabling the appropriate feature, such as:

...
features:
  podNodeSelector:
    enable: true
    config:
      configFilePath: "./podnodeselector.yaml"

The podnodeselector.yaml file is located on the local file system and contains the plugin configuration. More about the configuration format can be found in the Kubernetes docs.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #852

Special notes for your reviewer:

• To make the kube-system components able to get scheduled on all nodes, an empty scheduler.alpha.kubernetes.io/node-selector annotation is added to the kube-system namespace. The kube-system namespace mustn't be restricted in the plugin configuration, as pods such as CNI and kube-proxy must be scheduled on all nodes.
• Before the kube-system namespace gets annotated, kubeadm creates deployments/daemonsets for the needed components. The underlying pods have the incorrect node selectors and therefore are stuck in the Pending state. To resolve this problem, all pending pods in the kube-system namespace are removed, so they get recreated with the correct node selector.
• The AdmissionConfiguration API has been backported, so we don't have to import the k8s.io/apiserver package. Importing the package would require us to bump all Kubernetes dependencies to v1.18, as well as, controller-runtime to v0.5.0 or newer.

Does this PR introduce a user-facing change?:

Add support for the PodNodeSelector admission controller
BREAKING: Make paths for PodNodeSelector and StaticAudit configs relative to the manifest

/assign @kron4eg

[kron4eg]

While there is no technical obstacle to add this new feature to v1alpha1, I think we should avoid doing this, as v1alpha1 should be "conserved" and not changed anymore.

[xmudrii]

I've thought about this as well, but I think we should keep both APIs in sync until we don't officially announce the deprecation with the 1.0 release. After that, only v1beta1 should be extended.

[xmudrii]

@kron4eg PTAL

[kron4eg]

Suggested change

	ConfigFilePath string `json:"configFilePath"`
	Config string `json:"config"`

Let's have it inline instead of file

[xmudrii]

I've thought about this as well and I'm not sure this is a good approach. I think we should have the standardized API, even if it's a little bit redundant sometimes. A couple of things I have on mind:

• Config is a struct for all other features. Having it as a string here could introduce confusion
• The problem mentioned above could be solved by calling it ConfigFilePath like it's now, but then we come to another problem...
• If we ever have to add another field for any reason, we would drift from the standard API way too much and we would end up in a mess.

In the long run, I think it's better as it is right now.

[kron4eg]

Everything looks good, except one part that really bothers me, file references (namely ConfigFilePath fields), those are points to external resources that we have to dereference somehow. I know we already doing this and I'm concern about this too :)

Yet, the API design discussion probably doesn't belongs to this PR.

/lgtm
/approve

[kubermatic-bot]

LGTM label has been added.

Git tree hash: 2aa50f8ec7788a89f08d39c836dbf8ad7637c37c

[kubermatic-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kron4eg, xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [kron4eg,xmudrii]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment