Drop capabilities, enable seccomp and enforce runAsNonRoot

[aryan9600]

Further restricts the SecurityContext that the controller runs under, by enabling the default seccomp profile and dropping all linux capabilities.
This was set at container-level to ensure backwards compatibility with use cases in which sidecars are injected into the source-controller pod
without setting less restrictive settings.
Add a uid and gid for the container to enforce runAsNonRoot and ensure
the use of non root users.

BREAKING CHANGES:

1. The use of new seccomp API requires Kubernetes 1.19.
2. the controller container is now executed under 65534:65534 (userid:groupid).
   This change may break deployments that hard-coded the user name 'controller' in their PodSecurityPolicy.

Ref: fluxcd/flux2#2014

Signed-off-by: Sanskar Jaiswal sanskar.jaiswal@weave.works
Co-authored-by: Paulo Gomes paulo.gomes@weave.works

[pjbgf]

This was tested on Kubernetes 1.23.1 with restrict "pod security" (below) and it worked as expected.

apiVersion: v1
kind: Namespace
metadata:
  name: flux-system
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: latest

[stefanprodan]

We need to move the "Required for AWS IAM Role bindings" comment, like in the source-controller PR.

[stefanprodan]

@pjbgf can you confirm that without runAsNonRoot Kubernetes rejects the kustomize-controller pod? Can you post here the apply error please.

[stefanprodan]

@aryan9600 please link this PR to its upstream issue.

[pjbgf]

@stefanprodan here's the message when runAsNonRoot is not set to true:
(combined from similar events): Error creating: pods "kustomize-controller-b86cf589b-kg6k5" is forbidden: violates PodSecurity "restricted:latest": runAsNonRoot != true (container "manager" must not set securityContext.runAsNonRoot=false)