[proposal] Using Kyverno to support container runtime neutrally

[YTGhost]

Issue Description

Background

The Proposal(Using image proxy shim to support container runtime neutrally) was discussed at an interim community meeting on November 9, 2022. This proposal proposes to use cri shim to implement image proxy, so that we don't need to change the docker logic.

During the meeting we discussed what might be a more elegant approach than implementing a cri shim, thus trying to achieve the results we want in a shorter path.

The current community discussion has resulted in two scenarios：

1. Sealer run/apply renders resources that meet certain specifications when submitting k8s resources

2. Support via k8s webhook

This Proposal will investigate the case of support through k8s webhook

Introduction

What is a Kubernetes Pod Policy

In fact, what we want to implement is a Kubernetes Pod Policy. It mean a set of mechanisms for ensuring validating and remediation controls over Pods and their attributes. Using this mechanism, we can redirect image pull requests when the Pod is created.

After researching the Kubernetes Pod Policy, there are three research targets as follows:

• Pod Security Admission
• Gatekeeper
• Kyverno

Pod Security Admission

With the upcoming deprecation and subsequent removal of Pod Security Policies (PSPs) in Kubernetes. Pod Security Admission is the new, albeit incomplete, successor. But it only checks Pods and does not support modifications to Pods, so it does not meet our needs.

Gatekeeper Introduction

Gatekeeper is an open-source project and collaboration between a number of companies including Google and Microsoft, later donated to the CNCF. Now in its third iteration, Gatekeeper is the Kubernetes-specific implementation of Open Policy Agent (OPA), a general-purpose policy engine. Because of the relationship between Open Policy Agent with Gatekeeper, the project is often written "OPA/Gatekeeper" to acknowledge these ties. Gatekeeper enables request validation and, most recently, mutation. A major defining characteristic of Open Policy Agent is its reliance upon the use of a specialized programming language called Rego in order to implement the logic necessary for policy decisions. Through Rego, OPA is designed to be broadly applicable to disparate software and not Kubernetes exclusively, as well as achieve a high level of logic execution.

Kyverno Introduction

Kyverno is an open-source project originally from Nirmata and later donated to the CNCF. Like Gatekeeper, Kyverno is a Kubernetes policy engine with validation and mutation abilities, however it adds resource generation in addition to, most recently, API object lookups. Unlike Gatekeeper, Kyverno was written from the start solely for use in Kubernetes. A major defining characteristic of Kyverno is its generation ability and the architecture which does not require specialized knowledge of a programming language to render policy decisions. Primarily as a reaction to this requirement of a specialized language, Kyverno differentiates itself from Gatekeeper through its model of simplicity.

Kyverno vs Gatekeeper

Features/Capabilities	Gatekeeper	Kyverno
Validation	✓	✓
Mutation	✓*	✓
Generation	X	✓
Image Verification	X (via extensions)	✓
Image Registry lookups	X (via extensions)	✓
Extensions	✓	X
Policy as native resources	✓	✓
Metrics exposed	✓	✓
OpenAPI validation schema (kubectl explain)	X**	✓
High Availability	✓	✓
API object lookup	✓	✓
CLI with test ability	✓***	✓
Policy audit ability	✓	✓
Self-service reports	X	✓

* Beta status

** Not available with all CRDs

*** Separate CLI

Analysis

Gatekeeper Advantages

• Capable of expressing very complex policy
• Supports multiple replicas for availability and scale

Gatekeeper Disadvantages

• Requiring a programming language is a serious learning curve which generates large amounts of technical debt and increases time to value
• Mutation ability is in nascent stage
• No generation ability means its usefulness is limited to predominantly validation use cases
• Policy is complex, often lengthy, and requires multiple documents to implement

Kyverno Advantages

• Dramatic simplicity of policy expression allowing Kubernetes-style composition
• Solid mutation abilities
• Supports multiple replicas for availability and scale
• Generation and syncing ability is unique and allows new use cases
• Quick time to value and high degree of flexibility

Kyverno Disadvantages

• Highly complex policy may not be possible since no programming language is exposed
• Young, still largely unproven due to lower establishment

To summarize, if you need to use application policies across systems or if you need to implement highly complex policies, you can use Gatekeeper. Otherwise, Gatekeeper offers no benefit.

In our case, we only use it under kubernetes and need a simple strategy, so Kyverno is the best choice for us.

How Kyverno works

Admission Controllers

If you want to add a new Admission controller, you need to compile it with kube-apiserver and restart kube-apiserver, which is a tedious task.

Dynamic Admission Control

Two special admission controllers:

• MutatingAdmissionWebhook
• ValidatingAdmissionWebhook

You can use the two admission controllers to configure some custom HTTP servers to provide webhooks and execute user-specific logic in these HTTP servers, thus extending the capabilities of the admission controller without recompiling kube-apiserver

Kyverno's Architecture

• Webhook handles AdmissionReview requests from the Kubernetes API server
• Monitor component creates and manages required configurations
• PolicyController watches policy resources and initiates background scans based on the configured scan interval
• GenerateController manages the lifecycle of generated resources

Type: feature request

Describe what feature you want

Additional context

Add any other context or screenshots about the feature request here.

[starnop]

So far we've shown that it works by using Kyverno to support image policy, and there are two more issues that need to be addressed

1. Verify whether the Kyverno supports k0s/k3s. @starComingup may come to take a look.
2. Submit the proposal on how to implement image policy for sealer using Kyverno

[YTGhost]

@starComingup Regarding the supportability of kyverno for k0s/k3s, I verified that it works well. cc @starnop

[justadogistaken]

@YTGhost Can I look Kyverno in this way? That is Kyverno can help us make it easy to manage k8s webhook?

[YTGhost]

@YTGhost Can I look Kyverno in this way? That is Kyverno can help us make it easy to manage k8s webhook?

@justadogistaken Yes, kyverno makes it easy to create webhooks dynamically by creating policies.
The following is an example of a Policy that replaces image when creating or updating a Pod:

apiVersion : kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: redirect-registry
spec:
  background: false
  rules:
  - name: prepend-registry-containers
    match:
      resources:
        kinds:
        - Pod
    preconditions:
      all:
      - key: "{{request.operation}}"
        operator: In
        value:
        - CREATE
        - UPDATE
    mutate:
      foreach:
      - list: "request.object.spec.containers"
        patchStrategicMerge:
          spec:
            containers:
            - name: "{{ element.name }}"
              image: "sea.hub:5000/{{ images.containers.{{element.name}}.path}}:{{images.containers.{{element.name}}.tag}}"