CNTRLPLANE-2120: Add KMS foundations in encryption controllers in library-go #1900

ardaguclu · 2025-12-03T09:34:51Z

This PR is based on #1872 (changes in enhancements/kube-apiserver/kms-encryption-foundations.md).

There are many aspects that need to be implemented to support KMS in OpenShift. We have decided to open more granular EPs to better track the work.

This EPs main aim is to focus on the encryption controller changes in library-go. This EP defers some concepts to future in order to start with simpler, manageable iterations.

PoC PR openshift/library-go#2045 (this is just a PoC, original PR will be opened when this EP merges).

openshift-ci-robot · 2025-12-03T09:34:56Z

@ardaguclu: This pull request references CNTRLPLANE-2120 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

This PR is based on #1872 (changes in enhancements/kube-apiserver/kms-encryption-foundations.md).

There are many aspects that need to be implemented to support KMS in OpenShift. We have decided to open more granular EPs to better track the work.

This EPs main aim is to focus on the encryption controller changes in library-go. This EP defers some concepts to future in order to start with simpler, manageable iterations.

PoC PR openshift/library-go#2045 (this is just a PoC, original PR will be opened when this EP merges).

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-12-03T09:35:07Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign jaypoulz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

flavianmissi

Have to take a short break from reviewing, but leaving the comments I got so far.

enhancements/kube-apiserver/kms-encryption-foundations.md

ardaguclu · 2025-12-03T14:08:45Z

/cc @ibihim @flavianmissi

ardaguclu · 2025-12-04T07:24:41Z

@flavianmissi I was uncomfortable about the disconnects between the sections and the verbosity. So I overhauled the EP to have better clarity. Please let me know your thoughts.

enhancements/kube-apiserver/kms-encryption-foundations.md

openshift-ci · 2025-12-05T10:53:12Z

@ardaguclu: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

flavianmissi · 2025-12-05T12:26:56Z

/lgtm

ardaguclu · 2025-12-05T12:59:50Z

/cc @benluddy

ardaguclu · 2025-12-08T10:40:44Z

As we agreed with @flavianmissi, in next iterations there will be another condition to notify users to delete unused kms plugins from cluster, when prune_controller prunes them.

ibihim

Great work. I have some questions though as a beginner to downstream e2ee with kms

ibihim · 2025-12-11T12:09:20Z

enhancements/kube-apiserver/kms-encryption-foundations.md

+
+#### Encryption Controllers
+
+**keyController** manages encryption key lifecycle. Creates encryption key secrets in `openshift-config-managed` namespace. For KMS mode, creates empty secrets with KMS configuration hashes.


Based on what I read later, it:

hashes the APIServer resource and stores the hash as an annotation in the secret

stores the key_id as data.

This confused me and made me jump forth and back as I see only that it creates Secrest with KMS configuration hashes. Which configuration? The APIServer? The kube-apiserver encryption config?

Encryption controllers convert apiserver.config.openshift.io to Secret. This Secret is used to bidirectionally convert to KeyState <-> Secret <-> kube-apiserver encryption config. So for instance we should carry some data (kmsconfig hash or key_id) in somewhere to always to be converable to the other. Otherwise, we can not detect apiserver config is changed or not.

We'll need key_id too to detect key_id is rotated or not. But this is deferred to next releases.

ibihim · 2025-12-11T12:13:37Z

enhancements/kube-apiserver/kms-encryption-foundations.md

+
+**keyController** manages encryption key lifecycle. Creates encryption key secrets in `openshift-config-managed` namespace. For KMS mode, creates empty secrets with KMS configuration hashes.
+
+**stateController** generates EncryptionConfiguration for API server consumption. Implements distributed state machine ensuring all API servers converge to same revision. For KMS mode, generates configuration with deterministic Unix socket paths.


What is a deterministic Unix Socket path? Are there probabilistic paths?

Why do we want this? To identify if the KMS is out of date and the address changed? To run several KMS plugins somehow in parallel?

To run several KMS plugins somehow in parallel?

This is one of the reasons. Deterministic implies that plugin lifecycle runs kms plugin with a unix socket that can be generated by the encryption controllers to communicate with (plugin lifecycle and encryption controllers are not organically related). So that this works as a contract that provides a deterministic communication.

However, since we likely decide to only support external kms plugins. We won't need this functionality. I proposed this openshift/api#2622 API definition to directly use whatever user sets. Thus, this unix socket generation logic will be removed from the EP.

ibihim · 2025-12-11T12:15:01Z

enhancements/kube-apiserver/kms-encryption-foundations.md

+4. Reuse existing migration controller (no changes needed)
+
+**Tech Preview v2 additions:**
+- Poll KMS plugin Status endpoint for `key_id` changes in apiserver operators


How do we call the KMS plugin Status endpoint?

It gets exposed by the kube-apiserver endpoint, right?

That is a good question. We left it TBD for now. Accessing kms plugins require privileged container. API servers (kube-apiserver, oauth-apiserver, openshift-apiserver) are privileged. So they can access. However operators (cluster-kube-apiserver-operator, openshift-apiserver-operator, cluster-authentication-operator) are not privileged. We'll have to find a way for operators.

ibihim · 2025-12-11T12:17:23Z

enhancements/kube-apiserver/kms-encryption-foundations.md

+     namespace: openshift-config-managed
+     annotations:
+       encryption.apiserver.operator.openshift.io/mode: "kms"
+       encryption.apiserver.operator.openshift.io/kms-config-hash: "a1b2c3d4e5f67890"


We hash APIServer.spec.encryption, right? It wouldn't make sense to hash the whole spec as it contains non-encryption configuration.

It may or may not based on what user expects. If configuration needs migration, we'll have to hash it. However, since I proposed to simplify this with openshift/api#2622, probably we don't need this anymore.

ibihim · 2025-12-11T12:19:06Z

enhancements/kube-apiserver/kms-encryption-foundations.md

+             endpoint: unix:///var/run/kmsplugin/kms-a1b2c3d4e5f67890.socket
+             apiVersion: v2
+   ```
+   The deterministic socket path allows KMS plugin lifecycle management to use the same path.


What is KMS plugin lifecycle management?

This will be handled by another EP (see: #1872). Plugin lifecycle implies to run kms plugins somehow to be communicated by apiservers.

ibihim · 2025-12-11T12:28:07Z

enhancements/kube-apiserver/kms-encryption-foundations.md

+- `kmsConfigHash` (annotation) - Detects admin configuration changes
+- `kmsKeyIDHash` (data field) - Detects external key rotation
+
+Separate hashes handle scenarios where config changes without key rotation (updating Vault address) or key rotates without config changes (AWS automatic rotation).


This should be at the top of the over-arching chapter, such that it is better to understand what the sub-chapters cover. Maybe even in the EncryptionController section.

I'll update accordingly.

ibihim · 2025-12-11T12:30:17Z

enhancements/kube-apiserver/kms-encryption-foundations.md

+**From aescbc to KMS:**
+1. Admin updates APIServer: `type: kms` with KMS configuration.
+2. keyController creates KMS secret (empty data, with hash).
+3. migrationController re-encrypts resources using external KMS.


migrationController uses KAS, not KMS directly right? We don't plugin the KAS-O via UDS to the KMS plugin, do we?

We don't plugin the KAS-O via UDS to the KMS plugin

That is correct.

KAS-O initiates migration via Storage migration resource. So it does not involve directly. Instead it creates corresponding storage migration resource and the migration will be completed asynchronously

ibihim · 2025-12-11T12:34:40Z

enhancements/kube-apiserver/kms-encryption-foundations.md

+```go
+EncryptionSecretKMSConfigHash = "encryption.apiserver.operator.openshift.io/kms-config-hash"
+```
+Stores truncated hash (16 hex characters, 8 bytes) of KMS configuration for change detection.


We use the default hashing mechanism that we use for other resources as well, right?

This is fully up to us. It doesn't have to follow the other hashing methods. The entire goal is to compare the hash values to detect any changes and that is all.

ibihim · 2025-12-11T12:35:37Z

enhancements/kube-apiserver/kms-encryption-foundations.md

+```
+Stores truncated hash (16 hex characters, 8 bytes) of KMS configuration for change detection.
+
+> **Note:** The hash is truncated to 16 hex characters (8 bytes) to stay within Unix socket path length limits (typically 108 characters) while maintaining sufficient uniqueness for distinguishing different KMS configurations. This allows deterministic socket paths like `/var/run/kmsplugin/kms-a1b2c3d4e5f67890.socket`.


This comes really late in the doc. So we use the config hash in the unix socket path.

What is the benefit of it?
We can have several KMS plugins open at once?

This mechanism seems like obsolete anyway.

ibihim · 2025-12-11T14:24:18Z

enhancements/kube-apiserver/kms-encryption-foundations.md

+**Reverse Conversion** (stateController reads EncryptionConfiguration from API server pods):
+1. Extract hash from socket path: `kms-a1b2c3d4e5f67890.socket` → `a1b2c3d4e5f67890`
+2. Look up secret with matching `kms-config-hash` annotation
+3. Reconstruct KeyState with original KMS configuration


What does that mean?

We read the encryption configuration for APIServer.Encryption.

We hash it.

We get the appropriate secret / configmap that contains:

key_id (can be compared with current key_id)

config hash (known, used for comparison)

Encryption controllers rely on the bidirectional conversion between Secret <-> KeyState <-> kube-apiserver encryption configuration. So we need to find a way to carry the data between them.

However, with this openshift/api#2622, probably we won't need to carry kmsconfig hash in unix socket. Unix socket path itself will be discriminator.

ardaguclu · 2025-12-12T11:16:47Z

I'll update this EP base on the changes in openshift/api#2622
/hold

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Dec 3, 2025

openshift-ci bot requested review from hasbro17 and yuqi-zhang December 3, 2025 09:35

ardaguclu force-pushed the kms-encryption-controllers branch from c091cbc to f734b05 Compare December 3, 2025 09:43

flavianmissi reviewed Dec 3, 2025

View reviewed changes

enhancements/kube-apiserver/kms-encryption-foundations.md Outdated Show resolved Hide resolved

flavianmissi reviewed Dec 3, 2025

View reviewed changes

enhancements/kube-apiserver/kms-encryption-foundations.md Outdated Show resolved Hide resolved

openshift-ci bot requested review from flavianmissi and ibihim December 3, 2025 14:08

ardaguclu force-pushed the kms-encryption-controllers branch 2 times, most recently from 1794054 to 1ddc3d8 Compare December 4, 2025 07:19

ardaguclu force-pushed the kms-encryption-controllers branch 3 times, most recently from e920a9c to 5804b76 Compare December 4, 2025 08:35

flavianmissi reviewed Dec 4, 2025

View reviewed changes

enhancements/kube-apiserver/kms-encryption-foundations.md Outdated Show resolved Hide resolved

flavianmissi reviewed Dec 4, 2025

View reviewed changes

enhancements/kube-apiserver/kms-encryption-foundations.md Outdated Show resolved Hide resolved

flavianmissi reviewed Dec 4, 2025

View reviewed changes

enhancements/kube-apiserver/kms-encryption-foundations.md Outdated Show resolved Hide resolved

flavianmissi reviewed Dec 4, 2025

View reviewed changes

enhancements/kube-apiserver/kms-encryption-foundations.md Outdated Show resolved Hide resolved

Add KMS foundations in encryption controllers in library-go

8f79ed6

ardaguclu force-pushed the kms-encryption-controllers branch from f39a0d7 to 8f79ed6 Compare December 5, 2025 04:21

flavianmissi reviewed Dec 5, 2025

View reviewed changes

enhancements/kube-apiserver/kms-encryption-foundations.md Outdated Show resolved Hide resolved

flavianmissi reviewed Dec 5, 2025

View reviewed changes

enhancements/kube-apiserver/kms-encryption-foundations.md Show resolved Hide resolved

Remove hash calculation snippet to remove AWS dependency

c11e49f

openshift-ci bot assigned flavianmissi Dec 5, 2025

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 5, 2025

openshift-ci bot requested a review from benluddy December 5, 2025 13:00

ibihim reviewed Dec 11, 2025

View reviewed changes

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 12, 2025


		#### Encryption Controllers

		keyController manages encryption key lifecycle. Creates encryption key secrets in `openshift-config-managed` namespace. For KMS mode, creates empty secrets with KMS configuration hashes.


		keyController manages encryption key lifecycle. Creates encryption key secrets in `openshift-config-managed` namespace. For KMS mode, creates empty secrets with KMS configuration hashes.

		stateController generates EncryptionConfiguration for API server consumption. Implements distributed state machine ensuring all API servers converge to same revision. For KMS mode, generates configuration with deterministic Unix socket paths.

CNTRLPLANE-2120: Add KMS foundations in encryption controllers in library-go #1900

Are you sure you want to change the base?

CNTRLPLANE-2120: Add KMS foundations in encryption controllers in library-go #1900

Uh oh!

Conversation

ardaguclu commented Dec 3, 2025

Uh oh!

openshift-ci-robot commented Dec 3, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Dec 3, 2025

Uh oh!

flavianmissi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ardaguclu commented Dec 3, 2025

Uh oh!

ardaguclu commented Dec 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci bot commented Dec 5, 2025

Uh oh!

flavianmissi commented Dec 5, 2025

Uh oh!

ardaguclu commented Dec 5, 2025

Uh oh!

ardaguclu commented Dec 8, 2025

Uh oh!

ibihim left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

openshift-ci-robot commented Dec 3, 2025 •

edited by openshift-ci bot

Loading