Add encryption providers proposal #1213

moelsayed · 2021-01-12T08:08:29Z

What this PR does / why we need it:
A proposal draft to add Encryption Providers support in KubeOne.
Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1212

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

docs/proposals/20210112-encryption-roviders.md

kron4eg · 2021-01-15T12:35:11Z

/lgtm
/assign @xmudrii

kubermatic-bot · 2021-01-15T12:35:14Z

LGTM label has been added.

Git tree hash: b7ac88af2bee1fd2c1a40cdd0c3e142f4c387e6a

xmudrii

/approve

kubermatic-bot · 2021-01-20T09:47:08Z

LGTM label has been added.

Git tree hash: b7ac88af2bee1fd2c1a40cdd0c3e142f4c387e6a

kron4eg · 2021-01-20T09:49:44Z

/hold

kron4eg · 2021-01-20T11:06:50Z

docs/proposals/20210112-encryption-roviders.md

+kind: KubeOneCluster
+features:
+  encryptionProviders:
+    enabled: true


Keeping in mind future API extensions that will be inevitably required, simple boolean flag is not enough. Flags are needed per provider type, and not only flags but possible config options too.

So we are looking at 3 possible providers:

none

static (the one for being implemented now)

kms (the one for being implemented later)

Those should be expressed as additional objects inside encryptionProviders, like

#### VERY DRAFT, MUCH FANTASY #### ####### only for demo purpose ####### apiVersion: kubeone.io/v1beta1 kind: KubeOneCluster features: encryptionProviders: - kind: none - kind: static - kind: kms endpoint: unix:///tmp/socketfile.sock cachesize: 100 timeout: 3s

Alternatively, we could not reimplement parts of what's already done in upstream and "just" provide the

apiVersion: kubeone.io/v1beta1 kind: KubeOneCluster features: encryptionConfiguration: | apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration resources: - resources: - secrets providers: - kms: name: myKmsPlugin endpoint: unix:///tmp/socketfile.sock cachesize: 100 timeout: 3s - identity: {}

Given that second option, we can actually extract the endpoint from KMS providers and automatically mount them to the kube-apiserver pod ( with extra mounts in kubeadm config).

Provided that we can detect changes in encryptionConfiguration (compare actual file to encryptionConfiguration`, we can automate and orchestrate kubeapi pod restarts.

Providing ONLY encryptionConfiguration we could give all encryption options to the end user, including KMS support in one shot.

I agree with what @kron4eg said, but in that case, I'd leave the enable field in the API and implement it like @moelsayed initially proposed. For example:

features: encryptionProviders: # enable/disable enable: true # setting this value will disable automated rotation. customProviderFile: | apiVersion: apiserver.config.k8s.io/v1 kind: EncryptionConfiguration ...

The difference between the originally proposed API and this API is that I've removed the rotate field. Instead of the rotate field, we would keep the CLI flag as it's currently proposed.

If enable is set to false

Do nothing if the feature has never been enabled

Otherwise, disable the feature

If enable is set to true:

If customProviderFile is not set, generate it ourselves and handle key management

If customProviderFile is set, upload it to the instance. Additionally, if KMS is configured, create the appropriate volumes on the API server pods

Configure the API server and everything else as described in the proposal

Note: if customProviderFile is set and the CLI flag is set, KubeOne would either show a warning or fail.

In this case, we can quickly implement support for all setups. Technically, users could use Terraform/Ansible to deploy KMS and with such API, they could actually use it in their clusters. I think that it wouldn't require too much additional effort on our side, but we could solve everything (besides deploying KMS, which should be discussed) in a single run.

@kron4eg @xmudrii I updated the proposal accordingly. Please check.

For the initial implementation, I will stop at implementing the the custom config basic functionality. I will create a new issue/PR to address deploying the KMS and creating the required volume mounts for the API server. I am considering that out of scope for the initial implementation.

@moelsayed Can you create an Epic for this?

@xmudrii there is already an epic to track the proposal and the implementation: #769

@moelsayed I was thinking about an Epic for addressing the KMS deployment and the volume creation. You can also extend the current Epic. I only see issues relevant to the initial implementation in the Epic, but please correct me if I'm wrong.

What happens if:

enable: true, but no customProviderFile is given (e.g. it's empty)?

enable: true, but no customProviderFile is given and install/apply is run, and then customProviderFile becomes defined?

enable: true, but no customProviderFile is given (e.g. it's empty)?

We manage everything automatically. A new config file is created and pushed to the nodes. In this case key rotation is supported.

enable: true, but no customProviderFile is given and install/apply is run, and then customProviderFile becomes defined?

K1 should push the defined custom file but it will refuse to the rotate the keys if the rotate flag is set.

@xmudrii I updated the epic with an issue to add KMS #1242.

xmudrii

/lgtm
/approve
/hold
if @kron4eg wants to take a look as well

xmudrii · 2021-02-05T11:52:40Z

docs/proposals/20210112-encryption-roviders.md

+kind: KubeOneCluster
+features:
+  encryptionProviders:
+    enabled: true


@moelsayed Can you create an Epic for this?

kubermatic-bot · 2021-02-05T11:55:03Z

LGTM label has been added.

Git tree hash: 7a63f9e46ff8cf0819386706364abef0ed49d909

kron4eg

/lgtm
/approve

kron4eg · 2021-02-11T17:01:18Z

/hold cancel

kubermatic-bot · 2021-02-11T17:01:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kron4eg, moelsayed, xmudrii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kron4eg,xmudrii]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* upstream/master: (21 commits) Improve the installation script (kubermatic#1253) Update README.md (kubermatic#1250) Add the changelog for the v1.2.0-beta.1 release (kubermatic#1249) Fix credentials in addons (kubermatic#1248) fix(docs): fix broken master documentation link (kubermatic#1246) Add encryption providers proposal (kubermatic#1213) Use Docker for restarting API server on Flatcar (kubermatic#1245) Add the Kubernetes 1.20 jobs (kubermatic#1244) Restart unhealthy API servers when provisioning/upgrading clusters (kubermatic#1243) Add rsync on CentOS and Amazon Linux (kubermatic#1240) Update machine-controller to v1.25.0 (kubermatic#1238) Update the kubeone-e2e image (kubermatic#1239) Update KubeOne CI jobs (kubermatic#1237) Disallow and deprecate the PodPresets feature (kubermatic#1236) Fix confusing default in OpenIDConnect (kubermatic#1235) Add debian support (kubermatic#1233) Drop mounting flexvolume plugins into the openstack CCM (kubermatic#1234) Remove CoreOS (kubermatic#1232) Add the changelog for the v1.2.0-beta.0 release (kubermatic#1230) Add containerRuntime API to the full config (kubermatic#1229) ...

Add encryption providers proposal

f7d7a7c

kubermatic-bot added release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: yes Denotes that all commits in the pull request have the valid DCO signoff message. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 12, 2021

moelsayed requested review from kron4eg and xmudrii January 12, 2021 09:25

kron4eg self-assigned this Jan 12, 2021

kron4eg reviewed Jan 13, 2021

View reviewed changes

docs/proposals/20210112-encryption-roviders.md Show resolved Hide resolved

kron4eg reviewed Jan 13, 2021

View reviewed changes

docs/proposals/20210112-encryption-roviders.md Outdated Show resolved Hide resolved

moelsayed added 2 commits January 15, 2021 11:47

remove custom configuration

e46c5ed

add apply flag

c82429d

kubermatic-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 15, 2021

kubermatic-bot assigned xmudrii Jan 15, 2021

kubermatic-bot added the lgtm Indicates that a PR is ready to be merged. label Jan 15, 2021

kron4eg removed the lgtm Indicates that a PR is ready to be merged. label Jan 20, 2021

xmudrii approved these changes Jan 20, 2021

View reviewed changes

kubermatic-bot added the lgtm Indicates that a PR is ready to be merged. label Jan 20, 2021

kubermatic-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 20, 2021

kubermatic-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 20, 2021

kron4eg reviewed Jan 20, 2021

View reviewed changes

kron4eg removed approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. labels Jan 20, 2021

updated custom config

542fa59

kubermatic-bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 4, 2021

kubermatic-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 4, 2021

xmudrii approved these changes Feb 5, 2021

View reviewed changes

kubermatic-bot added the lgtm Indicates that a PR is ready to be merged. label Feb 5, 2021

kron4eg approved these changes Feb 11, 2021

View reviewed changes

kubermatic-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Feb 11, 2021

kubermatic-bot merged commit b9fa601 into kubermatic:master Feb 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add encryption providers proposal #1213

Add encryption providers proposal #1213

moelsayed commented Jan 12, 2021 •

edited

Loading

kron4eg commented Jan 15, 2021

kubermatic-bot commented Jan 15, 2021

xmudrii left a comment

kubermatic-bot commented Jan 20, 2021

kron4eg commented Jan 20, 2021

kron4eg Jan 20, 2021

kron4eg Jan 20, 2021

xmudrii Jan 20, 2021

moelsayed Feb 4, 2021

xmudrii Feb 5, 2021

moelsayed Feb 5, 2021

xmudrii Feb 5, 2021

kron4eg Feb 5, 2021

moelsayed Feb 8, 2021 •

edited

Loading

moelsayed Feb 8, 2021

xmudrii left a comment

xmudrii Feb 5, 2021

kubermatic-bot commented Feb 5, 2021

kron4eg left a comment

kron4eg commented Feb 11, 2021

kubermatic-bot commented Feb 11, 2021

Add encryption providers proposal #1213

Add encryption providers proposal #1213

Conversation

moelsayed commented Jan 12, 2021 • edited Loading

kron4eg commented Jan 15, 2021

kubermatic-bot commented Jan 15, 2021

xmudrii left a comment

Choose a reason for hiding this comment

kubermatic-bot commented Jan 20, 2021

kron4eg commented Jan 20, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

moelsayed Feb 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xmudrii left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kubermatic-bot commented Feb 5, 2021

kron4eg left a comment

Choose a reason for hiding this comment

kron4eg commented Feb 11, 2021

kubermatic-bot commented Feb 11, 2021

moelsayed commented Jan 12, 2021 •

edited

Loading

moelsayed Feb 8, 2021 •

edited

Loading