Provisioning each Control Plane Machine with Unique Credentials.

[zawachte]

User Story

As an operator, I would like to be able to provision each control plane machine with unique credentials. My use case is for KMS Plugins. KMS plugins need to run as static pods or services and can't rely on the api-server (configmaps, secrets). (Can't encrypt secrets if you rely on them!)

Today the best I can do is use KCP and add the credentials as a file, but the problem is that this file is for the whole KCP replica set and not an individual machine. Ideally, each KMS Plugin instance has its own "identity". In theory, I guess could try to do some sort of appending to the Files list for each time I scale up, but that sounds pretty messy and would result in controlplanes having other controlplanes creds.

I am not really sure how we would go about editing the KCP CRD to support this. Is there a pattern used today by other kubernetes "replicasets" to achieve this?

Another solution I thought of was letting Infrastructure Providers have the ability to edit the bootstrap kubeadmconfig before it's encoded as cloud-init. Then the infrastructure providers can add a way to add files or other configs? I don't really know if this route makes much sense, but it would be nice if infrastructure providers had some say in the bootstrap data. A way for infrastructure providers to always set some needed configs without relying on higher-level input.

/kind feature

[detiber]

Thinking somewhat generically about this with a bit of an AWS tinted set of glasses, I would expect to be able to accomplish something along these lines using a combination of IAM profiles that allow access to the needed services for KMS, and the individual identities of the control plane instances.

Of course that isn't overly applicable to all the places that Cluster API can run or for supporting a KMS provider that isn't tied closely to the cloud provider (such as Hashicorp Vault), so I do think it makes sense to try to enable this type of workflow in a more generic way.

[vincepri]

/milestone Next
/priority awaiting-more-evidence

[randomvariable]

Possibly a use case within #3761 . Similar to how we want to treat domain joins potentially.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fabriziopandini]

/remove-lifecycle stale

[vincepri]

/lifecycle frozen

[vincepri]

Actually, should we close this?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[vincepri]

@randomvariable @yastij Can this be part of machine attestation?

[randomvariable]

it's completely orthogonal. machine attestation isn't part of control plane provisioning (because you're providing the machine with the cluster key material). we'll need to treat it as part of whatever happens if we revisit #4221

[vincepri]

/lifecycle frozen

[vincepri]

/assign @randomvariable
to retriage once security review has been done

[randomvariable]

/area control-plane

[randomvariable]

Reading through this again, I wonder if this is a use case for #5175

[randomvariable]

I think i'll take my name off this for now, as I'll be reviewing the area labels on a frequent basis.

/unassign

[fabriziopandini]

/triage accepted
/help

[k8s-ci-robot]

@fabriziopandini:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/triage accepted
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[fabriziopandini]

(doing some cleanup on old issues without updates)
/close
Unfortunately no real progress toward a solution yet, so I'm closing

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

(doing some cleanup on old issues without updates)
/close
Unfortunately no real progress toward a solution yet, so I'm closing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.