[Question] Best practices for handling secrets in kustomize

[btrepp]

Hi All,

I really love the tool, and do understand the core philosophy of 'everything should come from the files'.
However theres a bit of a grey area when it comes to secrets.

Most kustomize code, is most likely checking into git.
Kustomize pushes you to include every thing into files.
Secrets are an important part of any app.
Thus the logical solution here is that kustomize is driving you to 'put your secrets in your git repository'.

Years of that being a bad idea have somewhat stuck with me. There appear to be a few hacks here, eg

1. setting env vars and having a strange feature in secretGenerator pick them up (but that might fall over with two keys in different objects needing different secrets).
2. writing your own templating solution to 'sit over' kustomize, which does feel like it's going against the spirit.
3. writing a plugin that pulls from another source.
4. punting and deploying secrets outside of kustomize, and setting there names static.

At the moment, the most sane seems to be 2, combining kustomize with another tool that can generate the secrets and using that to overlay the actual app, but that does get a bit complex with maybe multiple kustomization files being needed then.

The motivation for asking is to help solve argoproj/argo-cd#9083, but there are a few ways that could be done.

Is there any guidance for how secrets should be used with kustomize?. It's an amazing tool, and a solution that keeps the secrets using a hash, thus helping the deployments change is amazing.

[k8s-ci-robot]

@btrepp: This issue is currently awaiting triage.

SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the triage/accepted label.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[olegsu]

We use https://external-secrets.io/v0.5.3/, a CRD that creates "native" secrets from other types.

• The actual secret is stored in AWS.
• The secret generator is used to create empty secrets (or secrets with default empty/default values)
• The application may restart once on the first deploy

Hope that helped :)

[KnVerey]

Kustomize indeed has features that help to create secret resources from data, but does not have any opinions or functionality around helping you secure that data before (or after!) Kustomize interacts with it. Indeed Kubernetes secrets themselves are not ideal in terms of security (see all the caveats in those docs). There are many different approaches to either avoiding the use of k8s secrets in favour of a first-class secret store, or securing them as much as possible, and I'm not really in a position to advise on them. Sealed Secrets is an example of the latter that has documented Kustomize support. Kustomize's scope eliminates most options for being more helpful, since we cannot shell out to other tools or call external APIs for both security and design constraint reasons. SecretGenerator used to support external commands, but that was removed for security reasons. The issue for the removal spawned a lot of discussion that might be of interest to you: #692. The external plugin option (3) is probably the closest we could get to supporting more sophisticated integration with various secret handling solutions folks are using, and once the KRM Functions Registry is stood up, it could be a logical home for such plugins.

[btrepp]

Thanks @KnVerey and @olegsu

I had experimented with some of those tools, but found a few short comings with the kustomize plugins, and the k8s operators.

I ended up writing https://git.sr.ht/~btrepp/kubepassgen and a gitops plugin for argocd https://git.sr.ht/~btrepp/argo-pass-sidecar

I believe this tool keeps the kustomize spirit of 'no templates' as much alive as possible. I can use this to have secretGenerator generator empty secrets, and then patch them 'at the last second' by creating a new kustomize file.
This avoids any shelling out, and lets kustomize work exactly as intended.

#!/bin/sh
WORK=$(mktemp -d)
trap 'rm -rf -- "$WORK"' EXIT
BASE=$(realpath --relative-to=$WORK $PWD)

cat << EOF > $WORK/kustomization.yaml
resources:
- $BASE

patchesStrategicMerge:
- secret.yaml
EOF

kubepassgen > $WORK/secret.yaml
kustomize build $WORK

This is working in my home lab cluster, and is great because I don't have to rely on a cloud operator. Passwordstore supports git, and I have a private git server. So I just mainly glue existing tools together :).

I'm okay with closing this issue, though I think the root of 'best practice' is still probably not yet clear for kustomize. So there good be value in further discussion.

[CharlesWinter]

For anyone coming upon this thread, I've had a lot of success on past projects with something called sops, from mozilla. https://github.com/getsops/sops the principal is that you encrypt your secrets using keys that sops controls (the keys may be in something like KMS from GCP for example). and commit the encrypted file to the repo.

there are then plugins for kustomize + sops such as this https://github.com/viaduct-ai/kustomize-sops

that decrypt the secrets at deploy time from the file and build them into versioned k8s secrets. It's probably the best deployment strategy for this stuff I've come across.