Kustomize plugin

[guzmo]

Hi,

We are using sealed-secrets together with kustomize and it's a bit of pain to handle it for multiple environments right now, since you need a new name of the sealed-secret to trigger the rolling-upgrade of services. I saw that kustomize now supports plugin, would be great if some kind soul could create a plugin for sealed-secrets :) Prbly a lot of people would appreciate it.

https://github.com/kubernetes-sigs/kustomize/blob/master/docs/plugins.md

/ Cheers

[NicolasRouquette]

Yeah, I ran into the same problem.

In my case, I have a kustomization.yaml file with both secretGenerator and configMapGenerator.
In that case, I can't pipe the output of kubectl apply -k to kubeseal because kubeseal doesn't handle configMap in the output,

For example, given the following kustomize.yaml:

namespace: foo
secretGenerator:
  - name: bar
    files:
      - bar.txt
configMapGenerator:
  - name: baz
    literals:
      - key: foo

when doing: kubectl apply -dry-run -k . | kubeseal --controller-name sealed-secrets-v0-7-0 --controller-namespace foo, I get:

panic: converting (v1.ConfigMap) to (v1.Secret): StringData not present in src

goroutine 1 [running]:
main.main()
	/home/travis/gopath/src/github.com/bitnami-labs/sealed-secrets/cmd/kubeseal/main.go:233 +0x2a8

If I remove the configMapGenerator, then it's fine but in that case, it shifts the problem to having to run kustomize separately for what kubeseal can handle vs. what kubeseal can't handle.

So, with the new support for plugins (I had to build kustomize locally), I tried a plugin like this:

kustomize.yaml:

namespace: foo
secretGenerator:
  - name: bar
    files:
      - bar.txt
configMapGenerator:
  - name: baz
    literals:
      - key: foo
transformers:
  - kubeseal.yaml

where kubeseal.yaml is:

apiVersion: example.org/v1
kind: kubeseal
metadata:
  name: kubeseal
  options: --controller-name sealed-secrets-v0-7-0 --controller-namespace foo

and ~/.config/kustomize/example.org/v1/kubeseal/kubeseal is the following executable bash script:

#!/bin/bash

temp_dir=$(mktemp -d)

#
# For debugging, uncomment this.
#
#trap "/bin/rm -rf $temp_dir" 0 2 3 1

echo "=======" > ~/plugin.txt
echo "dir=$temp_dir" >> ~/plugin.txt
echo "=======" >> ~/plugin.txt
cat $1 >> ~/plugin.txt
echo "=======" >> ~/plugin.txt

OPTIONS=$(cat $1 | yq -r '.metadata.options')

echo "options: $OPTIONS" >> ~/plugin.txt
echo "=======" >> ~/plugin.txt

cat - > $temp_dir/input.yaml

# Filter secrets
cat $temp_dir/input.yaml | yq -y '.|select(.kind=="Secret")' > $temp_dir/secrets.yaml

# kubeseal strips the kustomize metadata annotation so we have to save it to inject it later.
ANNOTATION=$(cat $temp_dir/secrets.yaml | yq '.metadata.annotations."kustomize.config.k8s.io/id"')
INJECT=".metadata.annotations.\"kustomize.config.k8s.io/id\"=$ANNOTATION"

echo "annotation: $ANNOTATION" >> ~/plugin.txt
echo "=======" >> ~/plugin.txt
echo "inject: $INJECT" >> ~/plugin.txt
echo "=======" >> ~/plugin.txt

# Filter other contents
cat $temp_dir/input.yaml | yq -y '.|select(.kind!="Secret")' > $temp_dir/other.yaml

# Apply kubeseal to secrets
cat $temp_dir/secrets.yaml \
    | kubeseal $OPTIONS --format yaml \
    > $temp_dir/sealedsecrets.yaml

# Inject the kustomize metadata annotation back (kubeseal stripped it!)
cat $temp_dir/sealedsecrets.yaml \
    | yq -y --sort-keys "$INJECT" \
    > $temp_dir/sealedsecrets-with-annotation.yaml

# put everything together again

cat $temp_dir/other.yaml > $temp_dir/output.yaml
echo "---" >> $temp_dir/output.yaml
cat $temp_dir/sealedsecrets-with-annotation.yaml >> $temp_dir/output.yaml

# This is the custom-transformed output...

cat $temp_dir/output.yaml

The idea here is to apply kubeseal only to secrets and keep the other contents as-is.

In principle, we should then be able to do:

kustomize build --enable_alpha_plugins . | kubectl apply -f -

But that doesn't work.

Just:

kustomize build --enable_alpha_plugins .

produces an error:

Error: type SealedSecret is not supported for hashing in map[apiVersion:bitnami.com/v1alpha1 kind:SealedSecret metadata:map[creationTimestamp:<nil> ....

And indeed, I see why: https://github.com/kubernetes-sigs/kustomize/blob/master/k8sdeps/kunstruct/hasher.go#L25

Darn!

I wonder if the kustomize plugin feature requires relaxing this hashing function to that kustomize could be applied to non-standard resources like SealedSecret.

[mkmik]

I'm a little bit confused about this use case.
I'd really like to better understand what problem are you trying to solve.

Here is how I think about the problem space (and I might be missing something obvious, so please let me know):

If I understand correctly, the secretGenerator kustomize rule literally reads a cleartext file from the FS and encodes it into a k8s Secret resource (like kubectl create secret generic would)

secretGenerator:
  - name: bar
    files:
      - bar.txt

The goal of sealed-secrets is to avoid checking in the secrets in your version control system.
The main goal of kustomize is to "customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is".

A secondary goal of kustomize is to provide some helpers for common tasks such as constructing config maps or secrets from bits that you have laying around in other files (since you cannot otherwise "import" a value from a file in yaml). That works on the assumption that it's ok to have that information in clear in the filesystem.

Perhaps improvements in the usability of the kubseal utility (in particular w.r.t updating items of existing resources) would solve your use case at the root? (see #95)

[willholley]

For me, the benefit of secretGenerator is that it adds a hash suffix to every secret and updates all references to those secrets. This, in turn, forces pods referencing the secret to reload when the secrets change.

In terms of how this relates to sealed secrets, it would be useful to have kustomize be able to take a sealed secret manifest as input and append the hash suffix etc as it does for standard secrets. I think this is where @NicolasRouquette got to but ran into the problem that kustomize won't hash CRs (kubernetes-sigs/kustomize#1167).

The sealed secret controller would also need to be able to decrypt the renamed sealed secret when it's deployed (hand-waving issues with garbage collection).

[mkmik]

Thanks! This is actually similar to an issue reported with spinnaker: #62 (comment)

Perhaps we can find a way to address both issues

[ChrisScottThomas]

We've also just hit this, and are having to just pass sealed secrets as resources.

It'd be nice to be able to declare a kind: in the secretGenerator,

secretGenerator:
- kind: SealedSecret
- name: my-sealed-secret
   literals:
   - foo=YmFyCg==

and specify my kubseal cert in my overlay/$env/kustomization.yaml.

resources:
- ../../base/
sealedSecretCert:
- $env-cert.crt
patchesStrategicMerge:
- patch-secret.yaml

Which should give a plugin the data required to seal a new secret and pass it on kustomize build ... ?

[mkmik]

that would require teaching the core secretGenerator plugin about sealed secrets.

A possible horrible hack we could do is to literally create a Secret with fake items containing encrypted data

e.g.

- name: my-sealed-secret
   literals:
   - sealed_secret_sealed_foo=Ga12YmFyCg........

and have the controller unseal these in-place. Strategic merge patch (supported by kubectl apply and diff) would handle those extra fields. The main downside is that while scheduling pods referencing you'll see errors referring to missing items rather than unavailable secret (which is what you'd expect with sealed-secrets since the unsealing is asynchronous)

[ChrisScottThomas]

You would still need some method of sealing the secret with Kustomize, before you could pass any secret into the cluster. So you'd need a plugin (or Kustomize) to handle the sealing with certs, and a generator to keep the Kustomize pattern in tact?

[mkmik]

yeah I was glossing over that. We might just put the output of kubeseal into a file and files to load it:

<whatever_you_use_to_generate_your_cleartext_once> | kubeseal >sealedsecret_unseal_here.yaml

secretGenerator:
- name: mysecrets
  files:
  - sealedsecret_unseal_here.yaml

this will create a secret with a sealedsecret.yaml item, which the sealed secret controller would pick up, decrypt and merge the items in the current secret.

But, as I said, this looks like a hack. Probably a proper kustomize plugin would be better.

I'm wondering if we can write a generic kustomize plugin that deals with these "suffixed references" without having to be specific to sealed-secrets.

(EDIT: ideally we should be able to set some labels to the secret, to instruct the sealed secret controller to do this "merge" operation; I don't know how to add labels/annotations with kustomize's secretGenerator builtin and clumsily failed to find docs that would teach me that)

[mcristina422]

@mkmik

I don't know how to add labels/annotations with kustomize's secretGenerator builtin

I agree it's not obvious, but this is how you add labels/annotations to the builtin

apiVersion: builtin
kind: SecretGenerator
metadata:
  name: mySecret
  namespace: whatever
labels:
  foo: bar
annotations:
  alice: bob
literals:
- FRUIT=apple

would be rendered to

apiVersion: v1
data:
  FRUIT: YXBwbGU=
kind: Secret
metadata:
  annotations:
    alice: bob
  labels:
    foo: bar
  name: mySecret
  namespace: whatever
type: Opaque

[felipefzdz]

@mkmik what would be the label required to instruct the sealed secret controller to do this "merge" operation?

I'm trying to apply your suggested hack by doing this:

echo -n "key=value"  | kubectl create secret generic myOriginalSecret --dry-run --from-file=foo=/dev/stdin -o json | kubeseal > sealedsecret_unseal_here.yaml

cat sealedsecret_unseal_here.yaml

{
  "kind": "SealedSecret",
  "apiVersion": "bitnami.com/v1alpha1",
  "metadata": {
    "name": "myOriginalSecret",
    "namespace": "default",
    "creationTimestamp": null
  },
  "spec": {
    "template": {
      "metadata": {
        "name": "myOriginalSecret",
        "namespace": "default",
        "creationTimestamp": null
      }
    },
    "encryptedData": {
      "foo": "AgBJE7U4nJuM5G2wm4B9DvnnA5SY5Bh7SiBNqKu3fAXvt96CkOgovPa4zadgIa+TyZct00PD/8evl2Vc2huW5sVW32cEV3C3kKxNRop4Nmd2+HuiEdkRVWCeAKodfgN3Tgy/mKHVSnC9TvPqnTCnkTjOEknTprGrX+1ox1LmqLMX0mND4WCPEjxynXpdKAsQzOnL7wTLxg2YKiCGH5/KqX7F8/rmK3DdOq/FO9d/g9jEymziaKKRFT32z5os0w3qZtkMW8sqC78k1f6CqJmjONC2paHs1duVfro62X1l9rqSA3oP8h73JmEdpsyCYt+GgjVuCHo4VPRaqct+dxvircwQkgZ8PB6xsFzQLbKx3OauxwUcV2MAv7EDHLMsB3z+HSVIFn0SZOANw6wx+UTDisZdzJ8CoQsMLv1TcFPjUU6s6sh97DwSw/MCUC/wQGqesOh4c3xi01kJZf97uLH+RizSPelfGRo+OnDOlE92kqAI04N/4xRgaTraczgezzEpRLFfCCkrnLn360hFxe2NXC6xPwMAHj3jeQ2n4FotyexEJ809Ej7CMlsR0GVcyvsl+fsKKM9n9HcA/4xPkFH4IPFbZfncO0iXuIaiaNt4KP586MkQ4GV5p2Cxd7Tel1wRe2eEKMwx4XoHAwhTsLbGwdwQYO7t0h7SBnXzYwidWTd7+JXhHVcAEfXqdASmR8XubzmzW+4AE0GD5IO0F0FoDYE26ForNLm+96MDL6eqOWE/nDtY5XQUqCUXKpjxJ2uLPQ=="
    }
  },
  "status": {
    
  }
}

secretGenerator:
  - name: mysecrets
    files:
     - files/sealedsecret_unseal_here.yaml

Applying that kustomize manifest deploys the expected secret to the cluster:

mysecrets-t956kt4f7d                      Opaque                                1      46m

but nothing else. No logs on the sealed-controller showing awareness of the wrapped sealed secret. I guess that without that label doing the linking there's no way for the controller to react.

Sorry for asking, but has been progress on the non-hacky path for this use case?

[mkmik]

it was just an idea; there is nothing in the controller right now that would act that way.

[mtweten]

I started looking to use SealedSecrets with kustomize as well, and thought it would be relatively simple using the raw option to produce encrypted values and have a sealed secrets generator plugin such that you might do something like:

sealedSecretGenerator:
- name: mysecret
  namespace: myns
  literals:
    MYSECRET=<encrypted and base64 encoded output from kubeseal --raw>

I haven't actually gone through implementing it, but it seems like getting the automatic versioning and rollout will be blocked because of the hashing problem for custom types that @NicolasRouquette mentioned in #167 (comment). The builtin HashTransformer only supports ConfigMaps and Secrets, so it fails. Maybe the hashing could just be done in the sealedSecretGenerator plugin?

Past that, I think the name-reference transformer could be configured for sealed secrets to do appropriate name updates, but again I haven't actually tried it.

[RyosukeCla]

We developed sealed secret hash transformer to add hash to metadata.name in aims to update pods when sealed secret changed. Also we made configuration of name reference transformer for sealed secret. But we found that sealed secret which name is modified cannot be decrypted by default.

• sealed secret hash transformer: https://github.com/plaidev/kustomize-plugins
• chat log: https://kubernetes.slack.com/archives/CM0H415UG/p1574409839114400

[mkmik]

By using the namespace-wide scope when sealing the secret, the secret can be renamed at will as long as it stays in the same namespace

[mtweten]

This is great @RyosukeCla ! I did validate that the namespace-wide scope works as well.

[tehmoon]

Successfully wrote a kustomize transformers that will replace every Secret.v1 and pass it to kubeseal.
This requires a back and forth using jq and yq as there is no tool to rule them all :(
https://github.com/tehmoon/cheatsheet/blob/master/kubernetes/kustomize/plugins/sealed-secrets-encrypt.md
Let me know!

I also wrote https://github.com/tehmoon/cheatsheet/blob/master/kubernetes/kustomize/plugins/sealed-secrets-install.md to generate installing sealed-secret.

Ideal for gitops!!! Generation can be done offline since you can pass the --cert flag to kubeseal. Awesome project!!

[mkmik]

@tehmoon hi! Thanks. Can you please describe your wider workflow so I can better understand how this step with kustomize fits in?

[tehmoon]

@mkmik of course! I'm installing and operating a kubernetes cluster solely using GitOps with FluxCD.
Installing the operator is the only first manual step that has to be done. The rest is entirely from github.
The idea behind the workflow is that everything needs to be generated offline as developers/operators don't have access to the cluster directly, not even R/O.

If I take, for example, the way you install the mesh network Linkerd, you have to run linkerd install | kubectl apply -f -. In gitops, I simply store that in a yaml file in git (behind the scene it is done through kustomize). Since it requires certificates, if you don't generate your own, the installation will generate that for you. Again, since it's gitops, it means that the command will hardcode the certificates, which end up in git and it is no good.

This is where kubeseal comes into play. The naive approach is to:

• Generate certificates for linkerd
• Run linkerd install specifying the path to those certificates
• Manifests files are creates, certificates end up being in Secret.v1 in the file.
• Discard generated certificates
• Grab the Secret.v1 in the generated yaml and run kubeseal through them
• Replace the Secret.v1 in the file with their kubeseal equivalent

These are a lot of unnecessary steps which can all be avoided using a transformer plugin.
When using such a plugin. The whole generated file by kustomize is passed to the plugin. Each value you output from the plugin will be replaced directly. Ensuring that Secret.v1 are effectively not present in the file and only kubeseal are.
This streamlines the workflow as every dev/operator can use the plugin in their kustomize definition to automatically encrypt their secret, which they can safely remove from git when they are done using them locally.

It goes without saying that kubeseal --fetch-cert must be done at cluster initialization and that --cert <path> must also be communicated to everyone using the plugin so they know which certificate to encrypt the secret with.

[snuggie12]

Is there a reason why this wouldn't work for a plugin?

1. Make a SealedSecret as normal but in kubeseal you pass a flag like --hashed-secret-name.
2. Your SealedSecret has a .metadata.name of foo but in the template it uses the same hashing function as kustomize so your Secret has a .metadata.name like foo-abcdefg. AFAIK this requires a code change (see kubeseal outputs spec.template.metadata.name{,space} but is ignored #499 )
3. Then your kustomize transformer looks at all SealedSecrets and updates workloads to use the hashed name.

[nickjj]

With Helm you can trigger a pod roll out by changing a secret's manifest without needing to rename the secret by taking advantage of annotations and a bit of templating. You can do something like this in your deployment template:

  template:
    metadata:
      annotations:
        checksum/secret: {{ include ("shared-lib.secret") . | sha256sum }}

In the above case, that's including the contents of a secret template which is a standard Kubernetes secret. It then gets a checksum of the file which results in that annotation having a unique checksum. Re-rolling your secrets will produce a different checksum and that will trigger a rollout of your pods. You can think of it like cache busting your static files but for Kubernetes manifests.

Does anyone have any ideas on how to apply this sort of thing with Kustomize? Preferably in a way that works with GitOps (ArgoCD) where you're not having to run anything in CI beforehand, or ideally not having to manually run something locally to change the template before pushing your code up.

I think running it locally wouldn't be too hard to pull off in a shell script since all you would have to do is pop in a $checksum variable into your base deployment and then use sed to replace that with the checksum of your secret (or sealed secret if you use that instead). Getting a unique md5 hash could be done with the built-in md5sum command.

I am in the process of using sealed secrets (will be migrating to it soon) so I do think there will be a need to have some shell scripting before pushing code to help extract / edit / create a new sealed secret, so maybe the local approach is the way to go since if you're running a local kubeseal command to modify your sealed secrets then it's not really too much effort to also calculate and swap in the checksum annotation value? Would be great to have that as a built in, perhaps with some conventions to make using it substantially easier than rolling a custom solution.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[davinkevin]

As a user of kustomize + fluxcd, I would really like to have a solution to use hash-suffix with SealedSecret to have:

• Capacity to use immutable secret for audit purpose/analysis
• Keep all versions of secrets available for potential rollback
• Have automatic restart of deployment/statefulSets… in case of change in secret

SealedSecrets is a really good solution, but if it reduces possibilities offered by the standard system, it starts to became a pain and other solution can be investigated, like for example fluxCD + sops in our case.

[brunocascio]

Is there any workaround or way to do it?

[jashandeep-sohi]

Here's one way of doing this with newer versions of kustomize (tested on 5.2.1). At least for deploying from a valid SealedSecret.

Input

.
├── builtin-config.yaml
├── kustomization.yaml
├── pod.yaml
└── sealedsecret.yaml

1 directory, 4 files

==> builtin-config.yaml <==
nameReference:
  - kind: Secret
    fieldSpecs:
      - kind: SealedSecret
        path: metadata/name

      - kind: SealedSecret
        path: spec/template/metadata/name

==> kustomization.yaml <==
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ./pod.yaml
  - ./sealedsecret.yaml

secretGenerator:
  - name: test
    files:
      - ./sealedsecret.yaml
    options:
      annotations:
        config.kubernetes.io/local-config: "true"

configurations:
  - ./builtin-config.yaml

==> pod.yaml <==
# yaml-language-server: $schema=https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/master-standalone/pod-v1.json

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: something
      image: some-image
      envFrom:
        - secretRef:
            name: test

==> sealedsecret.yaml <==
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: test
  annotations:
    sealedsecrets.bitnami.com/namespace-wide: "true"
spec:
  encryptedData:
    a: "1...."
    b: "2..."
    c: "3..."
  template:
    metadata:
      name: test
    type: Opaque

Output

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  annotations:
    sealedsecrets.bitnami.com/namespace-wide: "true"
  name: test-cm5hd7d6b9
spec:
  encryptedData:
    a: 1....
    b: 2...
    c: 3...
  template:
    metadata:
      name: test-cm5hd7d6b9
    type: Opaque
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - envFrom:
    - secretRef:
        name: test-cm5hd7d6b9
    image: some-image
    name: something

The trick is to teach kustomize (builtin-config.yaml) to replace a SealedSecrets's name with that of a matching Secret.

This Secret is generated w/ a secretGenerator with input being the SealedSecret itself. So any changes to the SealedSecret causes the short-hash value to change.

Finally, to avoid emitting the Secret in the output, annotate it with config.kubernetes.io/local-config: "true".

Obviously, you also need a sealedsecrets.bitnami.com/namespace-wide: "true" on the SealedSecret because the name can change.

[jashandeep-sohi]

FYI, I got tired of doing the copy-pasta above again and again in every new Kustomization, so I've created a KRM function that can help me out: https://github.com/jashandeep-sohi/krm-fn-sealedsecrets

It too is kind of a hack, so your mileage may vary but the UX is much nicer for my use-case. Here's an example of doing something similar to a secretGenerator above, but in a more persistent way: https://github.com/jashandeep-sohi/krm-fn-sealedsecrets/tree/master/examples/kpt-seal-name-ref-kustomization

[JVMartin]

On latest version of kustomize, @jashandeep-sohi 's solution no longer works because as soon as I add the config.kubernetes.io/local-config: "true" to the generated secret, not only does it disappear from output but the nameReference also breaks, and no longer appends the hash to the sealed secret name.

Of course I can remove the local-config annotation, but then the secret actually gets applied to the cluster and so the sealed secret gets the hash, but cannot decrypt to overwrite the applied secret: failed update: Resource "test-6b49kf5h74" already exists and is not managed by SealedSecret.

To work around this, I just gave sealed secrets access to overwrite the applied stub secret instead:

  - name: test
    files:
      - ./sealedsecret.yaml
    options:
      annotations:
        sealedsecrets.bitnami.com/managed: "true"

This is not ideal, there is a potential race condition, but it works for now unless anyone has any better ideas.