How to generate ConfigMap clone in each namespace

[domruf]

I have a 'ca-certificates.crt' file which is stored as a ConfigMap and is required in every namespace in my cluster.
I therefore want to generate a clone of my 'ca-certificates.crt' ConfigMap in each new namespace when the namespace is created or a new version of my helm chart is installed.

I worked with kyverno before and there it is possible to generate new resources https://kyverno.io/docs/writing-policies/generate/.
Is there an equivalate in OPA? So far I found nothing about this on https://open-policy-agent.github.io/gatekeeper/website/docs/

[anderseknert]

Hi @domruf 👋

I don't believe that's a feature in Gatekeeper. If you really wanted to do something like that in "plain" OPA, you could use http.send to query the Kubernetes API directly and create resources that way, but honestly, I'm not sure that's a great idea.

The idea behind OPA, and by extension Gatekeeper, is first and foremost to allow establishing policy-based guardrails around infrastructure, access control, kubernetes, and so on. Policy defines the rules for an environment, and violations are immediately reported back to the user as expected. Introducing "resource creation" (and to some extent, even mutation) in this context is basically saying that policy should not just define the rules, but also potentially disastrous side effects... which is ironically pretty much what a policy engine should protect against in the first place. To add to that, these side-effects are not reported back to the user, and unless clearly communicated out-of-band, they'll have no way of knowing that what they actually asked for is what eventually happened.

Potentially useful? Maybe.
The job for a policy engine? 🤷

[srenatus]

I haven't used it myself, but this looks like it could deal with specific automatic maintenance tasks like this: https://github.com/flant/shell-operator 💭

This looks promising, perhaps you can adapt it:

configVersion: v1
kubernetes:
- name: execute_on_changes_of_namespace_labels
  kind: Namespace
  executeHookOnEvent: ["Modified"]
  jqFilter: ".metadata.labels"

[domruf]

Thanks @anderseknert,

I understand the argument that controlling policies and manipulate/creating resources to automatically fixing violations is not the same thing.
But since Gatekeeper already supports the manipulation part, I think it should support the creation part as well.
There are use cases like my 'ca-certificates.crt' example, where a manipulation depends on the creation of a resource.
Sure it can cause "potentially disastrous side effects" if implemented badly. But this is true for any kind of automation and "self healing".
I would argue it is true for the whole operator pattern kubernetes is based on.

I think my use case is quite common and the creation of resources would help a lot of people.
Many companies have self generated TLS certificates for their internal networks.
In these networks to access internal servers via https from a container, the container must use the companies ca-certificates.crt file.
When using/installing third-party deployments, providing the file can be hard and quite cumbersome.
AFAIS I can use manipulation to mount the file in each pod, but the volume mount depends on the ConfigMap to be available in the same namespace. I therefore must clone the ConfigMap to each namespace.

I'll look into the http.send feature.

[anderseknert]

Sure it can cause "potentially disastrous side effects" if implemented badly. But this is true for any kind of automation and "self healing".
I would argue it is true for the whole operator pattern kubernetes is based on.

For sure. I'm just not convinced OPA or Gatekeeper should be considered as "automation" tools, or as general purpose operators. Again, I'm not saying it couldn't be valuable, and it sounds to me like you have a valid use case in mind. I'm just not sure it falls under the responsibilities of a policy engine.

I'm not a Gatekeeper maintainer though, so maybe someone from that project can chip in and share their thoughts on the subject.

[maxsmythe]

But since Gatekeeper already supports the manipulation part, I think it should support the creation part as well.

It supports manipulation in the sense that it supports mutating webhooks. I agree that mutation is manipulation, but to a much lesser degree than resource creation, which would have the following issues:

• Gatekeeper would have the ability to create (and presumably delete) all resources, which is a significant increase in its security footprint.

• Creating resources based on other resources blurs RBAC boundaries. K8s controllers can do that as well, but it is usually some specialized resource (e.g. Prometheus config operators, Deployments, etc.), where that implication is well-understood.

• Depending on the resource created and whether there is ongoing enforcement of the existence of an object, there is the potential for controller fights, and the need to specify a conflict resolution strategy (e.g. for this example: is the entire config map replaced? Missing keys added?)

• Deletion strategy. This might be easy (set an owner reference and let k8s handle it), but there are edge cases: what if the created object was pre-existing? What about places where owner references don't work, such as namespace -> cluster scoped?

• How to handle updates to the policy: what if the labels on a namespace change such that it becomes out-of-scope for generation? What should happen with pre-existing resources? Other edge cases?

None of these questions are trivial. There is also the principal of sticking to limited sets of functionality that compose with other projects, so each project can do what it does best.

Hierarchical Namespace Controller provides copy ability:

https://github.com/kubernetes-sigs/hierarchical-namespaces/blob/master/docs/user-guide/quickstart.md#resources

though I'm not sure if it has the capacity to make all namespaces a child of a given root namespace, which would be necessary for zero-touch propagation to new namespaces.

cert-manager references reflector and kubed (now config syncer), which addresses the config map syncing use-case (incidentally, cert-manager may also have a solution for rotating these certificates if/when it becomes necessary, though I haven't dug into that).

GitOps workflows may also allow for hydration-type actions that inject resources.

None of this is to say that Gatekeeper couldn't do this. My personal preference would be to have a clear case for why we could do a better job than more focused solutions and the value it would add relative to the security risks and extra support surface.

[thomasmckay]

From what I'm seeing, asks for richer mutation and resource creation are driving the adoption of kyverno over gatekeeper. This would be a good topic to bring up in the community meeting, if you want deeper insight on how and why gatekeeper is like this.

[domruf]

Sorry for the late response.
Thanks to @srenatus for the tip about shell-operator. It is an interesting tool and I was able to write an generic operator that copies CMs to other namespaces.
But it is not exactly simple. So ATM I prefer kyverno. I find it to be much easier for my use cases.

#!/usr/bin/env bash

# clones ConfigMaps labeled with copyTo=allNamespaces to all namespaces

src_namespace=`cat /run/secrets/kubernetes.io/serviceaccount/namespace`
EXCLUDED_NAMESPACES="${src_namespace}
cattle-dashboards
cattle-fleet-system
cattle-impersonation-system
cattle-monitoring-system
cattle-system
fleet-clusters-system
fleet-local
fleet-system
ingress-nginx
kube-node-lease
kube-public
kube-system
kyverno
security-scan
trident
"

if [[ $1 == "--config" ]] ; then
    cat <<EOF
configVersion: v1
kubernetes:
- name: newNamespace
  apiVersion: v1
  kind: Namespace
  executeHookOnEvent: ["Added"]
- name: newCert
  apiVersion: v1
  kind: ConfigMap
  executeHookOnSynchronization: false
  executeHookOnEvent: ["Added", "Modified"]
  labelSelector:
    matchLabels:
      copyTo: allNamespaces
  namespace:
    nameSelector:
      matchNames: ["${src_namespace}"]
EOF
else
    bindingName=$(jq -r '.[0].binding' $BINDING_CONTEXT_PATH)
    if [[ $bindingName == "newNamespace" ]] ; then
        type=$(jq -r '.[0].type' $BINDING_CONTEXT_PATH)
        if [[ $type == "Synchronization" ]] ; then
            # all namespaces in sync objects
            trg_namespaces=`jq -r '.[0].objects | .[].object.metadata.name' $BINDING_CONTEXT_PATH`
        else
            # namespace in event object
            trg_namespaces=`jq -r '.[0].object.metadata.name' $BINDING_CONTEXT_PATH`
        fi
        src_configmaps=`kubectl get cm -A -l copyTo=allNamespaces -o jsonpath='{.items[*].metadata.name}'`
    elif [[ $bindingName == "newCert" ]] ; then
        # all namespaces
        trg_namespaces=`kubectl get ns -o jsonpath='{.items[*].metadata.name}'`
        src_configmaps=`jq -r '.[0].object.metadata.name' $BINDING_CONTEXT_PATH`
    fi

    for configmap_name in ${src_configmaps}
    do
        cm_files=`kubectl -n ${src_namespace} get cm ${configmap_name} -o json | jq -r '.data | keys[]'`
        from_file_parameters=""
        for f in ${cm_files}
        do
            kubectl -n ${src_namespace} get cm ${configmap_name} -o jsonpath="{.data.${f/\./\\.}}" > /tmp/${f}
            from_file_parameters+="--from-file=/tmp/${f} "
        done

        for trg_namespace in ${trg_namespaces}
        do
            if echo ${EXCLUDED_NAMESPACES} | grep "\b${trg_namespace}\b" > /dev/null ; then
                # echo "${trg_namespace} is excluded"
                continue
            fi
            echo "creating/updating ConfigMap ${configmap_name} in ${trg_namespace} with ${from_file_parameters}"
            kubectl -n ${trg_namespace} create cm ${configmap_name} ${from_file_parameters} -o yaml --dry-run=client | kubectl apply -f -
            kubectl -n ${trg_namespace} label cm --overwrite ${configmap_name} app.kubernetes.io/managed-by=shell-operator app.kubernetes.io/created-by=shell-operator
            # TODO: add hook to helm chart that removes all resources with app.kubernetes.io/managed-by=shell-operator
        done
    done
fi