Configure tokens and credentials from secrets/envs

[pierluigilenoci]

Is your feature request related to a problem? Please describe.

To date, the chart cannot be used with a GitOps approach.

Describe the solution you'd like

Give the ability to configure tokens and credentials from the environment variables instead of explicitly including them in the configuration. In this way integration with secrets (and vaults) is possible.

Describe alternatives you've considered

No alternative

Additional context

Not relevant

[pierluigilenoci]

Another issue for the same problem #319

[pierluigilenoci]

And another #211

[pierluigilenoci]

And a PR: #226

[pierluigilenoci]

And another PR: #233

[pierluigilenoci]

And here: https://github.com/ebrianne/helm-charts/issues/20 and https://github.com/ebrianne/helm-charts/pull/9

[pierluigilenoci]

I would say that the time has come to solve this problem. 😛

[pierluigilenoci]

I have seen @ebrianne's solution and it seems more of a workaround than a real solution. The proposed solution forces to keep an entire configMap somewhere with confidential information and, to my knowledge, in this way integration with a vault is not possible.

A more complete approach would be to allow to take tokens from environment variables (like Grafana or OAuth2 Proxy charts do for example). In this way, the ENV can be stored in secret and retrieved by a Vault.

[ebrianne]

This is not much of a workaround. I would say maybe a partial solution. I am only targeting the communication secret here that the user can provide directly and use for example SealedSecrets for GitOps. I did not think about the config (as a ConfigMap) itself, is that much sensitive information? I believe this could still be done in a similar way by providing a secret and mounting it but that may require some changes in botkube.

[pierluigilenoci]

I prefer not to save secrets in a repo, even if in a SealedSecret.

[pierluigilenoci]

@PrasadG193 any update on this?

[pierluigilenoci]

@PrasadG193 @thijsdejong @vaibhavp could you please take a look?

[PrasadG193]

@pierluigilenoci I think this is straightforward, do you mind submitting PR? Let me know if you need any help

[pierluigilenoci]

I'm sorry @PrasadG193 but right now I don't have time to prepare a PR. 😞
But since it is straightforward I am sure you will be able to do it in a short time given how many people are affected and the value it brings to the chart. 🚀
Thank you! ❤️

[pierluigilenoci]

@PrasadG193 any chances to see this fixed?

[pierluigilenoci]

@PrasadG193 any news about this? v0.12.4 is out but this is not implemented.

[pierluigilenoci]

@PrasadG193 could you please take a look?

[mszostok]

Botkube secrets configuration

Communication

values.yaml definition:

# Communication settings  
communications:  
  
  # Using existing Communication secret  
  existingSecretName: ""

  # Settings for Slack
  slack:
    enabled: false
    channel: 'SLACK_CHANNEL'
    token: 'SLACK_API_TOKEN'
    notiftype: short        
# ...trimmed...

To use already existing secret:

1. Create secret in proper format:

   NOTE: Secret MUST be created in the same Namespace where Botkube is installed.

   # Create Communication secret
   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: communication
     namespace: botkube
   type: Opaque
   stringData:
     comm_config.yaml: |  
       communications:
         slack:  
           enabled: true
           channel: 'random'
           token: 'xoxb-abcd-token'
           notiftype: short
   EOF

2. Install Botkube via Helm chart:

   helm repo add infracloudio https://infracloudio.github.io/charts

   helm install --version v0.12.4 botkube --namespace botkube \
   --set communications.existingSecretName=communication \
   --set config.settings.clustername=other-cluster \
   --set config.settings.kubectl.enabled=true \
   --set image.repository=infracloudio/botkube \
   --set image.tag=v0.12.4 \
   infracloudio/botkube

Kubeconfig

Available only on develop (v9.99.9-dev). Will be released in v0.13.0.

values.yaml definition:

kubeconfig:  
  # If true, enables overriding the kubernetes auth  
  enabled: false  
  # A base64 encoded kubeconfig that will be stored in a secret, mounted to the pod, and specified in the KUBECONFIG environment variable.  
  base64Config: ""  
  # A secret containing a kubeconfig to use.  
  existingSecret: ""

Install against external k8s cluster:

1. Create secret in proper format:

   NOTE: Secret MUST be created in the same Namespace where Botkube is installed.

   # Create kubeconfig secret
   export KUBECONFIG_PATH="/tmp/external-k8s.yaml"
   cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: Secret
   metadata:
     name: external-k8s-cfg
     namespace: botkube
   type: Opaque
   data:
     config: $(cat $KUBECONFIG_PATH | base64)
   EOF

2. Install Botkube:

   helm install --version v9.99.9-dev botkube --namespace botkube --create-namespace \
   --set communications.slack.enabled=true \
   --set communications.slack.channel=random \
   --set communications.slack.token="xoxb-abcd-token" \
   --set config.settings.clustername=other-cluster \
   --set config.settings.kubectl.enabled=true \
   --set image.repository=infracloudio/botkube \
   --set image.tag=v0.12.4 \
   --set kubeconfig.enabled=true \
   --set kubeconfig.existingSecret="external-k8s-cfg" \
   --set automountServiceAccountToken=false \
   infracloudio/botkube

BEWARE: Configuring the external k8s is tricky as we need to also set automountServiceAccountToken=false. It's because the client prefers internal (SA) configuration, see:

https://github.com/infracloudio/botkube/blob/0a2bf80b8cc467a934614fb0e4433a204955213a/pkg/utils/utils.go#L82-L88

IMO remembering to set additionally automountServiceAccountToken=false is misleading. I see two options:

1. If kubeconfig.enabled is set, disable automountServiceAccountToken
2. if kubeconfig.enabled is set, in InitKubeClient() func, default to KUBECONFIG env.

This could be address with #589, where we plan to get rid of global variables. Then we can start consuming kubeconfig.enabled property.

@PrasadG193 @pkosiec do you agree?

SSL

ssl:                                           # For using custom SSL certificates  
  enabled: false                               # Set to true and specify cert path in the next line after uncommenting  
  #cert:                                       # SSL Certificate file e.g certs/my-cert.crt

NOTE: No option to specify an external secret. Will be fixed by #597

Ingress

# Ingress settings to expose teams and lark endpoints  
ingress:  
  create: false  
  annotations:  
    kubernetes.io/ingress.class: nginx  
  host: 'HOST'  
  tls:  
    enabled: false  
    secretName: ''

Summary

In general, Botkube is able to consume sensitive data from external secrets. The Communication data can be specified already in v0.12.4. @pierluigilenoci could you share what else is needed?

I also added short description about Secret format and opt to specify SSL via external Secret in #597

[pierluigilenoci]

@mszostok thank you for your reply.

As I wrote explicitly in the issue description:

configure tokens and credentials from the environment variables

The configuration you suggest still requires manual intervention to create the Secret.
What I would like is to be able to fetch the secret from external vaults in order to automate the installation (GitOps approach).

[pierluigilenoci]

I used csi-secrets-store-provider-aws and csi-secrets-store-provider-azure to inject the secrets into the pods.

Azure example:
https://github.com/Azure/secrets-store-csi-driver-provider-azure/blob/master/examples/service-principal/pod-inline-volume-service-principal.yaml#L13-L23

AWS example:
https://github.com/aws/secrets-store-csi-driver-provider-aws/blob/main/examples/ExampleDeployment.yaml#L32-L47

[mszostok]

Thanks for the feedback!

Previously, the only option was to specify the sensitive data via helm install --set. Now chart also accept already existing Secret that can be created even automatically by e.g. Vault operator.

Now, the only option to specify communicator configuration is to pass the whole configuration file. To expose the individual environment variable we would need to use some lib e.g. https://github.com/knadh/koanf. For now, (as a workaround?), you can use the extraAnnotations. This could be used with Vault operator to inject the Secret automatically:

    vault.hashicorp.com/agent-inject: 'true'
    vault.hashicorp.com/role: 'internal-app'
    vault.hashicorp.com/agent-inject-secret-database-config.txt: 'internal/data/database/config'
    vault.hashicorp.com/agent-inject-template-database-config.txt: |
      {{- with secret "internal/data/database/config" -}}
      postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
      {{- end -}}
spec:
// .. trimmed...

(it's just an example data based on their tutorial)
In this way, you will still have an option to consume data from an external system (e.g. Vault) and construct the required Secret format.

However, I will try to create a showcase with the project that you mentioned, before we will put more customization in place. Will keep you posted 👍

[pierluigilenoci]

@mszostok I would like to do what you write for Hashicorp Vault using CSI and the default vaults of Cloud Providers (specifically AWS and Azure).

[mszostok]

I tested that with Secret Store CSI and Vault provider. It turns out that more than the environment variable, you need an option to attach a given volume with your configuration. It applies even if you want to use the Sync Secret feature:

The volume mount is required for the Sync With Kubernetes Secrets

source: https://secrets-store-csi-driver.sigs.k8s.io/topics/sync-as-kubernetes-secret.html

However, you are able to do that even with the current Helm chart, as BotKube allows using other Kubernetes Secret.

BotKube with Vault tutorial

Prerequisite

Install Vault in cluster according to this tutorial: https://learn.hashicorp.com/tutorials/vault/kubernetes-secret-store-driver#install-the-vault-helm-chart

NOTE: Add --set syncSecret.enabled=true during secrets-store-csi-driver installation.

Also, when configuring access, use:

vault policy write internal-app - <<EOF
path "secret/data/comm_config" {
  capabilities = ["read"]
}
EOF

vault write auth/kubernetes/role/database \
    bound_service_account_names=botkube-sa \
    bound_service_account_namespaces=default \
    policies=internal-app \
    ttl=20m

Steps

Click to expand!

1. Exec to Vault Pod:

   kubectl exec -it vault-0 -- /bin/sh

2. Create config file:

   cat > in.json <<EOF
   {
       "data": {
      	 "communications": {
      		 "slack": {
      			 "enabled": true,
      			 "channel": "random",
      			 "token": "xoxb-token-abc"
      		 }
      	 }
       }
   }
   EOF

3. Create KV:

   vault kv put -format=json secret/comm_config @in.json

4. Create values for BotKube installation:

cat > /tmp/values.yaml << ENDOFFILE
extraObjects:
    - apiVersion: secrets-store.csi.x-k8s.io/v1
      kind: SecretProviderClass
      metadata:
        name: vault-database
      spec:
        provider: vault
        secretObjects:
          - data:
              - key: comm_config.yaml
                objectName: "comm-config"
            secretName: communication
            type: Opaque
        parameters:
          vaultAddress: "http://vault.default:8200"
          roleName: "database"
          objects: |
            - objectName: "comm-config"
              secretPath: "secret/data/comm_config"
              secretKey: "data"
      - kind: Pod
        apiVersion: v1
        metadata:
          name: init-secret
        spec:
          serviceAccountName: botkube-sa
          containers:
            - image: alpine
              name: init
              volumeMounts:
                - name: secrets-store-inline
                  mountPath: "/mnt/secrets-store"
                  readOnly: true
          volumes:
            - name: secrets-store-inline
              csi:
                driver: secrets-store.csi.k8s.io
                readOnly: true
                volumeAttributes:
                  secretProviderClass: "vault-database"

communications:
  existingSecretName: "communication"
serviceAccount:
  create: false
  name: botkube-sa # this SA is pre-authorized to access vault-database
  ENDOFFILE

5. Install BotKube with additional extra objects:

   helm install botkube --version v9.99.9-dev infracloudio/botkube -f /tmp/values.yaml

Explanation

The SecretProviderClass CR allows us to create the K8s Secret. This secret is created only when the Volume is mounted to some Pod. We create a init Pod to fulfil this requirement. Later we use the created Secret during BotKube installation. Because the extraObjects becomes the BotKube release object, they are tight to a given installation and due to that fact it works well together. One simplification that can be introduced is an option to "override" mounted volume with Secret. See next section, for sample example.

BotKube with Vault tutorial (override Volume)

This approach requires modifying the BotKube Helm chart:

Click to expand!

diff --git a/helm/botkube/templates/communicationsecret.yaml b/helm/botkube/templates/communicationsecret.yaml
index 84f881f..4e59516 100644
--- a/helm/botkube/templates/communicationsecret.yaml
+++ b/helm/botkube/templates/communicationsecret.yaml
@@ -1,4 +1,4 @@
-{{- if not .Values.communications.existingSecretName -}}
+{{- if not (or .Values.communications.existingSecretName .Values.communications.existingVolume) -}}
 apiVersion: v1
 kind: Secret
 metadata:
diff --git a/helm/botkube/templates/deployment.yaml b/helm/botkube/templates/deployment.yaml
index 2e043da..a109b52 100644
--- a/helm/botkube/templates/deployment.yaml
+++ b/helm/botkube/templates/deployment.yaml
@@ -50,7 +50,13 @@ spec:
           {{ end }}
           volumeMounts:
             - name: config-volume
-              mountPath: "/config"
+              mountPath: "/config/resource_config.yaml"
+              subPath: "resource_config.yaml"
+          {{- if .Values.communications.existingVolume }}
+            - name: {{ .Values.communications.existingVolume.name }}
+              mountPath: "/config/comm_config.yaml"
+              subPath: "comm_config.yaml"
+          {{ end }}
           {{- if .Values.config.ssl.enabled }}
             - name: certs
               mountPath: "/etc/ssl/certs"
@@ -89,8 +95,13 @@ spec:
             sources:
             - configMap:
                 name: {{ include "botkube.fullname" . }}-configmap
+          {{- if not .Values.communications.existingVolume }}
             - secret:
                 name: {{ include "botkube.CommunicationsSecretName" . }}
+          {{ end }}
+      {{- if .Values.communications.existingVolume }}
+        -{{ toYaml .Values.communications.existingVolume | nindent 10 }}
+      {{ end }}
       {{- if .Values.config.ssl.enabled }}
         - name: certs
           secret:
diff --git a/helm/botkube/values.yaml b/helm/botkube/values.yaml
index 94e02b0..0f2db7a 100644
--- a/helm/botkube/values.yaml
+++ b/helm/botkube/values.yaml
@@ -307,6 +307,9 @@ communications:
   #        # Here specify settings for each app, like Slack, Mattermost etc.
   #        # NOTE: Use setting format visible below.
   existingSecretName: ""
+  # Using existing Volume where the Communication data is available.
+  # If used, the default and `existingSecretName` configuration is ignored.
+  existingVolume: {}

   # Settings for Slack
   slack:

NOTE: The subPath is not recommended, we will need to change that in code, if we will decided to implement it.

Then we are able to specify such installation values:

cat > /tmp/values.yaml << ENDOFFILE
extraObjects:
  - apiVersion: secrets-store.csi.x-k8s.io/v1
    kind: SecretProviderClass
    metadata:
      name: vault-database
    spec:
      provider: vault
      parameters:
        vaultAddress: "http://vault.default:8200"
        roleName: "database"
        objects: |
          - objectName: "comm_config.yaml"
            objectAlias: "comm_config.yaml"
            secretPath: "secret/data/comm_config"
            secretKey: "data"

communications:
  existingVolume:
    name: comm-config
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "vault-database"
serviceAccount:
  create: false
  name: botkube-sa # this SA is pre-authorized to access vault-database

  ENDOFFILE

helm install botkube -f /tmp/values.yaml ./helm/botkube

As a result, the installation is simplified and more seamless.

Other options

Allow a generic volume mount:

extraVolumes:
  - name: comm-config
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "vault-database"

extraVolumeMounts:
  - mountPath: /config # requires you to use some predefined path or you will need to override the `CONFIG_PATH` env.
    name: comm-config

However, this more complicated at the end as you also need to disable the default config volume and also ensure that the path of mounted volume matches the expected one, or have an option to override them.

Summary

IMO changing the current Helm chart is not required. Having an option to use the external Kubernetes Secret is already a generic solution which can be used to configure communication data with different sources.

@PrasadG193 @pkosiec @pierluigilenoci please share your thoughts.

Cheers!

[pkosiec]

Nice investigation @mszostok 🚀 I agree, from my point of view current Helm configuration is flexible enough. Later we can introduce extraVolumes/extraVolumeMounts configuration, but first we need to make sure to read communication/resource config from separate paths 👍

[pierluigilenoci]

@mszostok some notes:

• The solution you propose is to move the entire configuration into a vault (whatever it is) instead of reading from the vault only the secrets needed to inject them into the configuration.

• The default CSI provider for SecretProviderClasses is other than what you suggest (Hashicorp).
  Specifically, each cloud uses its own provider: azure and was.

I understand the desire to change as little as possible but the proposed solution does not seem a viable path.

Another example of a chart that does this:
https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml#L399-L455

I want to add that @ebrianne had already done a PR to achieve this result.
In the first comments of the discussion, I linked the proposals.

[mszostok]

Hi @pierluigilenoci

The solution you propose is to move the entire configuration into a vault (whatever it is) instead of reading from the vault only the secrets needed to inject them into the configuration.

Yep, but this has pros and cons. In one place, you have a single source of truth. Additionally, you can see the whole config at once, which requires less cognitive overhead.

The default CSI provider for SecretProviderClasses is other than what you suggest (Hashicorp).
Specifically, each cloud uses its own provider: azure and was.

What is different? There is a single interface that has a multiple provider. Provided example should work in all cases.

---

With the current Helm chart, the provided solution works pretty well, and you can use external sources.

But, in general, I also agree with you 👍 I created and tested an implementation which:

• allows providing communicator configuration via env variables
• env variables have higher priority that config from file
• chart has:
  • extraEnv
  • extraVolumeMounts
  • extraVolumes

With all of that, the values.yaml can look like:

extraObjects:
  - apiVersion: secrets-store.csi.x-k8s.io/v1
    kind: SecretProviderClass
    metadata:
      name: vault-database
    spec:
      provider: vault
      secretObjects:
        - data:
            - key: token
              objectName: "slack-token"
          secretName: communication-slack
          type: Opaque
      parameters:
        vaultAddress: "http://vault.default:8200"
        roleName: "database"
        objects: |
          - objectName: "slack-token"
            secretPath: "secret/data/slack"
            secretKey: "token"

communications:
  # Settings for Slack
  slack:
    enabled: true
    channel: 'random'
    notiftype: short
    # token - specified via env

extraEnv:
  - name: COMMUNICATION_SLACK_TOKEN
    valueFrom:
      secretKeyRef:
        name: communication-slack
        key: token

extraVolumeMounts:
  - name: secrets-store-inline
    mountPath: "/mnt/secrets-store"
    readOnly: true

extraVolumes:
  - name: secrets-store-inline
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "vault-database"

Where e.g. in Vault we have only the token value:

vault kv put secret/slack token=xoxb-token-abc

Here is the implementation:
https://github.com/pkosiec/botkube/compare/golangci-lint...mszostok:support-env-and-vols?expand=1

Once the #593 will be merged, I will create a dedicated PR. I understand that this will fulfil your use-case in 100%?

---

EDIT: PR ready for review: #601

[pierluigilenoci]

@mszostok let's say that's okay for now. 👌🏻
Obviously, the devil is in the details and I certainly overlook something. 😈 😞
I will let you know as soon as it will be possible to try it. 🤞🏻

[mszostok]

Hi, the PR was merged, and now new functionality is available on develop and will be release as a part of v0.13.0.

However, you can try it already with the develop version. To do that, follow this tutorial: https://www.botkube.io/configuration/communication/vault-csi/

And here is a recorded demo: