operator support for mounting secrets from CSI secret store

[jmazzitelli]

part of: kiali/kiali#6942

helm-charts companion PR: kiali/helm-charts#250

[jmazzitelli]

This is not ready yet. Need to write some jinja code to loop through the custom_secrets and convert snake_case so it retains the camelCase

[jmazzitelli]

jinja code added. Ready for review.

[jmazzitelli]

@rumstead can you review this little documentation blurb? This will show up in the kiali.io documentation website in the Kiali CR reference page - it will end up somewhere around here. I just want to make sure I'm not blowing smoke :) If you have some better documentation blurb to put here, please suggest it.

[rumstead]

LGTM. Should we add an example to the already existing examples?

Maybe:

    - name: "kiali-secret-csi"
      mount: "/mnt/secrets-store"
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: kiali-secretprovider

If you think we need more docs in the future like there are for custom dashboards I can contribute something.

[jmazzitelli]

LGTM. Should we add an example to the already existing examples?

We could. I can add it to this PR. I don't know what the true values would/should look like, so please suggest something that would be good for that example CR.

If you think we need more docs in the future like there are for custom dashboards I can contribute something.

Yes. I was thinking we should have a dedicated "something" in the docs for this. I was thinking it is a simple FAQ that we would add here which shows up here. Feel free to submit a PR over in the kiali.io repo.

[jmazzitelli]

I did some quick manual testing just to confirm there isn't obvious bugs here (and with the helm charts PR) and everything looks good. It appears CSI secrets must exist at the time of deployment (there is no optional option as discussed here) so my little blurb in the reference docs in my latest commit should handle that.

So this PR is ready to be merged.

[jmazzitelli]

For the record, I do not have CSI set up, so my testing just involved deploying a Kiali CR and seeing the server pod try to start up (even though it can't) and that it has the proper deployment yaml.

So in short, did the following quick test:

1. Deploy a dev build of the server and operator using make: make -e CLUSTER_TYPE=minikube -e HELM_CHARTS_REPO_PULL=false build build-ui cluster-push operator-create
2. I edited the kiali_cr_dev.yaml in the operator repo:

diff --git a/deploy/kiali/kiali_cr_dev.yaml b/deploy/kiali/kiali_cr_dev.yaml
index d80ebe4..62d46ba 100644
--- a/deploy/kiali/kiali_cr_dev.yaml
+++ b/deploy/kiali/kiali_cr_dev.yaml
@@ -28,6 +28,15 @@ spec:
     service_type: $SERVICE_TYPE
     logger:
       log_level: info
+    custom_secrets:
+    - name: csi-test
+      mount: /csi-test
+      optional: true
+      csi:
+        driver: secrets-store.csi.k8s.io
+        readOnly: true
+        volumeAttributes:
+          secretProviderClass: kiali-secretprovider

3. Deploy that Kiali CR via make: make -e CLUSTER_TYPE=minikube kiali-create

At this point you will see the server pod trying to start but failing because CSI isn't set up - but that confirms the CSI yaml is correctly configured:

$ kubectl describe pods -n istio-system -l app.kubernetes.io/name=kiali
Name:             kiali-696494dfd-bjp6k
Namespace:        istio-system
...
    Mounts:
      /csi-test from csi-test (rw)
      /kiali-cabundle from kiali-cabundle (rw)
      /kiali-configuration from kiali-configuration (rw)
      /kiali-override-secrets/login-token-signing-key from login-token-signing-key (ro)
      /kiali-secret from kiali-secret (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-j8l8r (ro)
...
Volumes:
...
  csi-test:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:            
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=kiali-secretprovider
...
Events:
  Type     Reason       Age                From               Message
  ----     ------       ----               ----               -------
  Normal   Scheduled    54s                default-scheduler  Successfully assigned istio-system/kiali-696494dfd-bjp6k to ci
  Warning  FailedMount  22s (x7 over 54s)  kubelet            MountVolume.SetUp failed for volume "csi-test" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers

You'll notice optional: true that I specified in the Kiali CR is ignored.