refactor!: Split into Deployment and DaemonSet (#645)

* refactor: Split into Deployment and DaemonSet

The splits the deployment of the secret-operator as a whole into
two parts:

- The controller is deployed via a Deployment which ensures only
  a single instance of the secret-operator in controller mode is
  running in a Kubernetes cluster. This can potentially lead to
  perfomance issues and as such should be monitored going forward.
- The CSI server is deployed via a DaemonSet (unchanged) as this
  server is needed on every node to provision requested secret
  volumes.

This refactor is introduced in preparation for #634, in which only
a single instance of the CRD conversion webhook must exist as
otherwise TLS certificate verification will fail with multiple
available certificates.

* chore: Add changelog entry

* chore: Update comment

* refactor: Adjust values.yaml file to be closer to listener-operator

* chore: Adjust changelog entry

Author: Techassi

CHANGELOG.md

@@ -15,15 +15,18 @@ All notable changes to this project will be documented in this file.
 
 ### Changed
 
+- Split operator deployment into Deployment and DaemonSet ([#645]).
+  - Introduce two different modes: `csi-server` and `controller`.
+  - The CSI server is deployed via a DaemonSet to be available on every node.
+  - The controller is deployed via a Deployment with a single replica.
 - Version CRD structs and enums as v1alpha1 ([#636]).
-- BREAKING: Rearrange values to be somewhat consistent with the listener-operator value changes ([#641]).
-  - `image.repository` has been moved to `secretOperator.image.repository`.
-  - `image.tag` has been moved to `secretOperator.image.tag`.
-  - `image.pullPolicy` has been moved to `secretOperator.image.pullPolicy`.
-  - `csiProvisioner` values have been moved to `externalProvisioner`.
-  - `csiNodeDriverRegistrar` values have been moved to `nodeDriverRegistrar`.
-  - `node.driver` values have been moved to `secretOperator`.
-  - `securityContext` values have been moved to `secretOperator.securityContext`.
+- BREAKING: Rearrange values to be somewhat consistent with the listener-operator value changes ([#641], [#645]).
+  - `csiProvisioner` values have been moved to `csiNodeDriver.externalProvisioner`.
+  - `csiNodeDriverRegistrar` values have been moved to `csiNodeDriver.nodeDriverRegistrar`.
+  - `node.driver.resources` values have been split into `controllerService.resources` and `csiNodeDriver.nodeService.resources`.
+  - `securityContext` values have been split into `controllerService.securityContext` and `.csiNodeDriver.nodeService.securityContext`.
+  - `podAnnotations`, `podSecurityContext`, `nodeSelector`, `tolerations`, and `affinity` have been split into `controllerService` and `csiNodeDriver`.
+  - `kubeletDir` has been move to `csiNodeDriver.kubeletDir`.
 - Bump csi-node-driver-registrar to `v2.15.0` ([#642]).
 - Bump csi-provisioner to `v5.3.0` ([#643]).
 
@@ -33,6 +36,7 @@ All notable changes to this project will be documented in this file.
 [#642]: https://github.com/stackabletech/secret-operator/pull/642
 [#643]: https://github.com/stackabletech/secret-operator/pull/643
 [#644]: https://github.com/stackabletech/secret-operator/pull/644
+[#645]: https://github.com/stackabletech/secret-operator/pull/645
 
 ## [25.7.0] - 2025-07-23
 

Tiltfile

@@ -26,6 +26,7 @@ if os.path.exists('result'):
 # oci.stackable.tech/sandbox/opa-operator:7y19m3d8clwxlv34v5q2x4p7v536s00g instead of
 # oci.stackable.tech/sandbox/opa-operator:0.0.0-dev (which does not exist)
 k8s_kind('Deployment', image_json_path='{.spec.template.metadata.annotations.internal\\.stackable\\.tech/image}')
+k8s_kind('DaemonSet', image_json_path='{.spec.template.metadata.annotations.internal\\.stackable\\.tech/image}')
 
 # Exclude stale CRDs from Helm chart, and apply the rest
 helm_crds, helm_non_crds = filter_yaml(

deploy/helm/secret-operator/templates/controller-deployment.yaml

@@ -0,0 +1,97 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: {{ include "operator.fullname" . }}
+  labels:
+    {{- include "operator.labels" . | nindent 4 }}
+spec:
+  selector:
+    matchLabels:
+      {{- include "operator.selectorLabels" . | nindent 6 }}
+  template:
+    metadata:
+      annotations:
+        internal.stackable.tech/image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
+        {{- with .Values.controllerService.podAnnotations }}
+          {{- toYaml . | nindent 8 }}
+        {{- end }}
+      labels:
+        {{- include "operator.selectorLabels" . | nindent 8 }}
+    spec:
+      {{- with .Values.image.pullSecrets }}
+      imagePullSecrets:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      # NOTE (@Techassi): Does it maybe make sense to have two different service accounts?
+      serviceAccountName: {{ include "operator.fullname" . }}-serviceaccount
+      securityContext:
+        {{- toYaml .Values.controllerService.podSecurityContext | nindent 8 }}
+      containers:
+        - name: {{ include "operator.appname" . }}
+          securityContext:
+            {{- toYaml .Values.controllerService.securityContext | nindent 12 }}
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          resources:
+            {{ .Values.controllerService.resources | toYaml | nindent 12 }}
+          # The arguments passed to the command being run in the container. The final command will
+          # look like `secret-operator run controller [OPTIONS]`. The controller needs to only run
+          # once in a Kubernetes cluster and as such is deployed as a Deployment with a single
+          # replica.
+          args:
+            - run
+            - controller
+          env:
+            # The following env vars are passed as clap (think CLI) arguments to the operator.
+            # They are picked up by clap using the structs defied in the operator.
+            # (which is turn pulls in https://github.com/stackabletech/operator-rs/blob/main/crates/stackable-operator/src/cli.rs)
+            # You can read there about the expected values and purposes.
+
+            # Sometimes products need to know the operator image, e.g. the opa-bundle-builder OPA
+            # sidecar uses the operator image.
+            - name: OPERATOR_IMAGE
+              # Tilt can use annotations as image paths, but not env variables
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.annotations['internal.stackable.tech/image']
+
+            # Namespace the operator Pod is running in, e.g. used to construct the conversion
+            # webhook endpoint.
+            - name: OPERATOR_NAMESPACE
+              valueFrom:
+                fieldRef:
+                  fieldPath: metadata.namespace
+
+            # The name of the Kubernetes Service that point to the operator Pod, e.g. used to
+            # construct the conversion webhook endpoint.
+            - name: OPERATOR_SERVICE_NAME
+              value: {{ include "operator.fullname" . }}
+
+            # Operators need to know the node name they are running on, to e.g. discover the
+            # Kubernetes domain name from the kubelet API.
+            - name: KUBERNETES_NODE_NAME
+              valueFrom:
+                fieldRef:
+                  fieldPath: spec.nodeName
+
+            {{- if .Values.kubernetesClusterDomain }}
+            - name: KUBERNETES_CLUSTER_DOMAIN
+              value: {{ .Values.kubernetesClusterDomain | quote }}
+            {{- end }}
+            {{- include "telemetry.envVars" . | nindent 12 }}
+      {{- with .Values.controllerService.nodeSelector }}
+      nodeSelector:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.controllerService.affinity }}
+      affinity:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.controllerService.tolerations }}
+      tolerations:
+        {{- toYaml . | nindent 8 }}
+      {{- end }}
+      {{- with .Values.controllerService.priorityClassName }}
+      priorityClassName: {{ . }}
+      {{- end }}

deploy/helm/secret-operator/templates/daemonset.yaml renamed to deploy/helm/secret-operator/templates/csi-node-driver-daemonset.yaml

@@ -2,7 +2,7 @@
 apiVersion: apps/v1
 kind: DaemonSet
 metadata:
-  name: {{ include "operator.fullname" . }}-daemonset
+  name: {{ include "operator.fullname" . }}-csi-node-driver
   labels:
     {{- include "operator.labels" . | nindent 4 }}
 spec:
@@ -11,28 +11,36 @@ spec:
       {{- include "operator.selectorLabels" . | nindent 6 }}
   template:
     metadata:
-      {{- with .Values.podAnnotations }}
       annotations:
-        {{- toYaml . | nindent 8 }}
-      {{- end }}
+        internal.stackable.tech/image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
+        {{- with .Values.csiNodeDriver.podAnnotations }}
+          {{- toYaml . | nindent 8 }}
+        {{- end }}
       labels:
         {{- include "operator.selectorLabels" . | nindent 8 }}
     spec:
       {{- with .Values.image.pullSecrets }}
       imagePullSecrets:
         {{- toYaml . | nindent 8 }}
       {{- end }}
+      # NOTE (@Techassi): Does it maybe make sense to have two different service accounts?
       serviceAccountName: {{ include "operator.fullname" . }}-serviceaccount
       securityContext:
-        {{- toYaml .Values.podSecurityContext | nindent 8 }}
+        {{- toYaml .Values.csiNodeDriver.podSecurityContext | nindent 8 }}
       containers:
-        - name: {{ include "operator.appname" . }}
+        - name: csi-node-service
           securityContext:
-            {{- toYaml .Values.secretOperator.securityContext | nindent 12 }}
-          image: "{{ .Values.secretOperator.image.repository }}:{{ .Values.secretOperator.image.tag | default .Chart.AppVersion }}"
-          imagePullPolicy: {{ .Values.secretOperator.image.pullPolicy }}
+            {{- toYaml .Values.csiNodeDriver.nodeService.securityContext | nindent 12 }}
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
           resources:
-            {{ .Values.secretOperator.resources | toYaml | nindent 12 }}
+            {{ .Values.csiNodeDriver.nodeService.resources | toYaml | nindent 12 }}
+          # The arguments passed to the command being run in the container. The final command will
+          # look like `secret-operator run csi-server [OPTIONS]`. The CSI server needs to run on
+          # every Kubernetes cluster node and as such is deployed as a DaemonSet.
+          args:
+            - run
+            - csi-node-service
           env:
             # The following env vars are passed as clap (think CLI) arguments to the operator.
             # They are picked up by clap using the structs defied in the operator.
@@ -42,7 +50,7 @@ spec:
             - name: CSI_ENDPOINT
               value: /csi/csi.sock
             - name: PRIVILEGED
-              value: {{ .Values.secretOperator.securityContext.privileged | quote }}
+              value: {{ .Values.csiNodeDriver.nodeService.securityContext.privileged | quote }}
 
             # Sometimes products need to know the operator image, e.g. the opa-bundle-builder OPA
             # sidecar uses the operator image.
@@ -80,32 +88,34 @@ spec:
             - name: csi
               mountPath: /csi
             - name: mountpoint
-              mountPath: {{ .Values.kubeletDir }}/pods
-              {{- if .Values.secretOperator.securityContext.privileged }}
+              mountPath: {{ .Values.csiNodeDriver.kubeletDir }}/pods
+              {{- if .Values.csiNodeDriver.nodeService.securityContext.privileged }}
               mountPropagation: Bidirectional
               {{- end }}
             - name: tmp
               mountPath: /tmp
+
         - name: external-provisioner
-          image: "{{ .Values.externalProvisioner.image.repository }}:{{ .Values.externalProvisioner.image.tag }}"
-          imagePullPolicy: {{ .Values.externalProvisioner.image.pullPolicy }}
+          image: "{{ .Values.csiNodeDriver.externalProvisioner.image.repository }}:{{ .Values.csiNodeDriver.externalProvisioner.image.tag }}"
+          imagePullPolicy: {{ .Values.csiNodeDriver.externalProvisioner.image.pullPolicy }}
           resources:
-            {{ .Values.externalProvisioner.resources | toYaml | nindent 12 }}
+            {{ .Values.csiNodeDriver.externalProvisioner.resources | toYaml | nindent 12 }}
           args:
             - --csi-address=/csi/csi.sock
             - --feature-gates=Topology=true
             - --extra-create-metadata
           volumeMounts:
             - name: csi
               mountPath: /csi
+
         - name: node-driver-registrar
-          image: "{{ .Values.nodeDriverRegistrar.image.repository }}:{{ .Values.nodeDriverRegistrar.image.tag }}"
-          imagePullPolicy: {{ .Values.nodeDriverRegistrar.image.pullPolicy }}
+          image: "{{ .Values.csiNodeDriver.nodeDriverRegistrar.image.repository }}:{{ .Values.csiNodeDriver.nodeDriverRegistrar.image.tag }}"
+          imagePullPolicy: {{ .Values.csiNodeDriver.nodeDriverRegistrar.image.pullPolicy }}
           resources:
-            {{ .Values.nodeDriverRegistrar.resources | toYaml | nindent 12 }}
+            {{ .Values.csiNodeDriver.nodeDriverRegistrar.resources | toYaml | nindent 12 }}
           args:
             - --csi-address=/csi/csi.sock
-            - --kubelet-registration-path={{ .Values.kubeletDir }}/plugins/secrets.stackable.tech/csi.sock
+            - --kubelet-registration-path={{ .Values.csiNodeDriver.kubeletDir }}/plugins/secrets.stackable.tech/csi.sock
           volumeMounts:
             - name: registration-sock
               mountPath: /registration
@@ -116,27 +126,27 @@ spec:
           hostPath:
             # node-driver-registrar appends a driver-unique filename to this path to avoid conflicts
             # see https://github.com/stackabletech/secret-operator/issues/229 for why this path should not be too long
-            path: {{ .Values.kubeletDir }}/plugins_registry
+            path: {{ .Values.csiNodeDriver.kubeletDir }}/plugins_registry
         - name: csi
           hostPath:
-            path: {{ .Values.kubeletDir }}/plugins/secrets.stackable.tech/
+            path: {{ .Values.csiNodeDriver.kubeletDir }}/plugins/secrets.stackable.tech/
         - name: mountpoint
           hostPath:
-            path: {{ .Values.kubeletDir }}/pods/
+            path: {{ .Values.csiNodeDriver.kubeletDir }}/pods/
         - name: tmp
           emptyDir: {}
-      {{- with .Values.nodeSelector }}
+      {{- with .Values.csiNodeDriver.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- with .Values.affinity }}
+      {{- with .Values.csiNodeDriver.affinity }}
       affinity:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- with .Values.tolerations }}
+      {{- with .Values.csiNodeDriver.tolerations }}
       tolerations:
         {{- toYaml . | nindent 8 }}
       {{- end }}
-      {{- with .Values.priorityClassName }}
+      {{- with .Values.csiNodeDriver.priorityClassName }}
       priorityClassName: {{ . }}
       {{- end }}