[Enhancement]: Support additionalVolumes

[cthtrifork]

Related problem

besides the storage.volumes used directly for kafka storage, it would be nice to set additionalVolumes for things such as extracting logfiles.

I won't mind working on the pull request, but I need to make sure there is acceptance of the solution.

Suggested solution

Add additionalVolumes to the CRD

              additionalVolumes:
                description: Additional volumes to mount to all pods in the cluster
                items:
                  properties:
                    configMap:
                      description: ConfigMap to use to populate the volume
                      properties:
                        defaultMode:
                          description: |-
                            defaultMode is optional: mode bits used to set permissions on created files by default.
                            Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511.
                            YAML accepts both octal and decimal values, JSON requires decimal values for mode bits.
                            Defaults to 0644.
                            Directories within the path are not affected by this setting.
                            This might be in conflict with other options that affect the file
                            mode, like fsGroup, and the result can be other mode bits set.
                          format: int32
                          type: integer
                        items:
                          description: |-
                            items if unspecified, each key-value pair in the Data field of the referenced
                            ConfigMap will be projected into the volume as a file whose name is the
                            key and content is the value. If specified, the listed keys will be
                            projected into the specified paths, and unlisted keys will not be
                            present. If a key is specified which is not present in the ConfigMap,
                            the volume setup will error unless it is marked optional. Paths must be
                            relative and may not contain the '..' path or start with '..'.
                          items:
                            description: Maps a string key to a path within a
                              volume.
                            properties:
                              key:
                                description: key is the key to project.
                                type: string
                              mode:
                                description: |-
                                  mode is Optional: mode bits used to set permissions on this file.
                                  Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511.
                                  YAML accepts both octal and decimal values, JSON requires decimal values for mode bits.
                                  If not specified, the volume defaultMode will be used.
                                  This might be in conflict with other options that affect the file
                                  mode, like fsGroup, and the result can be other mode bits set.
                                format: int32
                                type: integer
                              path:
                                description: |-
                                  path is the relative path of the file to map the key to.
                                  May not be an absolute path.
                                  May not contain the path element '..'.
                                  May not start with the string '..'.
                                type: string
                            required:
                            - key
                            - path
                            type: object
                          type: array
                        name:
                          description: |-
                            Name of the referent.
                            More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                            TODO: Add other useful fields. apiVersion, kind, uid?
                          type: string
                        optional:
                          description: optional specify whether the ConfigMap
                            or its keys must be defined
                          type: boolean
                      type: object
                      x-kubernetes-map-type: atomic
                    csi:
                      description: CSI object to use to populate the volume
                      properties:
                        driver:
                          description: |-
                            driver is the name of the CSI driver that handles this volume.
                            Consult with your admin for the correct name as registered in the cluster.
                          type: string
                        fsType:
                          description: |-
                            fsType to mount. Ex. "ext4", "xfs", "ntfs".
                            If not provided, the empty value is passed to the associated CSI driver
                            which will determine the default filesystem to apply.
                          type: string
                        nodePublishSecretRef:
                          description: |-
                            nodePublishSecretRef is a reference to the secret object containing
                            sensitive information to pass to the CSI driver to complete the CSI
                            NodePublishVolume and NodeUnpublishVolume calls.
                            This field is optional, and  may be empty if no secret is required. If the
                            secret object contains more than one secret, all secret references are passed.
                          properties:
                            name:
                              description: |-
                                Name of the referent.
                                More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
                                TODO: Add other useful fields. apiVersion, kind, uid?
                              type: string
                          type: object
                          x-kubernetes-map-type: atomic
                        readOnly:
                          description: |-
                            readOnly specifies a read-only configuration for the volume.
                            Defaults to false (read/write).
                          type: boolean
                        volumeAttributes:
                          additionalProperties:
                            type: string
                          description: |-
                            volumeAttributes stores driver-specific properties that are passed to the CSI
                            driver. Consult your driver's documentation for supported values.
                          type: object
                      required:
                      - driver
                      type: object
                    emptyDir:
                      description: EmptyDir to use to populate the volume
                      properties:
                        medium:
                          description: |-
                            medium represents what type of storage medium should back this directory.
                            The default is "" which means to use the node's default medium.
                            Must be an empty string (default) or Memory.
                            More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir
                          type: string
                        sizeLimit:
                          anyOf:
                          - type: integer
                          - type: string
                          description: |-
                            sizeLimit is the total amount of local storage required for this EmptyDir volume.
                            The size limit is also applicable for memory medium.
                            The maximum usage on memory medium EmptyDir would be the minimum value between
                            the SizeLimit specified here and the sum of memory limits of all containers in a pod.
                            The default is nil which means that the limit is undefined.
                            More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir
                          pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
                          x-kubernetes-int-or-string: true
                      type: object
                    name:
                      description: Name to use for the volume. Required.
                      type: string
                    path:
                      description: Path in the container to mount the volume at.
                        Required.
                      type: string
                    restartPods:
                      description: Whether to restart the pods on content change
                      type: boolean
                    secret:
                      description: Secret to use populate the volume
                      properties:
                        defaultMode:
                          description: |-
                            defaultMode is Optional: mode bits used to set permissions on created files by default.
                            Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511.
                            YAML accepts both octal and decimal values, JSON requires decimal values
                            for mode bits. Defaults to 0644.
                            Directories within the path are not affected by this setting.
                            This might be in conflict with other options that affect the file
                            mode, like fsGroup, and the result can be other mode bits set.
                          format: int32
                          type: integer
                        items:
                          description: |-
                            items If unspecified, each key-value pair in the Data field of the referenced
                            Secret will be projected into the volume as a file whose name is the
                            key and content is the value. If specified, the listed keys will be
                            projected into the specified paths, and unlisted keys will not be
                            present. If a key is specified which is not present in the Secret,
                            the volume setup will error unless it is marked optional. Paths must be
                            relative and may not contain the '..' path or start with '..'.
                          items:
                            description: Maps a string key to a path within a
                              volume.
                            properties:
                              key:
                                description: key is the key to project.
                                type: string
                              mode:
                                description: |-
                                  mode is Optional: mode bits used to set permissions on this file.
                                  Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511.
                                  YAML accepts both octal and decimal values, JSON requires decimal values for mode bits.
                                  If not specified, the volume defaultMode will be used.
                                  This might be in conflict with other options that affect the file
                                  mode, like fsGroup, and the result can be other mode bits set.
                                format: int32
                                type: integer
                              path:
                                description: |-
                                  path is the relative path of the file to map the key to.
                                  May not be an absolute path.
                                  May not contain the path element '..'.
                                  May not start with the string '..'.
                                type: string
                            required:
                            - key
                            - path
                            type: object
                          type: array
                        optional:
                          description: optional field specify whether the Secret
                            or its keys must be defined
                          type: boolean
                        secretName:
                          description: |-
                            secretName is the name of the secret in the pod's namespace to use.
                            More info: https://kubernetes.io/docs/concepts/storage/volumes#secret
                          type: string
                      type: object
                    subPath:
                      description: SubPath of the referenced volume to mount.
                      type: string
                  required:
                  - name
                  - path
                  type: object
                type: array

Alternatives

I can not see a way to access the log files with the existing CRD. Everything is either tmpdir with MEMORY as the type (hardcoded), secret+config mappings or the dedicated volumes for kafka storage.

Additional context

My usecase is that I have setup up spec.kafka.logging.loggers to have

log4j.logger.kafka.authorizer.logger: "DEBUG, authorizerAppender"

Where authorizerAppender is a custom logger that saves audit logs to a file. I do not want audit logs to be shown in stdout for security purposes.

Using additionalVolumes I could map the log directory to a volume shared by a OpenTelemetryCollector which uses a sidecar to read and send the audit logs securely to my audit storage solution.

[cthtrifork]

There is a lot of operators which support additionalVolumes. I would probably do something similar to OpenSearch:
https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/charts/opensearch-cluster/templates/opensearch-cluster-cr.yaml#L69 if accepted.

[scholzj]

This should be closed as a duplicate of #3693.

In general, this requires a proposal (https://github.com/strimzi/proposals). If you are interested in contributing this feature, that would be welcome - writing the proposal would be the first step. I think the preferred solution from the Strimzi side is to use the existing Pod and Container templates and configure the additional volumes / volume mount points there.

[cthtrifork]

@scholzj I don't mind closing this. I also agree on adding to the existing pod and container templates, but do you want a proposal before i prepare a PR?

[scholzj]

Yes, this is a major enhancement with API impact. So this should have a proposal written and approved first.

Of course, if you want to write a prototype implementation to accompany the proposal (to for example make sure it is feasible or when you want to run the Strimzi system tests), it is fine to have a Draft PR for it and we can for example run the System Tests on it. But it will not be merged/fully reviewed before the proposal is approved (and having the prototype PR does not guarantee in any way that the proposal will be accepted of course).

(But just to be clear, you are not required to have any kind of prototype implementation to open the proposal. From my own experience, sometimes it helps to write the proposal as you better understand what is needed. But for other proposals it is not really much helpful. So it depends a lot.)

[cthtrifork]

Thanks for the feedback. I will close this and work on a proposal + a draft PR.