Support mounting a volume to the pod

[morozov]

Is your feature request related to a problem? Please describe.
When analyzing out-of-memory problems happening in JVM, it helps to collect and look at the heap dump (see DBZ-1104 for the reference). In the Kubernetes environment, a dump should be stored on a medium external to the process that outlives the process in case of a failure, i.e. a volume (see the article). Currently, it's not possible to mount an arbitrary volume to a pod managed by Strimzi.

Describe the solution you'd like
Allow to mount a volume to a pod described by PodTemplate.

Describe alternatives you've considered
Dump the heap inside the container (e.g. /tmp). It's not clear how to extract the dump from a failed pod and whether it's a reliable enough approach.

Additional context
While there are existing issues that could be solved using a similar approach (#1401, #1439), I'm filing a new one since it represents different use case that may justify adding the feature.

[scholzj]

Thanks for opening the issue. I can understand the requirement. For the record, I think there are some challenges for example around RWO versus RWX storage etc. (just a note for whoever gets to this one day).

[scholzj]

Triaged on 31.3.2022: This might cover multiple different issues:

• Storing logs on disks (Support volume for pod running logs #1401 and [Enhancement] Support storing logs on persistent volumes instead of always sending them only to STDOUT #2216)
• using Kafka Connect connectors to write / read data to / from disk ([Enhancement] Kafka Connect volume mounts #2571 )

This will be complicated to implement => for example with regards to handling RWO versus RWX storage etc. So this needs to have a proposal.

Note: The #2216 had some unfinished work associated with it - that might be something to reuse ... or not.

[scholzj]

The #5277 issue asks for support for Secret Store CSI driver and using its volumes. Since it is based on a CSI driver as regular storage, perhaps it can be fixed with this issue as well?

[Migueljfs]

Hello, was any decision or implementation reached?

We currently use vault to store our connection config and would like to either inject it via vault-injector our use Secrets Store CSI Driver

[scholzj]

Maybe starting a separate discussion to explain your usecase and what do you mean with store our connection config would be useful: https://github.com/strimzi/strimzi-kafka-operator/discussions

[Migueljfs]

Apologies if I wasn't clear, I meant that we store secrets like database passwords, hostnames, etc in Vault. I would like for KafkaConnector to use these secrets for its config, either via vault-injector or by mounting the secrets (which seems to be the prefered option from what I've read here).

I think my issue is the same as this: #5277

[scholzj]

I don't think that makes it really celar how it fits into this. That is why I think having a separate discussion might be useful to clarify the use-case and requirement then doing so here.

[Migueljfs]

I'd rather not pollute this board with duplicates. My issue is literally the same as #5277 word for word.

This issue #3693 seems like the main issue for this feature request, so I was wondering if any progress was made

[jakubmalek]

Hi,
I found myself in a need for this feature as well but the issue seem to be open for quite a while now with many references.

IMHO the easiest solution would be to simply extend the PodTemplate spec with option to mount persistent volumes.
Although the volumes can be used for multiple purposes, dumping the heap, storing the logs etc. I see it more as custom feature, not tied to any specific application feature.
For this reason I think the provisioning of the PVC's should be out of the scope of Strimzi operator, and the appropriate access mode must be configured by the user depending on target use-case.
This way the implementation would be more or less only about translating the CRD into corresponding k8s-deployment or k8s-prod manifest.

So, I'm wondering how to move this topic forward. Do you think I can start with implementation PR already for the described proposition, or do you think it needs a discussion entry / proposal PR first ?

[scholzj]

I do not think it is as easy as you suggest. Without provisioning the PVCs / volumes, this can easily block the operator in many new ways. Also, you need to add the volumes, you need to add the volume mounts, you need to somehow add different volumes to different pods etc. So as decided in the triage - this needs a proposal.

As for the use case: yes, compelling use cases are needed for things like this. It is not only about the initial implementation. Features like this will have a relatively big ongoing maintenance cost. So they need to be justified and not added just because.

[jakubmalek]

I understand your concerns, but I'm thinking about it more as "you should know what you're doing" kind feature, where simply wants to customize the effective Kubernetes manifest for your workload.

[scholzj]

That might be a reasonable view. But which of the points I listed does it really cover? I guess it might cover this one: Without provisioning the PVCs / volumes, this can easily block the operator in many new ways. => but the others are in my opinion still valid.

[audirline]

Hello, do we have any news about this? Will this be implemented?
Thanks :)
#9128 is my use case

[mattssll]

also interested, I'm running into issues like this when spinning a connector in my kafka-connect cluster managed by strimzi:
Suppressed: java.io.IOException: No space left on device

[scholzj]

For tracking purposes: @cthtrifork plans to have a look at this (discussed in #10065).

[cthtrifork]

Proposal: strimzi/proposals#120

Please help shape it