Strimzi is rolling the pods after disk resize with LiveResize enabled

[maciej-tatarski]

We have live resize enabled but sometimes strimzi is still rolling the pods after volume expansion. It is happening because it is checking
FileSystemResizePending condition on a pvc.

FileSystemResizePending is showing on pvc even when live resize is enabled. I assumed this is bug on k8s side and raised this ticket: kubernetes-csi/external-resizer#414, but got a response that it is just temporary state and should be expected.

Should some new logic be introduced in strimzi to cater for this scenario during volume expansion?

[scholzj]

I'm not sure you can introduce new logic - or at least what would it be. I think Kubernetes does not handle this well because as an application, you currently don't know if the resizing happens live or if the pod needs to be rolled. As you say, on one platform, the FileSystemResizePending might be just a temporary state that means nothing. On others, this is treated as please roll the pod. So if we don't react to it, we might not roll the Pod without a reason for you. But we break the resizing for someone else who will now have to roll the pod manually etc.

So what would the new logic be? I'm sure you can add some flag to the operator configuration to indicate that your platform supports live resize and the operator might use it to ignore the FileSystemResizePending. But you cannot really assume it automatically as far as I know and there is no way to find this dynamically.

[maciej-tatarski]

I was actually thinking about a global flag or kafka object prop (like kafka.spec.kafka.storage.volumes[*].liveResize: true) to explicitly indicate that live resize is enabled on the disks

[scholzj]

I think this would likely behave based on environment and not based on disk? So maybe a global flag is better?

I personally would not have a problem with that if you were interested in contributing such feature (or if you wanted to open an enhancement for someone else to do it). If we would want to change the default (i.e. to not roll it by default), I think having a short proposal for it might be great as that would change the current behavior. Otherwise my personal take is that it would not need a proposal.

[maciej-tatarski]

There can be a scenario where you use mix of disks that support live resize and those that don't on a single platform. Wouldn't it be more flexible to provide it per disk? I can work on it when I have some bandwidth.

[scholzj]

Well, adding the flag to the storage configuration should have a proposal I guess as it changes the API. The implementation will also be much harder. And then finally the user experience when configuring it would be worse as you would need to configure it for each volume in each cluster and not just once.

I guess there might be cases like that where you have a mix of things. But in my experience, environments where different storage types are mixed are rare. So environments where different storage types are used and they have different level of support for live resizing would be even more rare.

I don't think I would reject it if you would come up with such a proposal. But I wonder if it is worth the effort and the user experience.

Also - while having two ways how to do the same thing is something I would try to avoid - I would rather start with the simple way to learn we need the more elaborate way than the other way around.

Do you have such an environment where it would be needed per-volume?

[maciej-tatarski]

No, we are using same volumes in our deployment. I think you have a good point that simple flag might be a better and simpler approach.