KOTS installer 5 minute timeout is too low

[adrienthebo]

Bug description

While observing a customer call we saw that an upgrade of a real-world environment was taking over 4 minutes; this is precariously close to the 5 minute helm upgrade timeout.

Additional debugging pointed the finger at shiftfs-module-loader. This init container compiles the shiftfs kernel module (which is slow) and the version of Kubernetes on some distributions updates daemonsets one node at a time. Combining the slow runtime of shiftfs-module-loader with the linear scaling properties of daemonsets, it's possible for the installation process to be very lengthy on larger clusters.

Steps to reproduce

n/a

Workspace affected

No response

Expected behavior

The helm upgrade process should have a timeout period long enough that it can't accidentally be reached. A value of 10 - 15 minutes is less risky.

Example repository

No response

Anything else?

No response

[mrsimonemms]

Cornelius and I settled on a 5m timeout for the reason that there is no way to stop a deployment from within the KOTS dashboard. If we set it to much longer than that then users will have to manually stop the job if they've started a deployment from the dashboard which is both undocumented and a pretty-awful experience.

What might be a better, more rounded experience would be to add a config option in the KOTS config (suggest the "advanced" section). By default, it would be 5m, otherwise it would be whatever the config value was (we probably want to keep the 1h setting in the case where there is no https-certificates secret as DNS promulgation and LetsEncrypt can take their sweet time).

That way, we can be quite granular for our customers rather than using a sledgehammer to crack a nut.

[corneliusludmann]

@adrienthebo and @mrsimonemms: What's the state of this issue? Is there still work needed or can this be closed?

[mrsimonemms]

Fixed by #14500