Investigate shutdown delay option

[philwebb]

The webinar presented by @ryanjbaxter has an interesting bit of configuration to delay shutdown using a preStop command. This article has some interesting background.

We might be able to offer a similar feature out-of-the-box in Boot and configure it automatically.

[ryanjbaxter]

@philwebb actually pulled that from the boot docs 🤫

[philwebb]

I guess I missed those doc updates. They must have happened when I was away so I'm glad I watched the webinar!

[ckoutsouridis]

i think this would be a great feature given that endpoints propagation and pod deletion happens asynchronously. + not all base images have a shell in order to run sleep in the preStop.

the 2.3 gracefull shutdown support is great, but i believe in k8s we will have to delay the shutdown still, in order to to make sure that new requests will not end up in an endpoint from a pod that is already gone.

[ttddyy]

We have an operational scenario that directly open a shell on the pod and restart the application. This allows faster restart of the app since it doesn't involve k8s pod lifecycle.

In such case, k8s does not know the application is restarting and it still send traffic to the pod during shutdown.
So, in our graceful shutdown logic(in addition to the one from boot 2.3), we put a sleep for the duration of readiness check frequency, to make sure k8s readiness check picks up that the app is about to restarting and stop routing the traffic before actual shutdown happens.

If Spring Boot provides sleep(delay) on its graceful shutdown logic, I can remove our sleep logic from our library, which is great.

[siwyd]

This would be a handy addition, as right now I always lose some metrics when the application shuts down because Prometheus hasn't had a chance to scrape the last 'meaningful' metrics before it's killed, causing a big (fake) dip in request rate for instance. If I could keep the application around for 30s while traffic is already being diverted to other instances, this would allow me to scrape those last metrics.