SIGTERM handling in kube-runtime

[clux]

Looking at how to safely run reflectors/watchers now and the signal handler one I don't have a good answer to anymore. I think that if we get SIGTERM, it's unclear where we put the signal handlers:

• in client during watch calls? -> PRO: will propagate errors up very quickly, CON: unexpected
• injected sentinel into watcher stream -> PRO: expected, CON: might have to wait 300s for watch call (too late on k8s)
• between watch events in runtime objects -> PRO: what we did before, CON: might have to wait on slow streams
• in main: PRO: less dependency bloat, CON: can kill a reconciler mid execution

We did use to do 3 in old Reflector, but it's not really a good solution.
Considering whether this is just something we need to do at the deepest level (even if it only affects the runtime). Maybe feature-flag it?

[clux]

StreamExt::take_until does look like it might work. There's also this stream_cancel crate that builds on top of it.

[clux]

Another possible solution is to wrap the drainer future in an abortable and trigger aborthandle.abort() on SIGTERM with tokio::signal. Though, not sure how deep it goes before it stops. Does it finish a reconcile, or could it stop in the middle of one (since it can spawn its own futures).

The docs on AbortHandle::abort are a bit unclear:

Notifies the Abortable future associated with this handle that it should abort. Note that if the future is currently being polled on another thread, it will not immediately stop running. Instead, it will continue to run until its poll method returns.

[clux]

This was done in #573