Applier hangs if schedule request buffer is full

[nightkr]

Current and expected behavior

See #925 for more details.

The applier currently stops trying to process new scheduling requests while trying to write back the result of a reconciliation. This leads to a deadlock where the buffer is never emptied because we're busy trying to fill it.

Thanks to @moustafab for reporting the issue and contributing a workaround.

Possible solution

1. Explicitly try to empty the buffer during writeback
2. Move the scheduler into a separate Tokio task
3. Remove the size limit on the schedule request buffer (current workaround taken in fix applier hangs which can happen with many watched objects #925)

Additional context

No response

Environment

This is a runtime issue, independent of K8s version

Configuration and features

kube 0.73.1

Affected crates

kube-runtime

Would you like to work on fixing this bug?

yes