Fix watch crash api unreachable #568
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
diconico07
requested review from
bfjelds,
kate-goldenring,
jiria,
Britel,
romoh and
adithyaj
as code owners
|
/version patch |
kate-goldenring
approved these changes
Apr 4, 2023
Signed-off-by: Nicolas Belouin <nicolas.belouin@suse.com>
According to Kube-rs documentation, when a watch error occurs (e.g because of the kube API being currently unavailable), the `Watcher` still forwards the error, but will retry if the receiver continues to wait for a new event. This commit changes how the `Watcher` is used in order to handle those errors and let the `Watcher` retry. A backoff mechanism is also setup so we don't end up retrying forever. Signed-off-by: Nicolas Belouin <nicolas.belouin@suse.com>
This commit changes the way config are watched and the events handled, this change have, as main purpose, to ensure the agent won't restart if it looses the connection to the kube API. Two mechanisms are employed to do this: - Change the Watcher use to make it re-try in case of error (same change that was done in 846662e for the controller) - Handle the `Restarted` event so that in case of de-sync with API we are still able to recover without restarting (i.e handle creation/modification and deletion of `Configuration resources that occured during the connection loss). The latter required some refactor of the `ConfigMap` to ensure we have all the required information to do delete operation without having the related event (after a `Restart` event only the current state of the resources are available, not their events). This refactor also fixed a yet undetected issue: We were unable to handle two `Configuration` resources with the same name even accross different namespaces. Signed-off-by: Nicolas Belouin <nicolas.belouin@suse.com>
diconico07
force-pushed
the
fix-watch-crash-api-unreachable
branch
from
38901f9
to
ffdfa86
Compare
All calls to `handle_config_add` used a newly created `KubeImpl` object this led to tests using the real Kube client rather than the mocked one. In order to fix this, use `Arc` (as required by `handle_config_add`) at creation time and propagate the `Arc` where needed. Also goes from `impl` to `dyn` wherever needed. Signed-off-by: Nicolas Belouin <nicolas.belouin@suse.com>
Signed-off-by: Nicolas Belouin <nicolas.belouin@suse.com>
diconico07
force-pushed
the
fix-watch-crash-api-unreachable
branch
from
ffdfa86
to
e491948
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
This PR harden the handling of the
Watchermechanisms in order to prevent the controller and the agent to crash when the kubernetes API is unreachable.Hopefully this will fixes these issues:
Fixes #222
Fixes #557
Special notes for your reviewer:
This PR includes an update of
kube-rsandk8s-openapito be able to use more friendly constructs for backoff mechanisms.The biggest part change is a refactor of
ConfigMapneeded to be able to correctly handleRestartedevents to even further eliminate agent's restart risks related to watch of resource.If applicable:
cargo fmt)cargo build)cargo clippy)cargo test)cargo doc)