Ensure system recovers quickly from failures or drift in state #326
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
thisisnotashwin
requested review from
lkysow,
a team and
ishustava
and removed request for
a team
lkysow
reviewed
Sep 17, 2020
thisisnotashwin
force-pushed
the
webhook-cert-recover
branch
from
292abf5 to
05edd2a
Compare
thisisnotashwin
force-pushed
the
webhook-cert-recover
branch
from
05edd2a to
8765f43
Compare
- helm upgrades will cause the caBundle to get reset on the mutating webhooks. By "reconciling" the state of the system every second, we ensure the drift in this state has a minimal impact on the uptime of the system. it will now verify that the certificates as well as the CA bundle are "correct" every second and update them if they arent.
thisisnotashwin
force-pushed
the
webhook-cert-recover
branch
from
8765f43 to
913f97a
Compare
lkysow
approved these changes
Sep 17, 2020
1 task
ishustava
reviewed
Sep 21, 2020
Co-authored-by: Iryna Shustava <ishustava@users.noreply.github.com>
ishustava
approved these changes
Sep 22, 2020
ndhanushkodi
pushed a commit
to ndhanushkodi/consul-k8s
that referenced
this pull request
Jul 9, 2021
Add PodSecurityPolicies for server-acl-init
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes proposed in this PR:
helm upgrades will cause the caBundle to get reset on the mutating webhooks. By "reconciling" the state of the system every second, we ensure the drift in this state has a minimal impact on the uptime of the system. it will now verify that the certificates as well as the CA bundle are "correct" every second and update them if they arent.
this will also ensure any edits made to the secret or the webhook configuration, which could lead to downtime in the system, are recovered from within a second.
How I've tested this PR:
How I expect reviewers to test this PR:
kubectl editthe secret with the certificate and/or the MWC for the webhook certs. The system should recover within a few seconds and the webhook and the webhook configuration should be able to communicate with each other successfully.Image for testing:
ashwinvenkatesh/consul-k8s:webhook-certs@sha256:25b85fd6ccbe7e141cd1541264c40cb15f28c1ba815a4df58c6a3949b583e91fChecklist: