This repository has been archived by the owner on Jun 20, 2024. It is now read-only.
kubernetes: Morph the livenessProbe into readinessProbe #3421
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
See #3417 wherein we argue that the liveness probe should be removed |
dlespiau
force-pushed
the
2018-09-28-moar-initialDelaySeconds
branch
from
e878abd to
64a8b77
Compare
The liveness probe has the not-so-great side effect of killing the weave pods if anything goes wrong, making debugging really difficult. Not only that, we also bumped into the livenessProbe killing pods before they had a chance to start (see below for the full story). The readinessProbe is only used for Kubernetes Services: no traffic will be routed to non-Ready pods when using the Service IP. We can use to to signal the user something in wrong with Weave Net while not having Kubernetes delete the pods behind our backs. I removed the ininitialDelaySeconds field as Kurbernetes won't kill the pod if not ready. Notes from a debugging session: Some nodes may have a vast amount of iptables rules, making each iptables command take a few seconds (~5-10s). At startup, when the weave container creates its chains, it runs quite a few iptables commands and the total time of execution was > 30s. It's important to note that, at this point, the weave process hasn't started as those iptables commands are run by an initial script. Kubernetes was trying to check for the container liveness, was failing, and then proceeded to kill the weave pods before they had a chance to fully start.
dlespiau
force-pushed
the
2018-09-28-moar-initialDelaySeconds
branch
from
64a8b77 to
8d44f49
Compare
dlespiau
changed the title
| httpGet: | ||
| host: 127.0.0.1 | ||
| path: /status | ||
| port: 6784 | ||
| initialDelaySeconds: 30 |
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
|
We should update Launch Generator too. |
|
I have a PR updating launch-generator accordingly but didn't mention it here as it's not part of the open source project. |
bboreham
approved these changes
Oct 16, 2018
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The liveness probe has the not-so-great side effect of killing the weave pods
if anything goes wrong, making debugging really difficult.
Not only that, we also bumped into the livenessProbe killing pods before they
had a chance to start (see below for the full story).
The readinessProbe is only used for Kubernetes Services: no traffic will be
routed to non-Ready pods when using the Service IP. We can use to to signal the
user something in wrong with Weave Net while not having Kubernetes delete the
pods behind our backs.
I removed the ininitialDelaySeconds field as Kurbernetes won't kill the pod if
not ready.
Notes from a debugging session:
Some nodes may have a vast amount of iptables rules, making each iptables
command take a few seconds (~5-10s). At startup, when the weave container
creates its chains, it runs quite a few iptables commands and the total time of
execution was > 30s.
It's important to note that, at this point, the weave process hasn't started as
those iptables commands are run by an initial script.
Kubernetes was trying to check for the container liveness, was failing, and
then proceeded to kill the weave pods before they had a chance to fully start.
Fixes: #3417