-
Notifications
You must be signed in to change notification settings - Fork 115
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Check for successful deploys of StatefulSets with an OnDelete update strategy #926
Conversation
0ba7a26
to
7920e4d
Compare
03a7722
to
c93e811
Compare
@@ -10,6 +10,7 @@ class Pod < KubernetesResource | |||
) | |||
|
|||
attr_accessor :stream_logs | |||
attr_reader :definition |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Allows StatefulSet resource to access each Pod's definition["metadata"]["labels"]["controller-revision-hash"]
value, a new addition to how we check for successful deploys. Is there any issue with making this value accessible?
d6dd4f9
to
1a325e6
Compare
As a configuration the default value Overall the PR looks good to me. Glad to see some good testing 👏 |
Good call-out @null-sleep - thinking about this again, I'm going to remove Context for PR reviewers: The reason it made sense for Deployments to have EDIT: Asked for additional thoughts on Slack and will make this update when we receive a go-ahead for this change. |
dc02c33
to
11ef9c2
Compare
Based on the above comment, this PR now only supports |
29e055c
to
f7dad22
Compare
This comment was marked as resolved.
This comment was marked as resolved.
f7dad22
to
0d9a51e
Compare
What are you trying to accomplish with this PR?
🏗️ ISSUE: #912
When checking for a successful deploy, if a StatefulSet's updateStrategy is set to
OnDelete
(default isRollingUpdate
), krane does not check if the rollout is successful and instead prints a warning message ("can't monitor the update since it doesn't occur until deletion") and moves on.We want to enable krane to accurately detect successful deploys for StatefulSets with an
updateStrategy
ofOnDelete
.How is this accomplished?
Update krane's
deploy_succeeded?
check to include StatefulSets withupdateStrategy: OnDelete
.updateStrategy: RollingUpdate
and checks that the following holds true:onDelete
'scurrentRevision
value is not updated reliably. Also,currentReplicas
relies oncurrentRevision
, so neither of these values can be used.updatedReplicas
instead. IfupdatedReplicas
doesn't matchreplicas
, then pods have not been updated to the latest revision.readyReplicas
can be used to check for their actual health.As confirmed over Slack, we're going with Katrina's suggestion of putting this behaviour behind an annotation to allow control on a per-resource basis using a style Krane users are already accustomed to. We're extending the
krane.shopify.io/required-rollout annotation
to be compatible with StatefulSet resources.Currently, for
Deployments
with annotations, the state of thereplicaSet
is checked to confirm a successful deploy. BecauseStatefulSet
resources don't usereplicaSets
to create pods, we suggest only usingfull
as an option for this annotation, with this definition:full
: deployment is successful when all pods are readyThe logic we propose in this PR is as follows:
What should reviewers focus on?
Testing
updateRevision
(usingk -n [namespace] --context [context] get sts [namespace] -o yaml | tail -n 10
) andcontroller-revision-hash
(usingk -n [namespace] --context [context] get pod [pod name] -o yaml | grep -i revision
) match after an update to our StatefulSet config (before the pod is marked as ready)