gateway2: Add readiness and liveness probes to the kube gloo gateway pod #10229

davidjumani · 2024-11-14T15:39:48Z

Description

This update allows users to enable and add custom readiness and liveness probes to the Kubernetes Gloo Gateway.
This allows zero downtime when upgrading gloo or when rolling out new pods

API changes

The following helm changes have been made :

kubeGateway.gatewayParameters.glooGateway.podTemplate.terminationGracePeriodSeconds to specify the terminationGracePeriodSeconds.
kubeGateway.gatewayParameters.glooGateway.podTemplate.gracefulShutdown to configure the graceful shutdown config for the envoy container.
kubeGateway.gatewayParameters.glooGateway.podTemplate.customLivenessProbe to specify a custom liveness probe for the envoy container.
kubeGateway.gatewayParameters.glooGateway.podTemplate.livenessProbeEnabled to enable the liveness probe. If the customLivenessProbe is not specified, no default liveness probe is set.
kubeGateway.gatewayParameters.glooGateway.podTemplate.customReadinessProbe to specify a custom readiness probe for the envoy container.
kubeGateway.gatewayParameters.glooGateway.podTemplate.probes to enable the readiness probe. If the customReadinessProbe is not specified, a default readiness probe is set. No default liveness probe is set

The following fields have been added to the GatewayParameters.Kube.PodTemplate field :

GracefulShutdown
TerminationGracePeriodSeconds
ReadinessProbe
LivenessProbe

Code changes

Updated the deployer to read the new values from the gateway params and generate the appropriate helm values for the gloo gateway pod deployment

CI changes

Added the zero downtime config to all Kubernetes tests to ensure this does not break an existing feature

Docs changes

TODO in a followup with @solo-io/solo-docs

Context

https://github.com/solo-io/solo-projects/issues/7084

Interesting decisions

The helm values were created to mimic the existing values to setup the probes and zero downtime config for edge classic. Ref: https://github.com/solo-io/solo-projects/issues/7084

Testing steps

Added unit tests

Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works

…eness-probe

solo-changelog-bot · 2024-11-14T16:31:00Z

Issues linked to changelog:
https://github.com/solo-io/solo-projects/issues/7084

…eness-probe

install/helm/gloo/templates/43-gatewayparameters.yaml

nfuden · 2024-11-15T17:39:00Z

changelog/v1.18.0-beta35/add-readiness-liveness-probe.yaml

+      - `kubeGateway.gatewayParameters.glooGateway.podTemplate.terminationGracePeriodSeconds` to specify the terminationGracePeriodSeconds.
+      - `kubeGateway.gatewayParameters.glooGateway.podTemplate.gracefulShutdown` to configure the graceful shutdown config for the envoy container.
+      - `kubeGateway.gatewayParameters.glooGateway.podTemplate.customLivenessProbe` to specify a custom liveness probe for the envoy container.
+      - `kubeGateway.gatewayParameters.glooGateway.podTemplate.livenessProbeEnabled` to enable the liveness probe. If the customLivenessProbe is not specified, no liveness probe is set.


Suggested change

- `kubeGateway.gatewayParameters.glooGateway.podTemplate.livenessProbeEnabled` to enable the liveness probe. If the customLivenessProbe is not specified, no liveness probe is set.

should we also remove the option in other places from the helm values?

jenshu · 2024-11-15T21:13:03Z

install/helm/gloo/generate/values.go

@@ -339,9 +339,19 @@ type GatewayParameters struct {
 	Stats           *GatewayParamsStatsConfig  `json:"stats,omitempty" desc:"Config used to manage the stats endpoints exposed on the deployed proxies"`
 	AIExtension     *GatewayParamsAIExtension  `json:"aiExtension,omitempty" desc:"Config used to manage the Gloo Gateway AI extension."`
 	FloatingUserId  *bool                      `json:"floatingUserId,omitempty" desc:"If true, allows the cluster to dynamically assign a user ID for the processes running in the container. Default is false."`
+	PodTemplate     *GatewayParamsPodTemplate  `json:"podTemplate,omitempty"`


nit: add description

jenshu · 2024-11-15T21:17:51Z

install/helm/gloo/generate/values.go

+type GatewayParamsPodTemplate struct {
+	GracefulShutdown              *GracefulShutdownSpec `json:"gracefulShutdown,omitempty"`
+	TerminationGracePeriodSeconds *int                  `json:"terminationGracePeriodSeconds,omitempty" desc:"Time in seconds to wait for the pod to terminate gracefully. See [kubernetes docs](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#istio-lifecycle) for more info."`
+	Probes                        *bool                 `json:"probes,omitempty" desc:"Set to true to enable a readiness probe (default is false). Then, you can also enable a liveness probe."`


since this is a bool flag can we call it enableProbes?

also, a little confused about the description:

Set to true to enable a readiness probe (default is false). Then, you can also enable a liveness probe.

should there be 2 different flags, enableLivenessProbe and enableReadinessProbe? or does this one flag enable both?

I want to mimic the edge classic gateway proxy settings where probes controls the readiness probe.
Since there's no default liveness probe, a custom one can be set with the customLivenessProbe without the need for a bool to enable it

gloo/install/helm/gloo/generate/values.go

Line 677 in 8f097f5

Probes *bool `json:"probes,omitempty" desc:"Set to true to enable a readiness probe (default is false). Then, you can also enable a liveness probe."`

jenshu · 2024-11-15T21:18:36Z

install/helm/gloo/generate/values.go

 	// TODO(npolshak): Add support for GlooMtls
 }

+// GatewayProxyPodTemplate contains the Helm API available to configure the PodTemplate on the gateway Deployment
+type GatewayParamsPodTemplate struct {
+	GracefulShutdown              *GracefulShutdownSpec `json:"gracefulShutdown,omitempty"`


nit: add descriptions to all fields

jenshu · 2024-11-15T21:23:30Z

install/helm/gloo/generate/values.go

+	GracefulShutdown              *GracefulShutdownSpec `json:"gracefulShutdown,omitempty"`
+	TerminationGracePeriodSeconds *int                  `json:"terminationGracePeriodSeconds,omitempty" desc:"Time in seconds to wait for the pod to terminate gracefully. See [kubernetes docs](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#istio-lifecycle) for more info."`
+	Probes                        *bool                 `json:"probes,omitempty" desc:"Set to true to enable a readiness probe (default is false). Then, you can also enable a liveness probe."`
+	CustomReadinessProbe          *corev1.Probe         `json:"customReadinessProbe,omitempty"`


in the description here, we can say that if enableProbes=true and no customReadinessProbe is provided, then a default one will be created

jenshu · 2024-11-15T21:25:55Z

projects/gateway2/deployer/merge.go

+	}
+
+	// Don't merge the command string as that can break the entire probe
+	dst.Command = overrideSlices(dst.Command, src.Command)


would be good to add a test to merge_test.go that tests this override

projects/gateway2/helm/gloo-gateway/templates/gateway/proxy-deployment.yaml

jenshu · 2024-11-15T21:31:27Z

projects/gateway2/helm/gloo-gateway/templates/gateway/proxy-deployment.yaml

+              command:
+              - /bin/sh
+              - -c
+              - wget --post-data "" -O /dev/null 127.0.0.1:19000/healthcheck/fail; sleep {{ $gateway.gracefulShutdown.sleepTimeSeconds | default "10" }}


will this work ok if sleepTimeSeconds is nil?

Yes since a default is set

…ployment.yaml Co-authored-by: Jenny Shu <28537278+jenshu@users.noreply.github.com>

jenshu · 2024-11-18T19:56:57Z

known flake #10365

…ss-liveness-probe

davidjumani added 9 commits November 13, 2024 11:48

update apis

be73faa

helm changes

7ee1992

add tests

33ebc01

add probes to tests

debf04a

add changelog

fd6152e

Merge branch 'main' of github.com:solo-io/gloo into add-readiness-liv…

2864b66

…eness-probe

codegen

74de62b

add merge

43266cd

add kubernetes tests

4aa0e38

davidjumani requested a review from a team as a code owner November 14, 2024 15:39

github-actions bot added keep pr updated work in progress labels Nov 14, 2024

nfuden changed the title ~~gateway2: Add readiness and liveness probes to the kube gloo gateway pod~~ [Migrated] add stopwatches for translator and envoy syncer Nov 14, 2024

nfuden closed this Nov 14, 2024

davidjumani changed the title ~~[Migrated] add stopwatches for translator and envoy syncer~~ [Migrated] gateway2: Add readiness and liveness probes to the kube gloo gateway pod Nov 14, 2024

davidjumani reopened this Nov 14, 2024

davidjumani changed the title ~~[Migrated] gateway2: Add readiness and liveness probes to the kube gloo gateway pod~~ gateway2: Add readiness and liveness probes to the kube gloo gateway pod Nov 14, 2024

davidjumani added 5 commits November 14, 2024 12:42

fixes

88a361b

Merge branch 'main' of github.com:solo-io/gloo into add-readiness-liv…

9318790

…eness-probe

fix codegen

15d209a

increase duration

6bbe51a

Merge branch 'main' into add-readiness-liveness-probe

9dbe84b

nfuden reviewed Nov 15, 2024

View reviewed changes

install/helm/gloo/templates/43-gatewayparameters.yaml Outdated Show resolved Hide resolved

davidjumani added 3 commits November 15, 2024 11:19

add zero downtime profile

6b5438b

Merge branch 'main' into add-readiness-liveness-probe

8297c90

remove default liveness probe

45c2f97

nfuden reviewed Nov 15, 2024

View reviewed changes

davidjumani added 2 commits November 15, 2024 12:50

Merge branch 'main' into add-readiness-liveness-probe

7b2601b

more clenaup

fe410b0

davidjumani added 4 commits November 15, 2024 14:13

cleanup

9ae80f1

update changelog

4c8c16a

update changelog again

93ba4f6

Merge branch 'main' into add-readiness-liveness-probe

3656fd7

nfuden approved these changes Nov 15, 2024

View reviewed changes

Merge branch 'main' into add-readiness-liveness-probe

f1452ab

jenshu reviewed Nov 15, 2024

View reviewed changes

davidjumani and others added 4 commits November 18, 2024 06:42

Update projects/gateway2/helm/gloo-gateway/templates/gateway/proxy-de…

e818e11

…ployment.yaml Co-authored-by: Jenny Shu <28537278+jenshu@users.noreply.github.com>

add docs

8077c25

Merge branch 'main' into add-readiness-liveness-probe

08837aa

Merge branch 'main' into add-readiness-liveness-probe

a21606c

Nadine2016 added documentation Improvements or additions to documentation and removed documentation Improvements or additions to documentation labels Nov 18, 2024

jenshu approved these changes Nov 18, 2024

View reviewed changes

Merge branch 'main' of ssh://github.com/solo-io/gloo into add-readine…

f56fd9c

…ss-liveness-probe

nfuden approved these changes Nov 18, 2024

View reviewed changes

nfuden merged commit b31586c into main Nov 18, 2024
17 of 18 checks passed

nfuden deleted the add-readiness-liveness-probe branch November 18, 2024 23:02

davidjumani mentioned this pull request Nov 27, 2024

livenessProbe is not added to gatewayparameters kgateway-dev/kgateway#10384

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gateway2: Add readiness and liveness probes to the kube gloo gateway pod #10229

gateway2: Add readiness and liveness probes to the kube gloo gateway pod #10229

davidjumani commented Nov 14, 2024 •

edited

Loading

solo-changelog-bot bot commented Nov 14, 2024

nfuden Nov 15, 2024

jenshu Nov 15, 2024

jenshu Nov 15, 2024

davidjumani Nov 18, 2024 •

edited

Loading

jenshu Nov 15, 2024

jenshu Nov 15, 2024

jenshu Nov 15, 2024

jenshu Nov 15, 2024

davidjumani Nov 18, 2024

jenshu commented Nov 18, 2024

gateway2: Add readiness and liveness probes to the kube gloo gateway pod #10229

gateway2: Add readiness and liveness probes to the kube gloo gateway pod #10229

Conversation

davidjumani commented Nov 14, 2024 • edited Loading

Description

API changes

Code changes

CI changes

Docs changes

Context

Interesting decisions

Testing steps

Checklist:

solo-changelog-bot bot commented Nov 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidjumani Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jenshu commented Nov 18, 2024

davidjumani commented Nov 14, 2024 •

edited

Loading

davidjumani Nov 18, 2024 •

edited

Loading