feat: add grpc health, deprecate old health #861

toddbaert · 2023-08-24T19:09:55Z

adds a standard gRPC health check
moves HTTP health checks off the metrics port, to the app port (we previously hadn't documented the health-checks are controlled by the metrics port config)
deprecates old health checks on metrics port (logs warning)

fixes: #831

closed in favor of: #863

netlify · 2023-08-24T19:10:00Z

✅ Deploy Preview for polite-licorice-3db33c ready!

Name	Link
🔨 Latest commit	`465add4`
🔍 Latest deploy log	https://app.netlify.com/sites/polite-licorice-3db33c/deploys/64e8da925bbf6a0008ca9aef
😎 Deploy Preview	https://deploy-preview-861--polite-licorice-3db33c.netlify.app/concepts/architecture
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

toddbaert · 2023-08-24T19:12:22Z

core/pkg/service/flag-evaluation/connect_service.go

@@ -129,16 +130,28 @@ func (s *ConnectService) setupServer(svcConf service.Configuration) (net.Listene

 path, handler := schemaConnectV1.NewServiceHandler(fes, append(svcConf.Options, marshalOpts)...)
 mux.Handle(path, handler)
+ mux.Handle(grpchealth.NewHandler(grpchealth.NewStaticChecker(


Standard implementation, recommended by buf/conntect.

You can test this locally with grpcurl (below command assumes you've downloaded the health proto to ~/Downloads):

grpcurl -import-path '/home/{your-user}/Downloads/' -proto health.proto -plaintext localhost:8013 grpc.health.v1.Health/Check

toddbaert · 2023-08-24T19:15:13Z

core/pkg/service/flag-evaluation/connect_service.go


 s.serverMtx.Lock()
 s.server = &http.Server{
 ReadHeaderTimeout: time.Second,
- Handler: handler,
+ Handler: mux,


The mux created above was actually unused - instead we just used the one handler originally.

toddbaert · 2023-08-24T19:15:56Z

core/pkg/service/flag-evaluation/connect_service.go

@@ -206,8 +219,10 @@ func (s *ConnectService) startMetricsServer(svcConf service.Configuration) error
 s.metricsServer.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 switch r.URL.Path {


In my new implementation, I've used mux instead of this kind of switch statement.

codecov · 2023-08-24T19:28:52Z

Codecov Report

Merging #861 (c26e569) into main (5b42486) will decrease coverage by 0.35%.
Report is 3 commits behind head on main.
The diff coverage is 27.77%.

❗ Current head c26e569 differs from pull request most recent head 465add4. Consider uploading reports for the commit 465add4 to get more accurate results

@@            Coverage Diff             @@
##             main     #861      +/-   ##
==========================================
- Coverage   72.67%   72.33%   -0.35%     
==========================================
  Files          27       27              
  Lines        2759     2772      +13     
==========================================
  Hits         2005     2005              
- Misses        667      678      +11     
- Partials       87       89       +2

Files Changed	Coverage Δ
...ore/pkg/service/flag-evaluation/connect_service.go	`62.87% <27.77%> (-5.31%)`	⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

toddbaert · 2023-08-24T19:30:21Z

core/pkg/service/flag-evaluation/connect_service.go

@@ -11,6 +11,7 @@ import (
 "time"

 schemaConnectV1 "buf.build/gen/go/open-feature/flagd/bufbuild/connect-go/schema/v1/schemav1connect"
+ "connectrpc.com/grpchealth"


this package is a connect-compatible implementation of a gRPC healthcheck.

* adds a standard gRPC health check * moves HTTP health checks off the metrics port, to the app port * deprecates old health checks on metrics port Signed-off-by: Todd Baert <todd.baert@dynatrace.com>

Signed-off-by: Todd Baert <todd.baert@dynatrace.com>

thisthat

We might consider adding a test using https://github.com/grpc-ecosystem/grpc-health-probe to check the gRPC probes are correct. Currently, the ZD tests use the HTTP probes.

thisthat · 2023-08-25T06:52:32Z

config/deployments/flagd/deployment.yaml

@@ -29,13 +29,13 @@ spec:
 readinessProbe:
 httpGet:
 path: /readyz
- port: 8014


please, adapt also the ZD tests manifest to use the new port :)

@thisthat flagd-proxy uses the sync package from core, NOT the flag-evaluation package. For this reason, the health-checks of flagd-proxy are not impacted by this change (which may or may not be good, since things will be a bit inconsistent.

Problem is, I'm not 100% sure I can add HTTP 1.1 health-check support to the sync package since it doesn't use connect.

toddbaert · 2023-08-25T14:03:25Z

We might consider adding a test using https://github.com/grpc-ecosystem/grpc-health-probe to check the gRPC probes are correct. Currently, the ZD tests use the HTTP probes.

great idea!

Kavindu-Dodan

overal looks good 🙌

approving and hope there will be improvements proposed by this comment - #861 (review)

Signed-off-by: Todd Baert <todd.baert@dynatrace.com>

toddbaert · 2023-08-25T19:33:59Z

Closed in favor of #863

toddbaert requested a review from a team as a code owner August 24, 2023 19:09

toddbaert commented Aug 24, 2023

View reviewed changes

toddbaert requested review from beeme1mr, thisthat, bacherfl and Kavindu-Dodan August 24, 2023 19:25

toddbaert force-pushed the grpc-healthcheck branch 2 times, most recently from 5374cac to 8414c80 Compare August 24, 2023 19:28

toddbaert commented Aug 24, 2023

View reviewed changes

toddbaert mentioned this pull request Aug 24, 2023

[FEATURE] gRPC Health Check must be implemented in case flagd is accessed via AWS ALB #831

Closed

toddbaert added 3 commits August 24, 2023 16:21

feat: add grpc health, deprecate old health

93a9069

* adds a standard gRPC health check * moves HTTP health checks off the metrics port, to the app port * deprecates old health checks on metrics port Signed-off-by: Todd Baert <todd.baert@dynatrace.com>

fixup: lint

4648c26

Signed-off-by: Todd Baert <todd.baert@dynatrace.com>

fixup: minor refactor

a00ae5a

Signed-off-by: Todd Baert <todd.baert@dynatrace.com>

toddbaert force-pushed the grpc-healthcheck branch from 572e216 to a00ae5a Compare August 24, 2023 20:21

thisthat reviewed Aug 25, 2023

View reviewed changes

Kavindu-Dodan approved these changes Aug 25, 2023

View reviewed changes

toddbaert added 3 commits August 25, 2023 11:04

fixup: more compiler options

9c2fa3e

Signed-off-by: Todd Baert <todd.baert@dynatrace.com>

fixup: conflictPolicy=ignore in prod

c26e569

Signed-off-by: Todd Baert <todd.baert@dynatrace.com>

fixup: more doc

465add4

Signed-off-by: Todd Baert <todd.baert@dynatrace.com>

toddbaert marked this pull request as draft August 25, 2023 17:11

toddbaert closed this Aug 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add grpc health, deprecate old health #861

feat: add grpc health, deprecate old health #861

toddbaert commented Aug 24, 2023 •

edited

Loading

netlify bot commented Aug 24, 2023 •

edited

Loading

toddbaert Aug 24, 2023 •

edited

Loading

toddbaert Aug 24, 2023 •

edited

Loading

toddbaert Aug 24, 2023

codecov bot commented Aug 24, 2023 •

edited

Loading

toddbaert Aug 24, 2023

thisthat left a comment

thisthat Aug 25, 2023

toddbaert Aug 25, 2023 •

edited

Loading

toddbaert commented Aug 25, 2023

Kavindu-Dodan left a comment

toddbaert commented Aug 25, 2023

		@@ -206,8 +219,10 @@ func (s *ConnectService) startMetricsServer(svcConf service.Configuration) error
		s.metricsServer.Handler = http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		switch r.URL.Path {

feat: add grpc health, deprecate old health #861

feat: add grpc health, deprecate old health #861

Conversation

toddbaert commented Aug 24, 2023 • edited Loading

netlify bot commented Aug 24, 2023 • edited Loading

✅ Deploy Preview for polite-licorice-3db33c ready!

toddbaert Aug 24, 2023 • edited Loading

Choose a reason for hiding this comment

toddbaert Aug 24, 2023 • edited Loading

Choose a reason for hiding this comment

toddbaert Aug 24, 2023

Choose a reason for hiding this comment

codecov bot commented Aug 24, 2023 • edited Loading

Codecov Report

toddbaert Aug 24, 2023

Choose a reason for hiding this comment

thisthat left a comment

Choose a reason for hiding this comment

thisthat Aug 25, 2023

Choose a reason for hiding this comment

toddbaert Aug 25, 2023 • edited Loading

Choose a reason for hiding this comment

toddbaert commented Aug 25, 2023

Kavindu-Dodan left a comment

Choose a reason for hiding this comment

toddbaert commented Aug 25, 2023

toddbaert commented Aug 24, 2023 •

edited

Loading

netlify bot commented Aug 24, 2023 •

edited

Loading

toddbaert Aug 24, 2023 •

edited

Loading

toddbaert Aug 24, 2023 •

edited

Loading

codecov bot commented Aug 24, 2023 •

edited

Loading

toddbaert Aug 25, 2023 •

edited

Loading