querier, receiver, sidecar, store: Add gRPC health check endpoints

[kakkoyun]

This PR adds gRPC Health handlers to gRPC servers.
Attempts to fix #1980.

Signed-off-by: Kemal Akkoyun kakkoyun@gmail.com

• I added CHANGELOG entry for this change.
• Change is not relevant to the end user.

Changes

• Adds gRPC Health handlers to gRPC servers.

Verification

Tested using https://github.com/grpc-ecosystem/grpc-health-probe

1. MINIO_ENABLED=1 ./scripts/quickstart.sh
2. Use health probe to check

➜ grpc-health-probe -addr=localhost:9999 -connect-timeout 250ms -rpc-timeout 100ms
timeout: failed to connect service "localhost:9999" within 250ms

➜ grpc-health-probe -addr=localhost:10905 -connect-timeout 250ms -rpc-timeout 100ms
status: SERVING

➜ grpc-health-probe -addr=localhost:10903 -connect-timeout 250ms -rpc-timeout 100ms
status: SERVING

➜ grpc-health-probe -addr=localhost:10913 -connect-timeout 250ms -rpc-timeout 100ms
status: SERVING

[kakkoyun]

Another flaky test run. Can someone re-run it?

[kakkoyun]

@GiedriusS I did another iteration on gRPC health checks. Rather than integrating with prober, I have added grpcHealthServer separately. Not all the components serve gRPC, so they don't use prober. I'm not very sure about API, I'm considering adding additional Resume and Pause methods to grpcserver.Server and not expose health.Server. WDYT?

@squat Also I'd be glad if you can also have a look at this.

[bwplotka]

Awesome, looking good. Some suggestions commented on below.

[kakkoyun]

@bwplotka I did another iteration on it. How does it look? Any suggestions?

[bwplotka]

Nice I like it 👍

Have one suggestion though.

[bwplotka]

Hm, wonder if it's consistent and clean to have log item added to gRPC methods but metrics added to HTTP methods. It's kind of weird. I would either:

• add single logging/metric prober (so prober responsible only to log/metric) and always combine all to the others
• add log line and metric to combine prober and use it everywhere. WDYT?

[kakkoyun]

I'll find a way to neatly combine observability layer for prober.

[bwplotka]

Perfecto 👌 Clean and nice, LGTM

[poolski]

Thank you! 😍

[bwplotka]

Please give us feedback if that works for you 💪🤞

…

On Thu, 30 Jan 2020, 11:53 Kyrill, ***@***.***> wrote: Thank you! 😍 — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#2008?email_source=notifications&email_token=ABVA3O2ZDCMRZQAKZVQS6XLRAK5T3A5CNFSM4KHSFV72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKKW2HY#issuecomment-580218143>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVA3O76GB347OC2LN7NZNDRAK5T3ANCNFSM4KHSFV7Q> .

[poolski]

It works perfectly! Thank you 👍