feat(kds): introduce zone health checks #7821

michaelbeaumont · 2023-09-22T13:56:36Z

Explanation

Zone:

Send health check periodically

Global:

When we get a health check, mark it in the corresponding ZoneInsight
Introduce new component ZoneWatch that listens for ZoneOpenedStream and marks the (tenantID, zone) for watching
Periodically check all watched (tenantID, zone) and if the time of last health check in ZoneInsight is too late, send ZoneWentOffline
All opened streams also listen for ZoneWentOffline events and the handler returns if said event is received, ending the stream

We store the info in ZoneInsight because:
All instances need to potentially kill streams but not every instance will receive a health check from connected zones

Tests

The need for time.Sleep in the tests comes about because it happens asynchronously that:

ZoneWatch subscribes to ZoneOpenedStream events in Start
in reality, this is guaranteed to happen before ZoneOpenedStream events are sent by the fact that we only send them in response to new gRPC streams being opened
ZoneOpenedStream is witnessed by ZoneWatch
the test adds a time.Sleep because we only want to update the health check time once and in particular after the zone starts being watched and then check that it's disconnected
in reality, a zone will continually send its health check ping so it eventually will be updated after the initial seen last time.

Todo

configurability

Checklist prior to review

Link to relevant issue as well as docs and UI issues --
This will not break child repos: it doesn't hardcode values (.e.g "kumahq" as a image registry) and it will work on Windows, system specific functions like syscall.Mkfifo have equivalent implementation on the other OS --
Tests (Unit test, E2E tests, manual test on universal and k8s) --
Do you need to update UPGRADE.md? --
Does it need to be backported according to the backporting policy? (this GH action will add "backport" label based on these file globs, if you want to prevent it from adding the "backport" label use no-backport-autolabel label) --
Do you need to explicitly set a > Changelog: entry here or add a ci/ label to run fewer/more tests?

pkg/kds/mux/zone_watch.go

pkg/core/features.go

pkg/kds/mux/client.go

pkg/kds/mux/zone_watch.go

pkg/kds/global/components.go

pkg/kds/mux/zone_watch.go

pkg/kds/service/server.go

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

jakubdyszkiewicz

I think we could introduce tests at least for ZoneWatch. Any ideas for E2E test?

pkg/kds/service/server.go

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

michaelbeaumont · 2023-10-10T11:41:40Z

I think we could introduce tests at least for ZoneWatch. Any ideas for E2E test?

I think e2e will be somewhat difficult because of the nature of isolating failure to the app level health check. The always possible, but ugly option is to introduce an option like "don't send health check pings" to the CP but IMO we should avoid this artificial, public test interface

@slonka mentioned potentially sending SIGSTOP to make the process stop sending pings but keep TCP connections open. The problem that I see is that HTTP2 PING ACKs also stop, which will trigger the gRPC keep alive to fail.

jakubdyszkiewicz · 2023-10-10T11:58:13Z

but IMO we should avoid this artificial, public test interface

I agree.

We could put Kong in the middle (we have test infra to deploy KIC) which apparently does not proxy PING/GOAWAY and then do SIGKILL on Zone CP. I wonder if this would work.

slonka · 2023-10-10T12:07:03Z

We could put Kong in the middle (we have test infra to deploy KIC) which apparently does not proxy PING/GOAWAY and then do SIGKILL on Zone CP. I wonder if this would work.

It should work 👍

michaelbeaumont · 2023-10-10T12:17:24Z

OK, if we have some infrastructure for starting up kong gateway it becomes significantly less work

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

…thcheck Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

… timing out Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

This reverts commit baa72b6.

…8017) Revert "feat(kuma-cp): introduce zone health checks (#7821)" This reverts commit baa72b6. --- Signed-off-by: slonka <slonka@users.noreply.github.com>

This reverts commit 386ab53. Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

* feat(kuma-cp): reintroduce zone health checks (#7821) This reverts commit 386ab53. * feat: don't error in zone if global CP doesn't support health check Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

michaelbeaumont force-pushed the feat/do-zone-healthcheck branch 3 times, most recently from d2211af to cb39b5e Compare September 29, 2023 16:06

michaelbeaumont commented Sep 29, 2023

View reviewed changes

pkg/kds/mux/zone_watch.go Outdated Show resolved Hide resolved

michaelbeaumont force-pushed the feat/do-zone-healthcheck branch 2 times, most recently from deaad86 to cbab23a Compare October 5, 2023 08:09

michaelbeaumont changed the title ~~Feat/do zone healthcheck~~ feat: introduce zone health checks Oct 5, 2023

lobkovilya reviewed Oct 5, 2023

View reviewed changes

michaelbeaumont force-pushed the feat/do-zone-healthcheck branch 2 times, most recently from 8cdb4c6 to 0f817ea Compare October 5, 2023 11:31

feat(kuma-cp): call zone HealthCheck periodically from zone

1dbb3a8

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

michaelbeaumont force-pushed the feat/do-zone-healthcheck branch 2 times, most recently from 81e8212 to 4ed9872 Compare October 5, 2023 14:24

michaelbeaumont requested a review from lobkovilya October 5, 2023 14:41

michaelbeaumont added 2 commits October 5, 2023 19:43

feat(kuma-cp): kill zone connections when health check fails

9230b77

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

feat(kuma-cp): only watch zone health if feature present

3c67e80

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

michaelbeaumont force-pushed the feat/do-zone-healthcheck branch from 4ed9872 to 4aa9128 Compare October 5, 2023 17:44

feat(kuma-cp): add config for global zone health check logic

09d2ec0

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

michaelbeaumont force-pushed the feat/do-zone-healthcheck branch 2 times, most recently from 7b6ae86 to 848d7aa Compare October 6, 2023 12:53

michaelbeaumont marked this pull request as ready for review October 6, 2023 13:07

michaelbeaumont requested a review from a team as a code owner October 6, 2023 13:07

michaelbeaumont requested review from bartsmykla and jakubdyszkiewicz and removed request for a team October 6, 2023 13:07

michaelbeaumont changed the title ~~feat: introduce zone health checks~~ feat(kuma-cp): introduce zone health checks Oct 6, 2023

michaelbeaumont added 2 commits October 9, 2023 09:46

feat(kuma-cp): use health check interval from global

81bc083

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

refactor: remove unnecessary goroutine

0e154d4

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

michaelbeaumont force-pushed the feat/do-zone-healthcheck branch from 848d7aa to 8693e31 Compare October 9, 2023 07:49

fix: properly validate zone health check config

b3a52d0

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

michaelbeaumont requested review from jakubdyszkiewicz and lobkovilya October 9, 2023 12:49

lobkovilya approved these changes Oct 10, 2023

View reviewed changes

jakubdyszkiewicz approved these changes Oct 10, 2023

View reviewed changes

pkg/kds/service/server.go Outdated Show resolved Hide resolved

chore: remove defunct struct

0cf5af0

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

refactor: only pass necessary part of config

2f333e0

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

michaelbeaumont force-pushed the feat/do-zone-healthcheck branch from 8087fbe to 8cae7a8 Compare October 10, 2023 13:26

michaelbeaumont added 2 commits October 10, 2023 15:29

test(ZoneWatch): add suite

0934669

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

Merge remote-tracking branch 'upstream/master' into feat/do-zone-heal…

fd543c8

…thcheck Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

michaelbeaumont force-pushed the feat/do-zone-healthcheck branch from 8cae7a8 to fd543c8 Compare October 10, 2023 13:32

michaelbeaumont added 4 commits October 10, 2023 15:38

test(ZoneWatch): add additional time.Sleep to ensure conditions

ff4ad60

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

test(ZoneWatch): remove FlakeAttempts

8da0f5a

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

feat: treat opening the stream as a healthcheck

0356948

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

test(ZoneWatch): add test that updating ZoneInsight keeps a zone from…

5749df9

… timing out Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

jakubdyszkiewicz approved these changes Oct 10, 2023

View reviewed changes

fix(kds): check feature before killing because of zone health check

61ed009

Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

michaelbeaumont merged commit baa72b6 into kumahq:master Oct 10, 2023
5 checks passed

michaelbeaumont deleted the feat/do-zone-healthcheck branch October 10, 2023 17:08

michaelbeaumont added the ci/run-full-matrix PR: Runs all possible e2e test combination (expensive use carefully) label Oct 11, 2023

slonka added a commit that referenced this pull request Oct 11, 2023

Revert "feat(kuma-cp): introduce zone health checks (#7821)"

7f5895d

This reverts commit baa72b6.

slonka mentioned this pull request Oct 11, 2023

chore(revert): revert "feat(kuma-cp): introduce zone health checks" #8017

Merged

michaelbeaumont added a commit to michaelbeaumont/kuma that referenced this pull request Oct 12, 2023

feat(kuma-cp): reintroduce zone health checks (kumahq#7821)

6cfb315

This reverts commit 386ab53. Signed-off-by: Mike Beaumont <mjboamail@gmail.com>

lahabana changed the title ~~feat(kuma-cp): introduce zone health checks~~ feat(kds): introduce zone health checks Nov 3, 2023

BrewTestBot mentioned this pull request Nov 15, 2023

kumactl 2.5.0 Homebrew/homebrew-core#154429

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(kds): introduce zone health checks #7821

feat(kds): introduce zone health checks #7821

michaelbeaumont commented Sep 22, 2023 •

edited

Loading

jakubdyszkiewicz left a comment

michaelbeaumont commented Oct 10, 2023

jakubdyszkiewicz commented Oct 10, 2023

slonka commented Oct 10, 2023

michaelbeaumont commented Oct 10, 2023

feat(kds): introduce zone health checks #7821

feat(kds): introduce zone health checks #7821

Conversation

michaelbeaumont commented Sep 22, 2023 • edited Loading

Explanation

Tests

Todo

Checklist prior to review

jakubdyszkiewicz left a comment

Choose a reason for hiding this comment

michaelbeaumont commented Oct 10, 2023

jakubdyszkiewicz commented Oct 10, 2023

slonka commented Oct 10, 2023

michaelbeaumont commented Oct 10, 2023

michaelbeaumont commented Sep 22, 2023 •

edited

Loading