-
Notifications
You must be signed in to change notification settings - Fork 333
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(kuma-cp): introduce zone health checks (#7821)
## Explanation Client: * Send health check periodically Server: * When we get a health check, mark it in the corresponding `ZoneInsight` * Introduce new component `ZoneWatch` that listens for `ZoneOpenedStream` and marks the `(tenantID, zone)` for watching * Periodically check all watched `(tenantID, zone)` and if the time of last health check in `ZoneInsight` is too late, send `ZoneWentOffline` * All opened streams also listen for `ZoneWentOffline` events and the handler returns if said event is received, ending the stream We store the info in `ZoneInsight` because: All instances need to potentially kill streams but not every instance will receive a health check from connected zones ## Tests The need for `time.Sleep` in the tests comes about because it happens asynchronously that: 1) `ZoneWatch` subscribes to `ZoneOpenedStream` events in `Start` in reality, this is guaranteed to happen before ZoneOpenedStream events are sent by the fact that we only send them in response to new gRPC streams being opened 2) `ZoneOpenedStream` is witnessed by `ZoneWatch` the test adds a `time.Sleep` because we only want to update the health check time once and in particular _after_ the zone starts being watched and then check that it's disconnected in reality, a zone will continually send its health check ping so it _eventually_ will be updated after the initial seen last time. Signed-off-by: Mike Beaumont <mjboamail@gmail.com>
- Loading branch information
1 parent
4c067a8
commit baa72b6
Showing
13 changed files
with
720 additions
and
168 deletions.
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.