Improve monitoring and feedback for errors during workspace cluster selection #8486

geropl · 2022-02-28T12:57:49Z

Description

This PR:

re-tries cluster-selection for a particular workspace instance for up to 2 times, with a 2s interval
introduces two new metrics: 'gitpod_server_instance_starts_success_total{retries}' and 'gitpod_server_instance_starts_failed_total'
adds a Graph for it on "Meta Overview" [sic]
adds an alert InstanceStartFailures (incl. runbook)

Related Issue(s)

Fixes #7829
Runbook PR: https://github.com/gitpod-io/runbooks/pull/327

How to test

start a workspace on this branch
kubectl port-forward deployment grafana 3000
Open that port and navigate to "Team WebApp / Meta Overview"

Release Notes

improve robustness of startWorkspace
improve feedback for errors during cluster selection
improve monitoring for cluster selection errors

Documentation

/werft with-helm
/werft with-observability
/werft no-preview

codecov · 2022-02-28T13:11:57Z

Codecov Report

Merging #8486 (572d8b4) into main (90fe82a) will decrease coverage by 17.74%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             main    #8486       +/-   ##
===========================================
- Coverage   28.92%   11.17%   -17.75%     
===========================================
  Files         150       18      -132     
  Lines       23031      993    -22038     
===========================================
- Hits         6662      111     -6551     
+ Misses      15769      880    -14889     
+ Partials      600        2      -598

Flag	Coverage Δ
components-blobserve-app	`?`
components-blobserve-lib	`?`
components-common-go-lib	`?`
components-content-service-api-go-lib	`?`
components-content-service-app	`?`
components-content-service-lib	`?`
components-ee-agent-smith-app	`?`
components-ee-agent-smith-lib	`?`
components-ee-kedge-app	`?`
components-gitpod-cli-app	`11.17% <ø> (ø)`
components-image-builder-api-go-lib	`?`
components-image-builder-bob-app	`?`
components-image-builder-bob-runc-facade	`?`
components-image-builder-mk3-app	`?`
components-installation-telemetry-app	`?`
components-local-app-app-darwin-amd64	`?`
components-local-app-app-darwin-arm64	`?`
components-local-app-app-linux-amd64	`?`
components-local-app-app-linux-arm64	`?`
components-local-app-app-windows-386	`?`
components-local-app-app-windows-amd64	`?`
components-local-app-app-windows-arm64	`?`
components-openvsx-proxy-app	`?`
components-openvsx-proxy-lib	`?`
components-registry-facade-app	`?`
components-registry-facade-lib	`?`
components-service-waiter-app	`?`
components-supervisor-app	`?`
components-workspacekit-app	`?`
components-ws-daemon-api-go-lib	`?`
components-ws-daemon-app	`?`
components-ws-daemon-lib	`?`
components-ws-daemon-nsinsider-app	`?`
components-ws-manager-api-go-lib	`?`
components-ws-manager-app	`?`
components-ws-proxy-app	`?`
dev-loadgen-app	`?`
dev-poolkeeper-app	`?`
install-installer-raw-app	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
install/installer/pkg/common/objects.go
components/ws-daemon/pkg/content/initializer.go
components/ws-daemon/pkg/cpulimit/dispatch.go
components/ws-manager/pkg/manager/annotations.go
components/supervisor/pkg/supervisor/git.go
components/common-go/log/redact.go
components/ee/kedge/pkg/kedge/collector.go
components/ws-manager/pkg/clock/clock.go
components/supervisor/pkg/ports/exposed-ports.go
components/blobserve/pkg/blobserve/refstore.go
... and 122 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90fe82a...572d8b4. Read the comment docs.

geropl · 2022-02-28T13:17:14Z

/werft run

👍 started the job as gitpod-build-gpl-7829-clu-sel.1

…nstance - 'gitpod_server_instance_starts_success_total' - 'gitpod_server_instance_starts_failed_total'

… Rate"

geropl · 2022-03-01T16:48:37Z

Note: the build is read because of some bug with the "observability" deployment process, but Gitpod and Grafana do work.
We'll have to override the PR check once this is approved.

components/server/src/workspace/workspace-starter.ts

AlexTugarev · 2022-03-07T11:01:23Z

components/server/src/workspace/workspace-starter.ts

            if (!resp) {
+                increaseFailedInstanceStartCounter("clusterSelectionFailed");


just in case tryStartOnCluster throws unhandled, should it make sense to increment as well?

Thx, excellent point! 💯 Will add

@AlexTugarev Here it is: 572d8b4

geropl · 2022-03-07T16:16:45Z

/werft run

👍 started the job as gitpod-build-gpl-7829-clu-sel.7

Hope to get a green build now with changed flags...

geropl · 2022-03-07T16:55:43Z

/werft no-preview=true

👎 unknown command: no-preview=true
Use /werft help to list the available commands

geropl · 2022-03-07T16:56:45Z

/werft run

👍 started the job as gitpod-build-gpl-7829-clu-sel.8

roboquat added release-note do-not-merge/work-in-progress size/L labels Feb 28, 2022

roboquat added size/XL and removed size/L labels Mar 1, 2022

geropl added 3 commits March 1, 2022 15:38

[server] Retry to start a WorkspaceInstance 2 times, with 2s break

640f2e5

[server] Introduce metrics around clusterSelection in startWorkspaceI…

40ed6e4

…nstance - 'gitpod_server_instance_starts_success_total' - 'gitpod_server_instance_starts_failed_total'

[ops] WebApp Overview: Add graph for "Instance Starts Success/Failure…

bca8c16

… Rate"

geropl force-pushed the gpl/7829-clu-sel branch from be1e70d to 52b706f Compare March 1, 2022 15:40

[ops] Add alert 'InstanceStartFailures'

23bdf16

geropl force-pushed the gpl/7829-clu-sel branch from 52b706f to 23bdf16 Compare March 1, 2022 16:38

geropl marked this pull request as ready for review March 1, 2022 16:41

geropl requested a review from a team March 1, 2022 16:41

roboquat removed the do-not-merge/work-in-progress label Mar 1, 2022

github-actions bot added the team: webapp Issue belongs to the WebApp team label Mar 1, 2022

AlexTugarev reviewed Mar 7, 2022

View reviewed changes

components/server/src/workspace/workspace-starter.ts Show resolved Hide resolved

AlexTugarev reviewed Mar 7, 2022

View reviewed changes

[server] FailedInstanceStart metrics: add case "startOnClusterFailed"

572d8b4

AlexTugarev approved these changes Mar 7, 2022

View reviewed changes

roboquat merged commit d030750 into main Mar 7, 2022

roboquat deleted the gpl/7829-clu-sel branch March 7, 2022 17:00

roboquat added deployed: webapp Meta team change is running in production deployed Change is completely running in production labels Mar 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve monitoring and feedback for errors during workspace cluster selection #8486

Improve monitoring and feedback for errors during workspace cluster selection #8486

geropl commented Feb 28, 2022 •

edited

Loading

codecov bot commented Feb 28, 2022 •

edited

Loading

geropl commented Feb 28, 2022 •

edited by werft-gitpod-dev-com bot

Loading

geropl commented Mar 1, 2022

AlexTugarev Mar 7, 2022

geropl Mar 7, 2022

geropl Mar 7, 2022

geropl commented Mar 7, 2022 •

edited by werft-gitpod-dev-com bot

Loading

geropl commented Mar 7, 2022 •

edited by werft-gitpod-dev-com bot

Loading

geropl commented Mar 7, 2022 •

edited by werft-gitpod-dev-com bot

Loading

		if (!resp) {
		increaseFailedInstanceStartCounter("clusterSelectionFailed");

Improve monitoring and feedback for errors during workspace cluster selection #8486

Improve monitoring and feedback for errors during workspace cluster selection #8486

Conversation

geropl commented Feb 28, 2022 • edited Loading

Description

Related Issue(s)

How to test

Release Notes

Documentation

codecov bot commented Feb 28, 2022 • edited Loading

Codecov Report

geropl commented Feb 28, 2022 • edited by werft-gitpod-dev-com bot Loading

geropl commented Mar 1, 2022

AlexTugarev Mar 7, 2022

Choose a reason for hiding this comment

geropl Mar 7, 2022

Choose a reason for hiding this comment

geropl Mar 7, 2022

Choose a reason for hiding this comment

geropl commented Mar 7, 2022 • edited by werft-gitpod-dev-com bot Loading

geropl commented Mar 7, 2022 • edited by werft-gitpod-dev-com bot Loading

geropl commented Mar 7, 2022 • edited by werft-gitpod-dev-com bot Loading

geropl commented Feb 28, 2022 •

edited

Loading

codecov bot commented Feb 28, 2022 •

edited

Loading

geropl commented Feb 28, 2022 •

edited by werft-gitpod-dev-com bot

Loading

geropl commented Mar 7, 2022 •

edited by werft-gitpod-dev-com bot

Loading

geropl commented Mar 7, 2022 •

edited by werft-gitpod-dev-com bot

Loading

geropl commented Mar 7, 2022 •

edited by werft-gitpod-dev-com bot

Loading