[observability] Add SLI numbers to the Workspace Success Criteria Dashboard

[atduarte]

Description

Currently, only the timeseries is shown, but in order to evaluate whether we are following our SLOs the dashboard should present a number.

See this for an example.

Related Issue(s)

N/A

How to test

Load this json in Grafana by:

1. Creating a new dashboard
2. Opening Dashboard Settings > JSON Model
3. Changing the JSON model except for the uid and version

Or see https://grafana.gitpod.io/d/Qljo7br7z/success-criteria-temporary-10382

Release Notes

NONE

Documentation

[kylos101]

@atduarte I'm not able to test this in the preview environment or prod (by loading the JSON). I must be doing something wrong! Can you share a Loom showing how you're loading the JSON? 🙏

@ArthurSens I tried testing this in the preview environment, but, I think cluster not having an actual value makes many of the dashboards not function correctly (including this one). Would it make sense to populate the cluster value for preview environments? I figure that would make testing dashboard changes easier (cause we do do, assuming we have the data, in preview environments).

[atduarte]

@kylos101 updated how to test to be more clear. Easier way is just visiting https://grafana.gitpod.io/d/Qljo7br7z/success-criteria-temporary-10382

[kylos101]

Beautiful

[ArthurSens]

@ArthurSens I tried testing this in the preview environment, but, I think cluster not having an actual value makes many of the dashboards not function correctly (including this one). Would it make sense to populate the cluster value for preview environments? I figure that would make testing dashboard changes easier (cause we do do, assuming we have the data, in preview environments).

The lack of a cluster label should not be a problem, because the filter says "anything that is different from ephemeral or meta" and empty fulfills this regex. You can confirm this by removing the cluster filter from the preview environment, it will keep showing "NO DATA".

The problem here is that your query measures workspace stop failures, while not a single failure happened. You need to generate this data somehow 😅. Maybe by doing some load tests?

[ArthurSens]

I agree that performing load tests should be easier for preview environments... Maybe a feature request for our newest previewctl CLI? cc @gitpod-io/platform 👀

[kylos101]

@ArthurSens I started and stopped workspaces many times - no data showed - even on the overview dashboard. I should have seen that at least 1 workspace was in the Running phase or seen an entry on the startup time heatmap, I think.

[ArthurSens]

@ArthurSens I started and stopped workspaces many times - no data showed - even on the overview dashboard. I should have seen that at least 1 workspace was in the Running phase or seen an entry on the startup time heatmap, I think.

But your query depends on a workspace to fail, not only to stop 😬

Stopping workspaces in a regular manner won't produce the data that you want

[kylos101]

@ArthurSens ignore the workspace success criteria dashboard for a moment in this PR. I mean in general, I was looking at the overview dashboard, to see the heatmap for start times, and count of workspaces by phase - and I didn't see any related data on the overview dashboard. If I start workspaces, should the overview dashboard show me stuff?

[ArthurSens]

Hmmm that is super awkward... Gitpod overview has always worked for me on previews, let me double check

[ArthurSens]

Yep, I could reproduce this problem on the preview from main branch. The problem is not lack of producing metrics, but something changed in Gitpod that makes Prometheus unable to scrape metrics from ws-manager. Not only ws-manager, but also some other components.

This problem is also happening in production/staging... not really a problem with preview environments, but with Gitpod itself 😬

[kylos101]

Now this makes sense given our conversation from earlier, thank you so much, @ArthurSens ! 🙏