[enhancement] More insight into the UX-perspective on workspace startup

[geropl]

Historically we have mostly been looking at backend metrics when it comes to workspace startup(/failure) rates. Also, we have not been very good at separating "user/input errors" (where we're missing sanity checks, better feedback, general UX improvements) and "backend issues" (bugs, but also problems we have with some rare kubernetes/cloud provider condition): Both get spilled into the frontend nearly unfiltered.

This has two implications:

• we're leaving users with little to no clue what went wrong - and if they can do sth to "make it work" for them
• more often then we'd like to we learn about incidents from our users before alerts show them: experience shows that a lot of problems show in the "frontend" way before they pile up in the "backend" where our existing alerts catch them

To get a bit more insight how big this is of an issue, and if and how we can improve on it, we should start reporting and collecting measurements in the frontend on:

• the steps a workspace startup is going through ("pages rendered"?)
• timing for those steps
• if workspace startup fails: record the error message we display to the user

This should not be too complicated at first, but help to get a rough understanding on how much and what is going wrong.
It also opens the doors for further improvements on:

• UX improvements: show nicer error messages, plus hints on how to resolve common problems ("cannot parse Dockerfile", etc.)
• eventually set up alerts on error rates to learn about incidents as early as our users

@csweichel Suggested using some recently added segment integration for this

[svenefftinge]

We want this information in analytics as well, so I suggest we add this information in form of tracked events.

[svenefftinge]

cc @jakobhero

[csweichel]

/schedule

[csweichel]

I'd propose we make use of our new API to track the following events from the frontend:

• frontend-workspace-{{phase}} which fires for every step the user sees
• frontend-workspace-failed if the frontend shows an error during workspace creation/startup
• frontend-workspace-open which is sent by the Gitpod extension in VSCode once it's been loaded

We'd correlate/identify the string of events using a frontend-generated ID, and once available the workspace ID. We'd send the context URL along for the ride.

@jakobhero wdyt?

[geropl]

I'd propose we make use of our new API to track the following events from the frontend:

I propose we frame it slightly different: Instead of reporting frontend-workspace-{{phase}} (which - to me - implies we'd fill it from the wsi events we see in the frontend) I'd argue we are better off using "views rendered" as primary identifier (adding the workspace phase as meta data of course). The difference is the one of perspective: I'm more interested in the sequence (and the timings) of the views that get rendered for the user, to make 100% sure we're reporting "what the user sees", as opposed to "what we think the user should see". The result is a "startWorkspace-trace of views".

[jakobhero]

I mostly agree with your suggestion @csweichel , but I am also liking @geropl 's idea of a views_rendered event that contains the rendered view as properties for a slightly different reason: each event will result in a distinct table in the warehouse, so imo it is a good practice to unite tracking calls that 1) are thematically similar and 2) contain the same properties under the same event name whenever it's feasible. Here, I would suggest to introduce two events:

1. event_rendered with the additional property of event, which either contains the workspace phase or 'vsc_opened'
2. error_rendered with additional error properties such as error_code and error_message
   do you agree?

One more note: the event and property names are determining the table and column names downstream. In order to have coherent naming in the warehouse, we have decided to use snake case for all event and property names.

[csweichel]

status_rendered and error_rendered would work well, too. My only concern with tying this to the views is that the data will change if the views change.

Note also, that much like with the segment data we're producing from bridge, we do not have events here. We have a number of status updates that come in which do not necessarily indicate change. That said, I'd reckon that from a frontend perspective we could use the status to make those "single events", or use the same messageID mechanism we're using on the backend.

[csweichel]

/assign