Investigate Language server metrics

[tsmaeder]

In #13270 we are asked to provide the following metrics:

The % of Language Servers requests that get a successful response
The % of Language Servers requests that get a successful response in under N seconds

However, since we do not control the language servers, I doubt we can provide these numbers directly: typically, language servers are hooked up to the VS Code API via https://github.com/Microsoft/vscode-languageserver-node, which we don't control.

So what I would propose is that we hook the VS Code API to provide those metrics. The purpose of this task is to prototype a solution that can provide that info to a yet-to-be-written API (see Investigate how to report metrics from workspaces #13375). I would expect that it this reporting API would be accessible from the theia back end server or from plugin host containers.

[JPinkney]

My investigations led me to a few things we need to do:

• enable the theia metrics extension in the che-theia generator
• use prometheus auto discovery to scrape the /metrics endpoint every few seconds
• gather all the data in the frontend of plugin-ext and pass that to the backend where the /metrics endpoint lives. Basics are: eclipse-theia/theia@master...JPinkney:plugin-metrics

[tsmaeder]

@JPinkney I'm afraid I don't quite understand your explanations:

1. why do we have to collect the metrics in the front end rather than the back end?
2. What's this with prometheus auto-discovery and why do we have to scrape it instead of pushing metrics to some endpoint?
3. again: once we have the metrics in the back end, how are they pushed to the che monitoring infrastructure?

[skabashnyuk]

What's this with prometheus auto-discovery and why do we have to scrape it instead of pushing metrics to some endpoint?

In general, Prometheus recommend using scrape instead of push. Pushing is used only for short living (less than scrape interval) task.

Another tricky part is authentication/authorization for both model push/pull. I think this is a separate story that can be handled in #13375