Metrics based on telemetry events generated by sparrow

[janciesla8818]

This PR adds new metrics based on sparrow telemetry events. The changes include:

• Adding histogram, sum and counter metric for HTTP requests
• Count of worker pools
• Count of workers per worker pool
• Total count of workers in all worker pools

Last three metrics are gathered using :wpool.stats to get the information from worker pool.

Telemetry poller is added to the project to collect periodic metrics. Counters of pools and workers are collected as events generated during telemetry poller events. The period for gathering metrics is 5 seconds which is the default value for telemetry poller.

[michalwski]

Thanks for the PR! It's good to have metrics showing how much time was spent in the HTTP/2 worker. Metrics showing worker pool statistics are also valuable.

On a side note, I think it'd be worth checking if we use the newset versions of the Telemetry releated libraries.

[michalwski]

Does this metric show time spent only on sending the request and waiting for the response?

[janciesla8818]

This metric shows only the time it takes to handle and send request. It only opens the stream and sends the request. It does not measure response time. In general, when this timer is small it it possible that we are getting errors with opening the stream.

[michalwski]

Thanks for checking this. I think it'd be good to add this info to the doc.

[janciesla8818]

Ok, will add that info.

[janciesla8818]

@michalwski, this part is already added to the doc..

[michalwski]

Hmm, I wonder how did you come to the conclusion that it measures only the time it takes to handle and send the request. If I'm correct this metric measures the time of this function in sparrow: https://github.com/esl/sparrow/blob/master/lib/sparrow/h2_worker.ex#L352-L430

Inside the function, we are waiting for the response and emit additional events depending on the response.

Also, I think the name of the metric starts with sparrow_h2_worker instead of sparrow_h_worker.

[michalwski]

All right, we discussed this offline with @janciesla8818. I was misled by the post_result var name and the case checking if this is a successful response or not. In fact, this var gives us information if the request was sent successfully or not. The response to the request arrives asynchronously (HTTP/2). Thanks to that we can send many requests using the same h2_worker and connection.

When it comes to the metric name, the Prometheus.Core lib removes all numbers from the metric name.