envoy prometheus endpoint fails promlint

[vjsamuel]

Title: envoy prometheus endpoint fails promlint

Description:

The admin endpoint of envoy when queried as a prometheus endpoint fails promlint. Promlint is a sanity check tool offered by Prometheus to check the correctness of a prometheus endpoint.

[optional Relevant Links:]

About promlint: Promlint is a metrics sanity checker exposed as part of promtool. As per their documentation:

Pass Prometheus metrics over stdin to lint them for consistency and correctness.
examples:
$ cat metrics.prom | promtool check metrics
$ curl -s http://localhost:9090/metrics | promtool check metrics
`)

The error reported by promlint is as follows:

curl -s https://gist.githubusercontent.com/vjsamuel/5898d02f8039d8d1bae4d8642182d1c3/raw/deff0c16f746bb2ab7a959c4d5ed9856d17c425d/gistfile1.txt | ./promtool check metrics
error while linting: text format parsing error in line 9: second TYPE line for metric name "envoy_listener_http_downstream_rq_xx", or TYPE reported after samples

Can this be fixed please so that the prometheus endpoint abides to the standard?

[dio]

This is interesting, while I'm not familiar with the code, should we fix this in tags level (how we tag things) or just how we present this as what Prometheus wants? For the latter I can see that we can have: envoy_listener_http_downstream_rq_5xx as metric_name for a tag with envoy_response_code_class="5", hence no duplication in TYPEs.

[vjsamuel]

The TYPE should be represented only once in the endpoint per unique metric name. Hence while looping through the counters/gauges there should be a check in place to do it only once. There might be other lint based errors once this is fixed. The entire endpoint should be reviewed for compliance hopefully with this issue.

[dio]

In this case, should we have an "integration" to promlint on designing the test scenario?

[vjsamuel]

@pitiwari awesome work! thanks for this!