-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Possibility of using a more lenient handling of errors in individual samples #1543
Comments
Hey 👋 In the Prometheus ecosystem, a single scrape from a single target is seen as a transaction. Client_golang was designed and implemented with the pull-based model in mind, where all metrics from a registry will be exposed together in an HTTP request. Client_golang follows the same transaction philosophy, which means that we serve all metrics currently in-store, or we don't serve any at all. In the example you've shown, a single MetricFamily I can see how this philosophy might not make sense in a push-based system, but I don't think we can change the current behavior without breaking the pull-based behavior 😬 |
While I agree with what @ArthurSens said, I also think it's ok for client_golang to help with the use case like exposing metrics that process does not control (aka exporter model in Prometheus world) e.g. KSM, cadivsor, node exporter etc. It can happen that underlying data e.g. cgroup metrics (let's imagine they have some help) have inconsistent help description. Ideally help is "unified" before it's registered. However to help with this case perhaps registry could have some options (or registry wrapper should exists) that either ignores this error and picks first help OR allows custom logic. Help wanted to enhance that (in non-breaking change). 🤗 I wouldn't change current error in current flow. |
When the client find that a help description have changed it creates an error that from my
otel-collectors
perspective look like this:The error is raised here:
client_golang/prometheus/registry.go
Line 640 in 33697e6
It's an error originally by the application server (in this case wildfly), but it abrupts the metric collection for one bad metric.
Is it possible to issue a warning on the logs instead?
Concerning the metric sample there is a choice whether to stick to the original description, adopt the new one or simply drop the metric sample.
The text was updated successfully, but these errors were encountered: