[receiver/prometheus] missing otelcol_scraper... metrics?

[karmingc]

Component(s)

receiver/prometheus

Describe the issue you're reporting

Hi, my current configuration is setting a "scraper" statefulset that uses the Prometheus receiver to scrape other collectors, including itself. However, I am currently seeing some logs such as Failed to scrape Prometheus endpoint, but I don't see any metrics related to the "scraper" that could indicate that.

Does the Prometheus receiver generate internal metrics such as otelcol_scraper_errored_metric_points and otelcol_scraper_scraped_metric_points?

What would be the alternative to monitor metrics about the Prometheus receiver?

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

Hello @karmingc, the collector is able to expose its own internal metrics on a port, or even send its own metrics through its own pipeline. Here's some documentation on how to accomplish this.

[karmingc]

Hello @karmingc, the collector is able to expose its own internal metrics on a port, or even send its own metrics through its own pipeline. Here's some documentation on how to accomplish this.

Hi @crobert-1, I updated my description as it might have been misleading. My setup is already scraping other collector's Prometheus endpoint. However, on the "scraper" pod, which is where the Prometheus receiver is configured, it seems to be missing metrics about itself.

I see that in the hostmetric receiver that it provides these 2 internal metrics: otelcol_scraper_errored_metric_points and otelcol_scraper_scraped_metric_points, but that doesn't seem the case for the Prometheus receiver. I am wondering what should the most appropriate approach to monitor failed scrapes.

[crobert-1]

My apologies, I misunderstood your question. Thanks for updating the description and clarifying!

The hostmetrics receiver is recording these metrics because it's using the core collector's receiver scraper helper functionality, which has these metrics built in. The prometheus receiver on the other hand has done all of its own scraper logic, and does not leverage the scraper helper's functionality.

There would be two main ways to enable this:

1. Move the prometheus receiver to rely on the scraper helper's functionality. This would be a major effort and possibly invalid if for some reason Prometheus requires specific functions that the scraper helper doesn't support.
2. Import the receiver helper package in the Prometheus receiver, then add a new ObsReport object that can record metrics of interest. This is done by numerous receivers in contrib and is likely the preferred option here.

Code references:
Referenced metrics being recorded
Top level call that adds internal metrics
ObsReport added to receiver

[karmingc]

Thanks for the insights. That brings me to the second part of my question, what would be the current approach, if any, to monitor metrics about the prometheus receiver related to its scraping? I would probably add that I'm not too familiar with Prometheus itself and we added it for the purpose of collecting metrics about the Collectors.

[crobert-1]

I'm not sure I fully understand your question, are you asking how to actually use (or monitor) the metrics you're getting from the prometheus receiver? Or are you wondering how to add more metrics to monitor the receiver? Something else?

[karmingc]

I'm more concerned about metrics of the prometheus receiver's scraping. Like in my example, I get logs about a failed scraping and I would like to know if there's a metric that reflects that.

To clarify, my understanding is that the current approach with Prometheus receiver won't generate the following metrics:

• otelcol_scraper_errored_metric_points
• otelcol_scraper_scraped_metric_points

like the hostmetrics receiver because it does not use the scraper helper. In that case, what would a roundabout to get scraped metrics and failed scraped metrics with the current version of the Prometheus receiver?

[crobert-1]

In that case, what would a roundabout to get scraped metrics and failed scraped metrics with the current version of the Prometheus receiver?

From what I can tell the Prometheus receiver potentially exposes some internal metrics about the scraper, so you'd probably have to manually look through those metrics to see what it exposes.

If you don't find anything useful, it would need to be implemented as I suggested.

[dashpole]

I would recommend using the up metric, which tracks scrape success/failure per-target. There are also scrape_* metrics that describe the number of series, like scrape_samples_scraped that can be useful.

[karmingc]

Thanks @dashpole, I forgot to update this discussion when I found the same metrics as you described in this Prometheus doc.

Appreciate your help as well @crobert-1 and the insights on the internal working of the receiver.