Grafana Agent support for scraping custom metrics from github.com/kubernetes-sigs/prometheus-adapter

[aengusrooneygrafana]

My first assumption is that I would like to stick with the Grafana Agent only to send Prom metrics to Grafana Cloud, as I understood we do not need to manage any other Prometheus components in this case.

I was able to successfully install the Grafana Agent Operator + Grafana Agent
Logs and metrics were flowing properly to the Grafana Cloud test stack
I was able to add/remove the ServiceMonitor's and the automation seems to work fine

The issue I hit was the fact we are using https://github.com/kubernetes-sigs/prometheus-adapter for custom metrics

This service requires to have Prometheus as a source of information: https://github.com/kubernetes-sigs/prometheus-adapter/blob/master/deploy/manifests/deployment.yaml#L34

Currently, we have it working with in-cluster Prometheus like this:

--prometheus-url=https://prometheus.monitoring.svc:9090/

Even though the Grafana Cloud exposes the endpoint to pull aggregated data from it as it is password protected, this prometheus-adapter does not seem to support it

My second choice and test was to use both the Grafana Agent and on-cluster Prometheus with configured remoteWrite to Grafana Cloud

I was able to successfully install the Grafana Agent Operator + Grafana Agent
I was able to successfully setup the Prometheus Operator + Prometheus + remoteWrite to Grafana Cloud
Logs and metrics were flying properly to the Grafana Cloud test stack I have
I was able to add/remove the ServiceMonitor's and the automation seems to work fine

Grafana Agent in this setup is actually handling only the LogsInstance and pushing logs to the Grafana Cloud.

To summarise:

The only solution I was able to come up with to make existing stuck preserves it functionality and supports sending logs to Grafana Cloud is:

Grafana Agent Operator + Grafana Agent (for logs)
Prometheus Operator + Prometheus (for metrics)

Driver: We would like to avoid the need to manage the Prometheus servers for the clusters on our side, but I do not know any other solution to make it happen than the one I have described.

Could you please provide some advice on this topic?

[rfratto]

I don't think there's anything for Grafana Agent to do here. Prometheus Adapter can be configured with credentials to connect to Prometheus. From its code, I see the following flags:

• -prometheus-url: URL for connecting to Prometheus.
• -prometheus-auth-incluster: Use auth details from the in-cluster kubeconfig when connecting to Prometheus.
• -prometheus-auth-config: kubeconfig file used to configure auth when connecting to Prometheus.
• -prometheus-ca-file: Optional CA file to use when connecting with Prometheus.
• -prometheus-client-tls-cert-file: Optional client TLS cert file to use when connecting with Prometheus, auto-renewal is not supported.
• -prometheus-client-tls-key-file: Optional client TLS key file to use when connecting with Prometheus, auto-renewal is not supported.
• -prometheus-token-file: Optional file containing the bearer token to use when connecting with Prometheus.
• -prometheus-header: Optional header to set on requests to prometheus-url. Can be repeated.
• -prometheus-verb: HTTP verb to set on requests to Prometheus. Possible values: "GET", "POST".
• -config: Configuration file containing details of how to transform between Prometheus metrics and custom metrics API resources.
• -metrics-relist-interval: interval at which to re-list the set of all available metrics from Prometheus.
• -metrics-max-age: period for which to query the set of available metrics from Prometheus.

[DWebb0]

@DWebb0, registering his interest here