k8s_* metrics not showing in Digital Ocean K8s

[prashant-shahi]

kubeletstats receiver not working as expected.

Error log:

2023-02-22T06:53:05.880Z	error	kubeletstatsreceiver@v0.66.0/scraper.go:80	call to /stats/summary endpoint failed	{"kind": "receiver", "name": "kubeletstats", "pipeline": "metrics", "error": "Get \"http://pool-iytqfvim8-qo0cj:10255/stats/summary\": dial tcp: lookup pool-iytqfvim8-qo0cj on 10.245.0.10:53: no such host"}
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver.(*kubletScraper).scrape
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver@v0.66.0/scraper.go:80
go.opentelemetry.io/collector/receiver/scraperhelper.ScrapeFunc.Scrape
	go.opentelemetry.io/collector@v0.66.0/receiver/scraperhelper/scraper.go:31
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport
	go.opentelemetry.io/collector@v0.66.0/receiver/scraperhelper/scrapercontroller.go:211
go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1
	go.opentelemetry.io/collector@v0.66.0/receiver/scraperhelper/scrapercontroller.go:193

[paulohmorais]

Same problem here

Digital Oean K8s

The IP from the error is kube-dns svc:

kube-system kube-dns ClusterIP 10.245.0.10 dns:53►0╱UDP dns-tcp:53►0 metrics:9153►0

Errors logs:

│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x 2023-09-06T22:25:02.668Z    error    scraperhelper/scrapercontroller.go:213    Error scraping metrics    {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics", "error": "Get \"https://w-pool-uy4qaitgx-y74ag:10250/stats/summary\": dial tcp: lookup w-pool-uy4qaitgx-y74ag on 10.245.0.10:53: no such host", "scraper": "kubeletstats"}                                                      │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport                                                                                                                                                                                                                                                                                                                                         │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x     go.opentelemetry.io/collector/receiver@v0.79.0/scraperhelper/scrapercontroller.go:213                                                                                                                                                                                                                                                                                                                                         │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1                                                                                                                                                                                                                                                                                                                                            │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x     go.opentelemetry.io/collector/receiver@v0.79.0/scraperhelper/scrapercontroller.go:192                                                                                                                                                                                                                                                                                                                                         │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x 2023-09-06T22:25:32.665Z    error    kubeletstatsreceiver@v0.79.0/scraper.go:68    call to /stats/summary endpoint failed    {"kind": "receiver", "name": "kubeletstats", "data_type": "metrics", "error": "Get \"https://w-pool-uy4qaitgx-y74ag:10250/stats/summary\": dial tcp: lookup w-pool-uy4qaitgx-y74ag on 10.245.0.10:53: no such host"}                                                             │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver.(*kubletScraper).scrape                                                                                                                                                                                                                                                                                                                   │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x     github.com/open-telemetry/opentelemetry-collector-contrib/receiver/kubeletstatsreceiver@v0.79.0/scraper.go:68                                                                                                                                                                                                                                                                                                                 │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x go.opentelemetry.io/collector/receiver/scraperhelper.ScrapeFunc.Scrape                                                                                                                                                                                                                                                                                                                                                            │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x     go.opentelemetry.io/collector/receiver@v0.79.0/scraperhelper/scraper.go:20                                                                                                                                                                                                                                                                                                                                                    │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).scrapeMetricsAndReport                                                                                                                                                                                                                                                                                                                                         │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x     go.opentelemetry.io/collector/receiver@v0.79.0/scraperhelper/scrapercontroller.go:210                                                                                                                                                                                                                                                                                                                                         │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x go.opentelemetry.io/collector/receiver/scraperhelper.(*controller).startScraping.func1                                                                                                                                                                                                                                                                                                                                            │
│ signoz-k8s-metrics-exporter-k8s-infra-otel-agent-dnl9x     go.opentelemetry.io/collector/receiver@v0.79.0/scraperhelper/scrapercontroller.go:192     

[prashant-shahi]

Tracking issue: open-telemetry/opentelemetry-collector-contrib#22843

[prashant-shahi]

@paulohmorais Meanwhile, you can use cAdvisor with otel-collector-metrics instead of kubeletstats receiver. As mentioned: SigNoz/dashboards#13 (comment)

[prashant-shahi]

Related PR

• otel-agent: collect kubelet stats from host IP #380