Node-exporter + kube-state-metrics configuration steps missing

[rbuffi]

Hello,

I deployed Prometheus + node-exporter + kube-state-metrics and followed the instructions but dashboards are not populating. I can see some metrics in the plugin itself like cluster status, node and applications overview.
Is there some additional configuration I need to do to point prometheus to node-exporter / kube-state-metrics?

Regards,

Ronald

[rbuffi]

Allthough I replaced the token and certificate file in prometheus I see the following errors:

msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" in the namespace "monitoring""
level=error ts=2020-08-25T08:39:16.118Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:363: Failed to list *v1.Pod: pods is forbidden: User "system:serviceaccount:default:default" cannot list resource "pods" in API group "" in the namespace "kube-system""
level=error ts=2020-08-25T08:39:20.278Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:362: Failed to list *v1.Service: services is forbidden: User "system:serviceaccount:default:default" cannot list resource "services" in API group "" in the namespace "default""

[rbuffi]

After specifying the serviceaccount in the pometheus deployment the right account is used. I see some targets in prometheus but there are not a lot metrics being collected. Also the dashboards stay empty.

level=info ts=2020-08-25T23:40:33.097Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=28.837431ms
level=info ts=2020-08-25T23:40:33.424Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.447403ms
level=info ts=2020-08-25T23:40:33.425Z caller=checkpoint.go:96 component=tsdb msg="Creating checkpoint" from_segment=22 to_segment=23 mint=1598335200000
level=info ts=2020-08-25T23:40:33.814Z caller=head.go:884 component=tsdb msg="WAL checkpoint complete" first=22 last=23 duration=389.804627ms
level=info ts=2020-08-25T23:40:34.145Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.06915ms
level=info ts=2020-08-25T23:40:34.480Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.588383ms
level=info ts=2020-08-25T23:40:34.481Z caller=checkpoint.go:96 component=tsdb msg="Creating checkpoint" from_segment=24 to_segment=25 mint=1598349600000
level=info ts=2020-08-25T23:40:34.886Z caller=head.go:884 component=tsdb msg="WAL checkpoint complete" first=24 last=25 duration=406.052433ms
level=info ts=2020-08-25T23:40:35.235Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.838507ms
level=info ts=2020-08-25T23:40:35.588Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=39.801732ms
level=info ts=2020-08-25T23:40:35.590Z caller=checkpoint.go:96 component=tsdb msg="Creating checkpoint" from_segment=26 to_segment=27 mint=1598364000000
level=info ts=2020-08-25T23:40:35.988Z caller=head.go:884 component=tsdb msg="WAL checkpoint complete" first=26 last=27 duration=399.974385ms
level=info ts=2020-08-25T23:40:36.349Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.356627ms
level=info ts=2020-08-25T23:40:36.679Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.133853ms
level=info ts=2020-08-25T23:40:36.680Z caller=checkpoint.go:96 component=tsdb msg="Creating checkpoint" from_segment=28 to_segment=29 mint=1598378400000
level=info ts=2020-08-25T23:40:37.060Z caller=head.go:884 component=tsdb msg="WAL checkpoint complete" first=28 last=29 duration=380.93007ms
level=info ts=2020-08-25T23:40:37.384Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.456885ms
level=info ts=2020-08-25T23:40:37.755Z caller=head.go:804 component=tsdb msg="Head GC completed" duration=29.179204ms
level=info ts=2020-08-25T23:40:37.756Z caller=checkpoint.go:96 component=tsdb msg="Creating checkpoint" from_segment=30 to_segment=31 mint=1598392800000
level=info ts=2020-08-25T23:40:38.121Z caller=head.go:884 component=tsdb msg="WAL checkpoint complete" first=30 last=31 duration=366.108777ms

[rbuffi]

I only see one active target. Can somebody help me out. I think I am close to a working set-up :)

monitoring/monitoring-prometheus-oper-alertmanager/0 (0/0 active targets)
monitoring/monitoring-prometheus-oper-apiserver/0 (1/14 active targets)
monitoring/monitoring-prometheus-oper-coredns/0 (0/18 active targets)
monitoring/monitoring-prometheus-oper-grafana/0 (0/0 active targets)
monitoring/monitoring-prometheus-oper-kube-controller-manager/0 (0/18 active targets)
monitoring/monitoring-prometheus-oper-kube-etcd/0 (0/18 active targets)
monitoring/monitoring-prometheus-oper-kube-proxy/0 (0/18 active targets)
monitoring/monitoring-prometheus-oper-kube-scheduler/0 (0/18 active targets)
monitoring/monitoring-prometheus-oper-kube-state-metrics/0 (0/0 active targets)
monitoring/monitoring-prometheus-oper-kubelet/0 (0/18 active targets)
monitoring/monitoring-prometheus-oper-kubelet/1 (0/18 active targets)
monitoring/monitoring-prometheus-oper-node-exporter/0 (0/0 active targets)
monitoring/monitoring-prometheus-oper-operator/0 (0/0 active targets)
monitoring/monitoring-prometheus-oper-prometheus/0 (0/0 active targets)