Dashboards Showing No Data

[bitva77]

What happened?

Most dashboards are showing No data

Did you expect to see some different?

Sort of expected to see like networking or memory data in the Grafana dashboards.

How to reproduce it (as minimally and precisely as possible):

jsonnetfile.json:

{
  "version": 1,
  "dependencies": [
    {
      "source": {
        "git": {
          "remote": "https://github.com/prometheus-operator/kube-prometheus.git",
          "subdir": "jsonnet/kube-prometheus"
        }
      },
      "version": "master"
    }
  ],
  "legacyImports": true
}

kube-prometheus.jsonnet:

// Modified version of:
// https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/example.jsonnet

local kp = ((import 'kube-prometheus/kube-prometheus.libsonnet') + {
   _config+:: {
     alertmanager+: {
       config: importstr 'alertmanager-config.yaml',
     },
   },
 }) +
           (import 'kube-prometheus/kube-prometheus-kubeadm.libsonnet') +
           (import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet') +
           (import 'kube-prometheus/kube-prometheus-node-ports.libsonnet') +
           (import 'kube-prometheus/kube-prometheus-all-namespaces.libsonnet') + {
  _config+:: {
    namespace: 'monitoring',

    prometheus+:: {
      namespaces: [],
    },
  },
};
{ ['00namespace-' + name + '.json']: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name + '.json']: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name + '.json']: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name + '.json']: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name + '.json']: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name + '.json']: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name + '.json']: kp.grafana[name] for name in std.objectFields(kp.grafana) }

build.sh:

#!/usr/bin/env bash

# This script uses arg $1 (name of *.jsonnet file to use) to generate the manifests/*.yaml files.

GOPATH=$HOME/gocode
PATH=$PATH:$HOME/gocode/bin:$HOME/go/bin
export PATH GOPATH

set -e
set -x
# only exit with zero if all commands of the pipeline exit successfully
set -o pipefail

# Make sure to start with a clean 'manifests' dir
rm -rf manifests
mkdir manifests

# optional, but we would like to generate yaml, not json
jsonnet -J vendor -m manifests "${1-kube-prometheus.jsonnet}" | xargs -I{} sh -c 'sleep .2; cat {} | gojsontoyaml > {}.yaml; rm -f {}' -- {}

Environment

• Prometheus Operator version:

quay.io/prometheus-operator/prometheus-operator:v0.42.1

• Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes cluster kind:

kubeadm. Original install 1.13 - upgraded via kubeadm all the way to 1.19

• Manifests:

see above

• Prometheus Operator Logs:

no errors

Anything else we need to know?:

No errors from any of the pods. All prometheus targets are green and OK. I'm guessing this is a K8s 1.19 compatibility issue? I thought master would suffice but I see the same issue using 0.6 or master.

[bitva77]

This appears to be related to this: kubernetes/kubernetes#94281

I was running Docker 18.09. After upgrading that to 19.03, it appears that most graphs now have data. All Prometheus monitoring points were fine it was just performance related graphs that were empty.

My network graphs have empty but that may be because of Flannel maybe? Will look into that more next week.

Anywho, not a kube-prometheus bug here. Sorry for the false alarm. Might be worth mentioning the Docker version bug above with Kubernetes 1.19 in the support matrix.