[victoriametrics-k8s-stack] Grafana dashboard multicluster mode

[jpigree]

Hi. I want to create a monitoring stack with VictoriaMetrics to monitor multiple Kubernetes cluster. I used the 'victoria-metrics-k8s-stack' chart (version: 0.14.17) to install a VMCluster on one of my Kubernetes cluster and a Grafana instance.

In order to have multicluster dashboards (dashboards with a cluster parameter to filter metrics), I wrote the following in my values file with no effect:

grafana:
  sidecar:
    dashboards:
      enabled: true
      additionalDashboardLabels: {}
      additionalDashboardAnnotations: {}
      multicluster: true

I grep'ed into the chart and from what I saw, the field is used in a script which is never used in the chart.

$ cd victoria-metrics-k8s-stack
$ grep -R "sidecar.dashboards.multicluster" *
hack/sync_grafana_dashboards.py:                '\{\{ if .Values.grafana.sidecar.dashboards.multicluster \}\}0\{\{ else \}\}2\{\{ end \}\}',
README.md:| grafana.sidecar.dashboards.multicluster | bool | `false` |  |
$ grep -R "sync_grafana_dashboards.py" -l *                                                                                                  
hack/README.md
RELEASE_GUIDE.md

I understand that the script unhide the cluster variable in dashboards so it should be okay for dashboards already having the cluster variable defined and used. However, not all dashboards in the chart are made with a cluster variable to begin with (example here).

In short, my questions are:

• What is the purpose of this multicluster field?
• Is there a known way to make the packaged dashboards multicluster?

Thanks for your help

[jpigree]

After a bit of thinking, I think the easiest way to implement this without reworking a lot of dashboards is to send metrics to different tenants and to create one datasource for each. Since, a lot of opensource Kubernetes dashboards have the datasource as a visible variable, we will be able to switch the cluster metrics shown by switching the datasource.

[bdschaap]

I have a similar use case. Though if I use multiple tenants in victoriametrics then I won't be able to query metrics across tenants due to their isolation. I discovered that I can use Custom query parameters in the data source to still use a single tenant but filter clusters through the use of extra_filters. I set extra_filters[]={cluster="cluster1"} to filter on cluster1 and then create a datasource per cluster.

[dhess]

I can reproduce this, and it's evident by comparing Helm manifests with multicluster enabled or not that this setting has no effect.

[dhess]

Oh, nice, @dotdc recently added multi-cluster support to their dashboards, so for the Kubernetes dashboards, at least, it should only take an upstream refresh to resolve this:

dotdc/grafana-dashboards-kubernetes#90

[dotdc]

Hi 👋
Yes, it should work now!
If you're interested to add multi-cluster support to other dashboards, we used this solution:

• [enhancement] cluster variable support dotdc/grafana-dashboards-kubernetes#15 (comment)

Tested on Prometheus 2.0.0 (2017-11-08), and maybe works on older versions too, so it should be safe for most users 😉

[jpigree]

Hi. Thanks for the fix. I will give it a try to confirm this issue is solved.

[AndrewChubatiuk]

it should be fixed since k8s-stack v0.19.0

[dotdc]

If it's the case, maybe we can close this issue?