[stable/grafana] unable to configure dashboards in values file

[previ]

I cannot define dashboard to be imported in grafana at install time.

Description:
Configuring dashboard to be auto imported (by dashboard id) at install time, they are ignored.

Helm: 2.13.1
Kubernetes: 1.13.5

Which chart: stable/grafana 3.5.11

What happened: dashboards configured are actually ignored.

What you expected to happen: dashboard configured to be impoterd to be actually imported.

How to reproduce it (as minimally and precisely as possible):
use the following lines in the values.yanl:

dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
    - name: 'default'
      orgId: 1
      folder: ''
      type: file
      disableDeletion: false
      editable: true
      options:
        path: /var/lib/grafana/dashboards/default

dashboards:
  default:
    kubernetes-cluster:
      gnetId: 7249
      datasource: Prometheus

Anything else we need to know:

[irasnyd]

I hit this same issue today.

FYI, this works with the stable/grafana chart version 3.5.4, but does not work in the latest version 3.5.12.

[irasnyd]

Further testing shows these results:

Version	Result
3.5.4	Works Perfectly
3.5.5	Works Perfectly
3.5.6	YAML parse error (see below)
3.5.7	Grafana crashes on startup (see below)
3.5.8	Grafana crashes on startup (logs are identical to 3.5.7)

stable/grafana version 3.5.6 YAML parse error:

$ helm upgrade --install my-grafana stable/grafana --version 3.5.6 -f ~/kube-metrics/grafana-values.yaml --namespace default 
UPGRADE FAILED
Error: YAML parse error on grafana/templates/deployment.yaml: error converting YAML to JSON: yaml: line 39: did not find expected key
Error: UPGRADE FAILED: YAML parse error on grafana/templates/deployment.yaml: error converting YAML to JSON: yaml: line 39: did not find expected key

stable/grafana version 3.5.7 crash logs:

$ k logs -f my-grafana-5c956c7965-m7qws 
t=2019-07-20T00:00:23+0000 lvl=info msg="Starting Grafana" logger=server version=6.2.4 commit=ee01897 branch=HEAD compiled=2019-06-18T10:39:19+0000
t=2019-07-20T00:00:23+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2019-07-20T00:00:23+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.logs=/var/log/grafana"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.provisioning=/etc/grafana/provisioning"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.log.mode=console"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_SECURITY_ADMIN_USER=admin"
t=2019-07-20T00:00:23+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_SECURITY_ADMIN_PASSWORD=*********"
t=2019-07-20T00:00:23+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
t=2019-07-20T00:00:23+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
t=2019-07-20T00:00:23+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
t=2019-07-20T00:00:23+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
t=2019-07-20T00:00:23+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning
t=2019-07-20T00:00:23+0000 lvl=info msg="App mode production" logger=settings
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing SqlStore" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
t=2019-07-20T00:00:23+0000 lvl=info msg="Starting DB migration" logger=migrator
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing HTTPServer" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing InternalMetricsService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing RemoteCache" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing QuotaService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing LoginService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing PluginManager" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Starting plugin search" logger=plugins
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing RenderingService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing AlertingService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing HooksService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing SearchService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing ServerLockService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing TracingService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing UsageStatsService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing CleanUpService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing NotificationService" logger=server
t=2019-07-20T00:00:23+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server
t=2019-07-20T00:00:23+0000 lvl=eror msg="Cannot read directory" logger=provisioning.dashboard type=file name=provisioned error="stat /var/lib/grafana/dashboards/provisioned: no such file or directory"
t=2019-07-20T00:00:23+0000 lvl=eror msg="Failed to read content of symlinked path" logger=provisioning.dashboard type=file name=provisioned path=/var/lib/grafana/dashboards/provisioned error="lstat /var/lib/grafana/dashboards/provisioned: no such file or directory"
t=2019-07-20T00:00:23+0000 lvl=info msg="falling back to original path due to EvalSymlink/Abs failure" logger=provisioning.dashboard type=file name=provisioned
t=2019-07-20T00:00:23+0000 lvl=eror msg="Server shutdown" logger=server reason="Service init failed: Failed to provision dashboards: Failed to provision config provisioned: stat /var/lib/grafana/dashboards/provisioned: no such file or directory"

Here is the complete differences between the chart version 3.5.5 and 3.5.7:

$ diff -Nru grafana-3.5.{5,7}
diff -Nru grafana-3.5.5/Chart.yaml grafana-3.5.7/Chart.yaml
--- grafana-3.5.5/Chart.yaml	1969-12-31 16:00:00.000000000 -0800
+++ grafana-3.5.7/Chart.yaml	1969-12-31 16:00:00.000000000 -0800
@@ -15,4 +15,4 @@
 name: grafana
 sources:
 - https://github.com/grafana/grafana
-version: 3.5.5
+version: 3.5.7
diff -Nru grafana-3.5.5/templates/deployment.yaml grafana-3.5.7/templates/deployment.yaml
--- grafana-3.5.5/templates/deployment.yaml	1969-12-31 16:00:00.000000000 -0800
+++ grafana-3.5.7/templates/deployment.yaml	1969-12-31 16:00:00.000000000 -0800
@@ -75,7 +75,8 @@
         - name: download-dashboards
           image: "{{ .Values.downloadDashboardsImage.repository }}:{{ .Values.downloadDashboardsImage.tag }}"
           imagePullPolicy: {{ .Values.downloadDashboardsImage.pullPolicy }}
-          command: ["sh", "/etc/grafana/download_dashboards.sh"]
+          command: ["/bin/sh"]
+          args: [ "-c", "mkdir -p /var/lib/grafana/dashboards/default", "/etc/grafana/download_dashboards.sh" ]
           volumeMounts:
             - name: config
               mountPath: "/etc/grafana/download_dashboards.sh"

[obeyler]

this should be solve by #15702

[obeyler]

The PR #15702 is merged could you check if it closes this issue ?
Thanks by advance @previ

[previ]

Tested stable/grafana chart version 3.6.0, same values as before; now I get Init:Error at container download-dashboards starting.
Error message in init container log: /bin/sh: /etc/grafana/download_dashboards.sh: Permission denied.

[irasnyd]

I confirmed that deploying the Helm chart version 3.7.0 does not work. The download-dashboards initContainer gives the following log message:

$ kubectl logs -f my-grafana-6cf4b4c8b9-qfm2p -c download-dashboards
/bin/sh: /etc/grafana/download_dashboards.sh: Permission denied

This is because the /etc/grafana/download_dashboards.sh file is not executable. AFAICT, it is not possible to instruct Kubernetes to set the executable mode bits on a file which is mounted from a ConfigMap.

I did this instead, which works as expected (the dashboards are provisioned correctly):

diff --git a/stable/grafana/templates/deployment.yaml b/stable/grafana/templates/deployment.yaml
index 0e01d32e0f86..d82578a38aff 100644
--- a/stable/grafana/templates/deployment.yaml
+++ b/stable/grafana/templates/deployment.yaml
@@ -76,7 +76,7 @@ spec:
           image: "{{ .Values.downloadDashboardsImage.repository }}:{{ .Values.downloadDashboardsImage.tag }}"
           imagePullPolicy: {{ .Values.downloadDashboardsImage.pullPolicy }}
           command: ["/bin/sh"]
-          args: [ "-c", "mkdir -p /var/lib/grafana/dashboards/default && /etc/grafana/download_dashboards.sh" ]
+          args: [ "-c", "mkdir -p /var/lib/grafana/dashboards/default && /bin/sh /etc/grafana/download_dashboards.sh" ]
           volumeMounts:
             - name: config
               mountPath: "/etc/grafana/download_dashboards.sh"

[previ]

solved in 3.7.3