stable/grafana Datasources added from chart config doesn't work but same datasource added manually works

[shubhambhattar]

Describe the bug
I am using stable/grafana chart with some custom config (given below).

grafana:
  rbac:
    create: false
    pspEnabled: false
    pspUseAppArmor: false

  datasources:
    datasources.yaml:
      apiVersion: 1
      datasources:
      - name: Graphite
        type: graphite
        url: http://graphite.monitoring.svc.cluster.local:8080
        access: direct
        isDefault: true

  ingress:
    enabled: true
    annotations: 
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/rewrite-target: /
      nginx.com/health-checks: "true"
      nginx.ingress.kubernetes.io/enable-access-log: "false"
    hosts: 
      - <hostname>

This installs grafana and the default datasource (graphite) is setup. But it doesn't work. If I add the same datasource manually from Grafana UI, it works.

Version of Helm and Kubernetes:

$ helm3 version
version.BuildInfo{Version:"v3.0.3", GitCommit:"ac925eb7279f4a6955df663a0128044a8a6b7593", GitTreeState:"clean", GoVersion:"go1.13.6"}

Which chart: stable/grafana

What happened: Datasource added while deploying chart using custom values.yaml file doesn't work.

What you expected to happen: The datasource added automatically should work.

How to reproduce it (as minimally and precisely as possible):
The environment is as below:

 $ tree setup/grafana/
setup/grafana/
├── Chart.lock
├── Chart.yaml
├── charts
│   └── grafana-5.0.0.tgz
├── templates
│   └── _helpers.tpl
└── values.yaml

2 directories, 5 files
$ cat setup/grafana/values.yaml
# Default values for grafana.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.


grafana:
  rbac:
    create: false
    pspEnabled: false
    pspUseAppArmor: false

  datasources:
    datasources.yaml:
      apiVersion: 1
      datasources:
      - name: Graphite
        type: graphite
        url: http://graphite.monitoring.svc.cluster.local:8080
        access: direct
        isDefault: true

  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/rewrite-target: /
      nginx.com/health-checks: "true"
      nginx.ingress.kubernetes.io/enable-access-log: "false"
    hosts:
      - <hostname>

Anything else we need to know:
graphite is already installed in using https://github.com/kiwigrid/helm-charts/tree/master/charts/graphite

[dvulpe]

This issues appears to have been introduced with this PR: #20985 and a possible workaround is to declare the previous configuration for the datasource sidecar in the extraInitContainers section.

The reason for the regression is the sidecar responsible with pulling the datasource configuration from configmap/secret objects has been moved from initContainers to containers, but Grafana only appears to provision datasources at startup.

It’s possible to manage datasources in Grafana by adding one or more yaml config files in the provisioning/datasources directory. Each config file can contain a list of datasources that will be added or updated during start up. If the datasource already exists, Grafana will update it to match the configuration file. The config file can also contain a list of datasources that should be deleted. That list is called deleteDatasources. Grafana will delete datasources listed in deleteDatasources before inserting/updating those in the datasource list.

[Annegies]

We're also waiting for this PR to fix the Grafana chart so we can update.
Thanks @dvulpe 👏