[grafana] How to deal with 1MB chart size limit when creating dashboards?

[caleb15]

Helm has a 1MB size limit for charts, which means if you're putting the dashboards in a chart it can only support roughly ~40 dashboards or so. If you have different dashboards for each environment you can easily hit that number of dashboards or more. I'm wondering how people with a lot of dashboards avoid this issue.

https://helm.sh/docs/chart_template_guide/accessing_files/

It is okay to add extra files to your Helm chart. These files will be bundled. Be careful, though. Charts must be smaller than 1M because of the storage limitations of Kubernetes objects.

[caleb15]

I had a really good discussion about the various options to overcome this in the grafana labs slack.

Transcript:

Is there anyone using the grafana helm chart w/ roughly 40+ dashboards? I'm wondering how one works around the helm 1MB chart size limit. (edited) 
14 replies

Adam Gardner  1 hour ago
We don't currently have that many dashboards, but, I'm in the weeds with that chart myself at the moment. I think you'd probably use the dashboards sidecar and have a separate chart (or multiple charts) that add the dashboard configmaps.

Adam Gardner  1 hour ago
Alternatively, you could put the dashboard JSON at some URL (S3 bucket, maybe?) and have the chart download them from there using the dashboards.<provider>.<dashboard-name>.url value, although, you'd have to deal with auth if it's supposed to come from a secured place.


Caleb Sparks  1 hour ago
Yeah, seems kind of hacky but I guess you could do multiple charts... :neutral_face: The other possibilities I thought of were:
1) Simply embedding all the dashboards in the values of the helmfile.
2) Using helmfile w/ vanilla kubernetes or kustomize instead of helm
3) Making our dashboards open-source and use local-dashboard to point to the raw github files
3) is probably a no-go because although we don't have sensitive data in the charts it would still give hackers a insight into our infrastructure. 1 or 2 might work
 Alternatively, you could put the dashboard JSON at some URL (S3 bucket, maybe?) and have the chart download them from there using the dashboards.<provider>.<dashboard-name>.url value, although, you'd have to deal with auth if it's supposed to come from a secured place.
Oh, now there's an idea! We could have a private git repo deploy to a s3 bucket and only the eks nodes would be able to access the bucket. Hmmm :thinking_face: (edited) 

Adam Gardner  1 hour ago
Re: #1 isn't that the normal way of doing it? How else would you do it, other than the values file?

Adam Gardner  1 hour ago
And even if you put all the JSON in the values file, that doesn't actually fix it, does it?

Caleb Sparks  1 hour ago
 How else would you do it, other than the values file?
I started making a chart that would create a configmap for each dashboard, and the sidecar would load the configmaps. But then I found the 1MB size limit for the entire chart. :sob:

Adam Gardner  1 hour ago
Yeah, that seems like the worst of both worlds; but putting the JSON directly in values will hit the 1MB limit eventually too.

Adam Gardner  1 hour ago
I think the idea with the sidecar is that you have your application helm charts come with their own dashboards (assuming you have application-specific dashboards). There's even a few public helm charts that can optionally deploy these

Adam Gardner  1 hour ago
Currently I'm using Helmfile, and I've got a directory of JSON files, and helmfile uses readFile to load them in and inject them into the chart values. Probably not scalable in the long run, but I'm only sketching things out at the moment


Adam Gardner  1 hour ago
One way to break it up would be one release for each folder; unless you have 40+ dashboards in the same folder, it'd probably work

Caleb Sparks  1 hour ago
Yeah, that seems like the worst of both worlds; but putting the JSON directly in values will hit the 1MB limit eventually too.
I was reading through the chart and yep, looks like you're right. It creates one configmap for each provider, so unless you had multiple different providers (eww) then the configmap would grow over 1MB. Seems like it would be better if they did a configmap per file, but maybe they had a reason they structured it per provider.
https://github.com/grafana/helm-charts/blob/bc30504c85b19f8fdfae7f8451ac05993e6b9087/charts/grafana/templates/dashboards-json-configmap.yaml

Adam Gardner  1 hour ago
I'm using a provider per folder, personally, because it's the only way I can see to get the chart to deploy different dashboards to different folders (Grafana itself can have a single provider infer which dashboards go in which folder from the on-disk folder structure, but the Helm Chart doesn't expose that capability in any meaningful way) (edited) 

Adam Gardner  1 hour ago
Oh, or, maybe that's a lie. The chart's naming conventions imply that dashboards.foo.bar.json, bar is the dashboard filename and foo is the name of the provider, but, I'm not sure if that's actually enforced; for practical purposes foo may only be the name of the folder.

Adam Gardner  1 hour ago
Another option (which I hate, but which is probably viable for some) is to simply bake the dashboard files into a custom docker image, either based on the grafana image, or a separate clean image that runs as a sidecar to Grafana and shares data with it.

[sebastiangaiser]

Hey, when using the configmapgenerator this could happen the same. A ~workaround could be to remove all spaces and minify the file via jq -rc "." filename.json. For me this reduced the json from e.g. ~1.3MB to ~0.4MB.

[aeonzh]

You can add dashboard as configmaps not part of the chart by enabling sidecar for dashboards

[sebastiangaiser]

@aeonzh the problem is not "the" Helm chart. That's a limitation of ConfigMaps. Docs

A ConfigMap is not designed to hold large chunks of data. The data stored in a ConfigMap cannot exceed 1 MiB.

[aeonzh]

I believe that the 1MiB limitation applies to one single ConfigMap object.

What I mean is that you could split the ~40 dashboards into ~40 ConfigMaps. The sidecar should detect them and create dashboards for you à la controller, unless I misunderstood how the sidecar works...

I believe that's how kube-prometheus-stack manages the out-of-the-box dashboards after you install it: https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack/templates/grafana/dashboards-1.14