Allow .Files.Get and .File.Glob like Helm does

[smeierhofer]

I want to use .Files.Get and .Files.Glob just like I can in the Helm Chart. I want to define the Helmfile release along with a license file and configuration files needed by the release. I will define many releases and the configuration files for each have a lot in common and so I want to define the common chunks in fragment files and include those fragments in many configuration files.

To accomplish this, in my Helmfile release's values template, I want to do the following:

licenseData: {{ .Files.Get "license.bin" | b64enc }}
configData:
  {{ (tpl (.Files.Glob "config/*").AsConfig . ) | indent 2 }}

The Helm Chart defines a ConfigMap that has:

binaryData:
  license.bin: {{ .Values.licenseData }}
data:
  {{ toYaml .Values.configData | indent 2 }}

Then the ConfigMap is mounted into a container. The result is that the license file and configuration files specified by the Helmfile values template are available in the running container. Notice that the Helmfile's values template uses the "tpl" function as a way to process the config files as templates just like I can in a Helm Chart. This allows the config file to include common configuration fragments.

I could use the readFile function in place of .Files.Get, but it is nice if the Helmfile syntax is the same as the Helm Chart syntax. This is similar to existing issues #724 and #414 .

[bitsofinfo]

would also like this!

[sshishov]

Any update here?

[mumoshu]

Not really. Are we sure that this is different from referencing files contained in charts? I mean, reading and globbing files in the specified directory whose path is relative to helmfile.yaml would be enough?

[smeierhofer]

In our project, the helmfile and all the configuration files for the various helm releases are in one folder like this:

/releases
     /helmfile.yaml
     /commonProperties.yaml
     /configurationFiles.yaml.gotmpl     --> This file reads all config files in the config folder
     /customer1_release
          /config
               /configFile1.xml
               /configFile2.xml
               /configFile3.properties
          /secrets
               /secrets-properties.yaml
     /customer2_release
          /......

The "configurationFiles.yaml.gotmpl" Go template file reads in all the configuration files in the "config" folder using "exec sh" and "echo <name>/config/*" to create a list the files and then "range" loops over the listed files and reads each using "readFile". The result is Helm Chart values such as:

app:
     config:
          configFile1.xml: |
               file-contents...
          configFile2.xml: |
               file-contents...
          configFile3.properties: |
               file-contents...

The Helm Chart doesn't contain any configuration files. It does define a config map that uses the above values to define files. For example:

data:
  {{- tpl (toYaml .Values.app.config) . | nindent 2 }}

So..... Helmfile could make this easier to do by providing a function to read in an entire folder of text-based configuration files and adding them to the Helm Chart Values. As it is, we used the "exec sh" to list the files and "readFile" to read them in.

[sboardwell]

For anyone looking for an example, here is what we have come up with:

The helmfile.yaml

repositories:
- name: incubator
  url: https://charts.helm.sh/incubator
releases:
- name: configmap
  labels:
    config: true
  namespace: my-ns
  chart: incubator/raw
  version: 0.2.5
  values:
  - _configmap.yaml.gotmpl

And the _configmap.yaml.gotmpl:

resources:
- apiVersion: v1
  kind: ConfigMap
  metadata:
    name: my-configmap
  data:
{{- range $_, $file := ( exec "bash" (list "-c" "echo -n path/to/files/*") | splitList " " ) }}
{{ regexReplaceAll ".*/" $file "" | indent 4 }}: |
{{ readFile $file | indent 6 }}
{{ end }}

[danieltaub96]

@sboardwell Thank you for the solution, still trying to understand how you created the config map in your helm chart eventually?

[sboardwell]

@danieltaub96 I used the raw chart from the incubator repository and just used the configmap in the resources section of the raw helm chart. You then have two releases, your own helm chart which depends on the raw chart with the configmap.

[max-rocket-internet]

@mumoshu we have a bunch of helmfile templates in a directory that we want to render without listing them by name.

Currently we have this (it's simplified):

{{- $helmfiles := list "cluster-overprovisioner.yaml" "common.yaml" "node-local-dns.yaml" }}

helmfiles:
{{- range $filename := $helmfiles }}

- path: templates/{{ $filename }}
  values:
  - {{ toYaml $.Values | nindent 4 }}
  - global_config:
      ....

{{- end }}

We are passing some more complex values to them. It seems $.Files.Glob would help here but it's not in included in helmfile. Is there another way to do this? We have many templates.

[yxxhero]

@max-rocket-internet please see helmfile/helmfile.

[yxxhero]

https://helmfile.readthedocs.io/en/latest/templating_funcs/#readdirentries

[max-rocket-internet]

Ah nice, we switched to this:

helmfiles:

{{ range $index, $filename := readDir "templates" }}
...

Much nicer 🎉

[yxxhero]

@max-rocket-internet if you have more issues. please post a new issue into helmfile/helmfile.

[max-rocket-internet]

I think we could close this no? readDir and readFile can be used in place of Helm .File functions.