Added SSM integration

[rms1000watt]

There's been some interest for helmfile integration with SSM. Here is an example of what it can look like.

For our current workflows, we have Bash scripts that export Env Vars via aws-env then run helmfile. It would be way cleaner to have this integrated

[osterman]

Ideally we could support multiple paths. Perhaps use ; delimiter like other PATH vars

[lanmalkieri]

Definitely think we should have the ability to support multiple paths.

Would you rather define each path in the environment? Or just directly in the helmfile? I think it makes more sense to just put right in the helmfile.

[rms1000watt]

@lanmalkieri I'm looking at updating the helmfile.yml spec to include ssm: at the root level of the YAML.

ssm:
  region: us-west-2
  prefix: /path/in/ssm

The idea is then the prefix is optional when you start using ssm directly

{{ ssm "postgres_pass" }}

instead of

{{ ssm "/path/in/ssm/postgres_pass" }}

But of course you can do something like..

{{ ssm "/path/one/postgres_pass" }}
{{ ssm "/path/two/redis_pass" }}

[osterman]

For greater interoperability with chamber which lowercases all parameters, it would be nice if there was an option to uppercase them for environment variables.

[osterman]

I think this is more of a warning or debug.

[osterman]

Maybe warn if overwriting an existing variable

[sgandon]

Hi guys, could you explain what is SSM, and what does this have in common with helmfile and this PR brings.
I am not sure to understand the need for this ?

[rms1000watt]

@sgandon Ah, yes, will do!

What is AWS SSM

AWS SSM: https://docs.aws.amazon.com/systems-manager/latest/userguide/what-is-systems-manager.html

Specifically, we use Parameter Store within SSM: https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-parameter-store.html

At a high level, AWS SSM Parameter Store is a free KV store provided by AWS. I've used it across various projects & companies. @osterman has https://cloudposse.com/ where his teams use it across multiple projects & companies as well. So it's really popular.

What's the intersection between SSM & Helmfile?

Helmfile is very powerful for many reasons! One of the awesome features is that we can {{requireEnv "postgres_password"}} inside the Helmfile YAML. But then the question is, how do you get that (secret) value into your environment so Helmfile can use it?

What we do currently is use tools like https://github.com/segmentio/chamber and https://github.com/Droplr/aws-env to extract SSM variables and add to our Env before running helmfile commands. This results in extra logic/dependencies in our CI/CD workflows.

This PR was aimed at solving that issue of the extra logic/dependencies.

Next steps

There are a few different approaches to take:

• Address the suggestions in this PR proposed by @osterman and merge
• Consider a larger approach at handling secret values & adding to the environment (outside of AWS SSM). (AWS SSM would be one implementation of a Golang interface for this SecretGetter idea.)
• Consider a larger approach of retrieving secret values directly from source as defined in the helmfile (for example serverless.com does SSM retrieval natively, amongst others: https://serverless.com/framework/docs/providers/aws/guide/variables#reference-variables-using-the-ssm-parameter-store)

Honestly, the last approach would probably be the most robust. What do you think?

[aegershman]

Interesting. Just piggybacking to say I have a use-case for something similar, except using credhub as the backing store for secret retrieval. Current solution is like what you described in original post: export secrets as an environment variable with .envrc and direnv or something, but it's not ideal. Just commenting with this use-case to point out that there could be value in other secret management backends in the future 👍

[sgandon]

@rms1000watt thanks a lot for the very clear explanation.
I think this is a very interresting idea and I also believe we need some more pluggable and general mechanism for handling values coming from any kind of persistant storage.
You seem to be focused on secret values, but is there also a need for handling any kind of values ?
I would imagin some additions to the helmfile yaml definition in order to configure specific external key/values stores.

[rms1000watt]

@sgandon I could mock up something similar to how serverless does it:
https://github.com/rms1000watt/hello-world-serverless-golang/blob/master/serverless.yml#L3-L9
https://serverless.com/framework/docs/providers/aws/guide/serverless.yml/

But I think it would be more specific to the implementation like:

ssm:
  region: us-west-2
  prefix: /path/within/ssm
  # credentials for connecting to AWS would be omitted here and follow standard Golang SDK for connecting to AWS

repositories:
  - name: "my-repo"
    url: "s3://my-private-helm-chart-repo"

releases:
  - name: hello-world
    chart: my-repo/hello-world
    values:
    - hello-world:
        url: http://www.fake-url.com
        postgres_password: {{ ssm "postgres_password" }}

If and only if ssm is defined at the top level of the helmfile.yaml, then the ssm key for funcMap would work

(Add ssm to funcMap here: https://github.com/roboll/helmfile/blob/master/tmpl/context_funcs.go#L17-L40)

And to @aegershman 's point, the same structure could happen with credhub (although i've never used it and am not sure if this is makes sense with credhub).

But for other KV stores (consul, etcd) this might make sense. It's sort of like dependency injection at the YAML level.

What do you guys think?

[sgandon]

That is a great proposal !.
I was more thining of a generic way of getting access to the values with a kind of build-in object like .Values but for external KV stores.
But I think your approach with a specific function is even more flexible cause then you can have very fancy parameters if required, like in you case the path within ssm could also be a parameter of the ssm function : {{ ssm "posgres_password" "/path/within/ssm"}}.
And also mix multiple key stores if required.
This is really nice.

[mumoshu]

You've brought me a very good idea making environment variables as the interface between helmfile and kv-backend like ssm. Good job.

[mumoshu]

I think we can merge this as an experimental feature and start implementing more like this once you wrap behind a feature flag. Perhaps something like if _, exists := os.LookupEnv("SSM_PATH"); exists { ssm.Run() }?

[sgandon]

I agree this would be nice to have the ssm.Run not triggerd everytime helmfile is launched and you don't care about ssm at all.

[rms1000watt]

@sgandon @mumoshu look at the second line of the ssm.Run() function. It does that check already. I just didn’t want to make the main function messy with a bunch of checks.

[rms1000watt]

But also, I don’t mind just working on that more flexible solution proposed above. Like, we can keep this PR open (but on hold) just in case I hit a snag with that code implementation.

I really just wanted to throw something together here to start the conversation so we could structure something awesome together. ☺️

[rms1000watt]

@sgandon @mumoshu @osterman Here is a newer proposal I'm going to start working on, after some discussions with @lanmalkieri. This incorporates some of their requests to handle multiple paths. (This assumes they're all from the same AWS account. Future iterations could specify an assume role or something? But that's down the road for SSM.)

ssm:
  - region: us-west-2
    prefix: /global/path/to/ssm
    name: global
  - region: us-east-1
    prefix: /us-east-1/path/to/ssm
    name: east1

repositories:
  - name: "my-repo"
    url: "s3://my-private-helm-chart-repo"

releases:
  - name: hello-world
    chart: my-repo/hello-world
    values:
    - hello-world:
        url: http://www.fake-url.com
        postgres_password: {{ ssm "global:postgres_password" }}
        redis_password: {{ ssm "east1:redis_password" }}

[osterman]

Also, just throwing this out there... Gomplate supports lots of data sources. For an interface, it could be helpful to reference.

https://github.com/hairyhenderson/gomplate/blob/master/docs/content/datasources.md

related

• Using helmfiles, but unable to pass Environment down to helmfile.yaml #388 (comment)
• Feat: Allow simple Vault integration #392 (comment)

[rms1000watt]

@osterman the only reason I'm not crazy about this solution: #392 (comment)

is because the datasources is buried within a release. When you have multiple releases in one helmfile, you'd have to redeclare the datasources.

But yeah, I like that gomplate handles a lot of datasources already. That would give a ton of functionality out of the box.. hmm..

datasources:
  ssmGlobal: aws+smp:///global/path/to/ssm
  ssmEast1: aws+smp:///us-east-1/path/to/ssm
  myvault: vault://vault.example.com:8200///secret-engine/my-application

repositories:
  - name: "my-repo"
    url: "s3://my-private-helm-chart-repo"

releases:
  - name: hello-world
    chart: my-repo/hello-world
    values:
    - hello-world:
        url: http://www.fake-url.com
        postgres_password: {{ (ds "ssmGlobal" "postgres_password").Value }}
        redis_password: {{ (ds "ssmEast1" "redis_password").Value }}
        es_password: {{ (ds "myvault" "es_password").Value }}

Somethingggg like that.. would have to play around with it

[rms1000watt]

@osterman Gomplate might not be flexible enough in the long run for SSM. I might run with it for now though. I can imagine cases where we're pulling SSM values from different accounts and/or regions, and I'm not seeing anything in Gomplate to specify that: https://github.com/hairyhenderson/gomplate/blob/master/docs/content/datasources.md#using-awssmp-datasources

[rms1000watt]

@osterman Doing this with native gomplate syntax in the helmfile.yml feels weird:

es_password: {{ gomplate `ds=aws+smp:///dev/us-west-2/ssm` `{{ (ds "ds" "es_password").Value }}` }}

with code:

	cfg := &gomplate.Config{
		DataSources: []string{ds},
		Input:       key,
		OutputFiles: []string{tmpfile.Name()},
	}

	if err = gomplate.RunTemplates(cfg); err != nil {
		return
	}

I have it working with the code above.

But to make it more convenient for helmfile users.. I'd want to to create a translation layer so it's really simple. But at that point, then there's no real benefit to the end user if I use gomplate internally or not. So I'm actually going to look back at this syntax to make it easier for end users:

ssm:
  - region: us-west-2
    prefix: /global/path/to/ssm
    name: global
  - region: us-east-1
    prefix: /us-east-1/path/to/ssm
    name: east1

releases:
  - name: hello-world
    chart: stable/hello-world
    values:
    - hello-world:
        url: http://www.fake-url.com
        postgres_password: {{ ssm "global:postgres_password" }}
        redis_password: {{ ssm "east1:redis_password" }}

[rms1000watt]

Also, gomplate is weird to try and integrate into your code base (unless I'm totally just looking at it the wrong way). The repo doesn't expose that many public functions from their packages: https://godoc.org/github.com/hairyhenderson/gomplate

So I had to do things like write to a temporary file for each variable, instead of keeping them in memory. (gomplate seems like it was designed for just CLI use only? I'm open to learning more if anyone else has a different opinion.)

[rms1000watt]

Closing in favor of: #573

[sgandon]

I don't know about gomplate but I think this is really promizing. Thanks a lot for having a detailed look at it. You are pointing out many limitation of the library but do you think we could contribute to it to make more easily "integratable" ? cause it is nice promise to avoid duplicating some code already working and test that provide the feature we want.