-
Notifications
You must be signed in to change notification settings - Fork 564
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: helmfile build
to produce rendered and flattened state file for GitOps
#780
Comments
cc/ @sstarcher @osterman @bitsofinfo @Stono I'm considering to enhance Helmfile this way for better auditing of Helmfile deployments with GitOps, and better security by optionally limiting CI to have no access to production clusters and exposing no github webhook endpoint. WDYT? |
I dig it!!! Solid idea. Some questions:
|
I fully agree that if we can remove or make all templating of helmfiles first-class citizens it would be much better and easier to handle and debug. Templating of a helmfile itself is essentially a crutch and a dirty work around. Embedded secrets.yaml could result in a massive release set. It would likely be better to create it as a separate object and to reference them as refs. Everything in ReleaseSet should be cleanly diff-able. |
Your assumption is correct :) Btw, implementation-wise it should be a matter of Helmfile checking existence of
Exactly. I believe that's important.
Seems like a valid feature request! I like it as much as I like
Yep, I believe so too.
Yep. You'll still need it. FWIW, I'm currently forwarding |
I do envision that, after this enhancement, helmfile can be used in combination with other templating tool based on jsonnet, Lua, ECMAScript or anything else. The minimum feature of Helmfile will be a solid ReleaseSet reconciler that supports DAG, hooks, and many pluggable values/secrets backends.
I agree, but I'm not yet sure how it should fully work. Anything other than
Perhaps small Maybe it's fine to omit the ability to embed small Then the remaining problem is how Helmfile should get the URL of a secrets.yaml. Let's say you have a "source" repo that contains releases:
- name: myapp
chart: mychart
secrets:
- secrets.yaml
Can we safely assume that |
Wait... so you are going to remove all templating capabilities in helmfiles? that would be bad as I really really utilize this feature. Its a huge reason I am using this tool OR... are u just saying that by now being able to consume this new |
My assumption was that the ReleaseSet would be store in Kubernetes as a CR and my recommendation was for the secrets to be treated the same way as a CR inside of Kubernetes so the github repo would not need to be referenced. |
Not necessarily, i think this should be pretty simple as you laid out in the top post.
Where Keep it super simple and don't build in any use-case specific assumptions. |
No. I'll keep maintaining the
That's it!
You need two sets of things in Git:
In your model, how would |
Yep, that makes sense. But how would you version-control and install secrets into your cluster that are referenced from the CR then? |
Perfect.
I don't uses I get what you are describing though and I can see the references being a relevant concern. However, that said, all I am saying is that if
|
Okay. So the purpose of sops-encrypted secrets embedded in You've mentioned two modes of If you have no |
I totally understand, however I'd still advocate for a literal output |
@bitsofinfo Thanks. That makes sense :) I would start with the mode 1 only then. One reason is that mode 2 can be implemented on top of that without affecting overall helmfile architecture, and another is without |
You could take the helm approach and version things incrementally or you could take the kops approach and only keep the latest, but be able to regenerate it from the original git repo. |
Do you mean that
Do you mean keeping only the latest decrypted secrets in the cluster? |
I'll add flags to allow setting annotations and labels in the resulting resource. You'll use annotations and labels to propagate any metadata from CI to CD. Perhaps it will look like |
Those options args look good to me. So if we want to specify multiple annotations/labels does the operator declare the arguments multiple times or comma separated kv pairs? What about an option to capture the "build" command arguments to add as an additional annotation? |
@bitsofinfo Probably I'd start with a simpler implementation which is likely to require a flag per one k-v pair. Extending it it accept comma-separated kvs later should not be hard. A flag to capture |
1. Added `helmfile build` command to print final state Motivation: useful for debugging purposes and some CI scenarios Ref #780 2. Template interpolation is now recursive (you can cross-reference release fields) like: ```yaml templates: release: name: {{`app-{{ .Release.Namespace }}`}} namespace: {{`{{ .Release.Labels.ns }}`}} labels: ns: dev ``` 3. Experimental: Added some boolean release fields interpolation in templates: ```yaml templates: release: name: {{`app-{{ .Release.Namespace }}`}} namespace: dev installedTemplate: {{`{{ eq .Release.Namespace "dev" }}`}} ``` Resolves #818 4. Added more template interpolations: Labels, SetValues 5. Added template interpolation for inline Values 6. Added `helmfile list` command to print target releases in simple tabular form 7. Added release names in some `helm` output messages, e.g.: `Comparing release=%v, chart=%v`
I've been looking for a way to implement a gitops workflow using helmfile so I was very excited to see this issues thread. However IMO storing the secrets in K8S would be less than ideal in many scenarios. One of the things I love about the helmfile/helm secrets approach is it's completely agnostic in-terms of the backend store. Many folks have compliance issues storing secrets in K8S base64 and therefore use other backends like vault or the like. The current agnosticism of helmfile where it leaves it up to the helm chart to ultimately determine where those secrets are stored is something I think should be maintained. Secondly, this approach would require the application repo (where the While using a git reference resolves the above issues, I can't say I'm super fond of this approach either. One of the benefits of using sops is that it enables tracking of secrets. I think it would be ideal when a PR is created in the gitops repo, it should be viewable in that PR whether a secret was changed. Perhaps there is way to accomplish this with a git ref by only updating the ref when the secret value changed? (Although this doesn't seem super elegant to point to old shas in scenarios where there is no change). Secondly, requiring git access to the original repo and requiring git to run helmfile in the gitops repo seems a bit bloated IMO. Personally I would be fond of baking the necessary sops into the rendered file along with the encrypted secret. Even if the secret is gonna be long in certain scenarios, the simplicity, traceability, transparency and security of this approach is something I personally would really appreciate. I can't thank you enough for this amazing tool @mumoshu! |
@aweis89 Hey! Thanks a lot for your detailed feedback. I fully agree and I think that's where evolving Helmfile's
Regardless of the vals integration, as you pointed out, you still get raw secrets in But the original https://github.com/variantdev/vals#helm So, I believe we only need two things done to fully support GitOps in Helmfile.
|
It definitely seems like Judging the from the fact that value references to files don't get interpolated with
|
Your view seems definitely correct 😉 |
Problem
Helmfile's advanced templating features consisting of "release templates", "sub-helmfiles", "state values", "environments" are all GREAT for making your desired states DRY by avoiding repeated configs across environments.
But advanced templating has downside - less predictable results. This makes it very hard to debug and audit deployments made by Helmfile.
Helmfile Diff
helmfile diff
helps by previewing changes to be made to the cluster before deploying. But you need real cluster connectivity from your CI to run diff, which is a security concern. And sometimes, you still need another level of visibility to "intermediate" result produced by Helmfile - rendered helmfile.yaml template.Debug logs, Dry-Run, Captured Releases, ...
People usually use
--helmfile --log-level=debug
for that purpose.helmfile --dry-run
#118 is being discussed for additional visibility and better U/X.@bitsofinfo has proposed ability to write rendered releases to the local disk in #752. This sounds like a good idea for visibility and auditability. But one gotcha is that it may lack correctness due to how Helmfile is implemented today. Helmfile as of today applies rendered releases in-place. So if Helmfile is enhanced to write a slightly different representation of releases to disk, it's theoretically possible that what Helmfile wrote and what Helmfile applied differs, which makes the whole purpose of the feature meaningless.
GitOps
GitOps is recently emerging for better audited and reproducible deployments. But it doesn't really help Helmfile.
Highly templatized
helmfile.yaml
commited into Git for GitOps doesn't automatically provide useful git-diff - because the diff isn't based on rendered helmfile.yaml. You need either CI or your brain to render the helmfile.yaml template and take useful diff to give confidence that a new helmfile deployment doesn't break your system, which is hard.Solution
I'm proposing the combination of the followings:
helmfile build
that writes a ReleaseSetA ReleaseSet is produced from the ordinary helmfile.yaml template you all have today and looks like:
All the releases, values and encrypted secrets and the selected environment included in your helmfileand its sub-helmfiles and will be rendered and flattened.
The environment will be erased hence there's no occurrence of "environment" in the build output.
We need a DAG of "dependencies"(#715 and #723) to flatten sub-helmfiles.
The typical usage of
helmfile build
would look as follows.On your laptop(for testing) or CI(for production),
helmfile build
is run to produce a flattened ReleaseSet:On your laptop(for testing) or CD(for production), the CD pipeline recognizes
approved: "false"
in your releaset set and runshelmfile diff
so that you can review changes at K8s manifests level for approval:$ helmfile -f helmfile.prod.yaml diff | some-cmd-to-send-github-comment
You approve it by commenting on the PR to the "helmfile-template-repo" with
/approve
.On your laptop(for testing), or CI triggered via a GitHub pull request comment(for production), some command to modify the ReleasetSet to have a
approved: "true"
annotation:On your laptop or CD, the CD pipeline recognizes
approved: "true"
and therefore runshelmfile apply
:This way, you have complete audit logs of what's being rendered and deployed by Helmfile in Git.
The text was updated successfully, but these errors were encountered: