Show diff for actual state vs desired state

[dudicoco]

Hi,
At the moment, if you make any manual changes to resources (not via helm) helm diff will not reflect these changes.
I suggest that the output should reflect the desired vs actual state of the resources, just like running terraform plan.

[databus23]

That would be nice but its not straightforward. The problem is that Kubernetes controllers/operators add a lot of fields when working with the resources. For example the common metadata section alone is modified a lot. Also a lot of fields in the Spec section of almost all resources are filled with default values (e.g. spec.revisionHistoryLimit of Deployments)
So its not easy to create a meaningful diff. You would always see a lot of noise that would hide the actual changes.

[dudicoco]

@databus23 what about using kubectl-diff behind the scenes after rendering the templates?

[databus23]

what about using kubectl-diff behind the scenes after rendering the templates?

Not sure I follow you. What do you mean by that? Against what would you diff the rendered templates? The actual state from the cluster is littered with dozens of default values that are not in the rendered templates. This would in almost all cases create a huge diff that has no meaning.

[dudicoco]

What I mean is that when running helm-diff the following will happen:

1. The package will be rendered.
2. helm diff will run kubectl diff package.yaml

Example of output which includes the diff from the actual state of the clustered:

kubectl diff -f test.yaml
diff -u -N /var/folders/fj/scl5dj6d13b7mb34kxzwwfp80000gn/T/LIVE-912218788/v1.Pod.default.liveness-exec /var/folders/fj/scl5dj6d13b7mb34kxzwwfp80000gn/T/MERGED-153443763/v1.Pod.default.liveness-exec
--- /var/folders/fj/scl5dj6d13b7mb34kxzwwfp80000gn/T/LIVE-912218788/v1.Pod.default.liveness-exec	2020-01-20 16:04:27.000000000 +0200
+++ /var/folders/fj/scl5dj6d13b7mb34kxzwwfp80000gn/T/MERGED-153443763/v1.Pod.default.liveness-exec	2020-01-20 16:04:27.000000000 +0200
@@ -7,7 +7,7 @@
     kubernetes.io/psp: eks.privileged
   creationTimestamp: "2020-01-20T13:42:14Z"
   labels:
-    test: liveness-exec
+    test: liveness
   name: liveness-exec
   namespace: default
   resourceVersion: "34961066"

[corradomatt]

I tested this using kubectl diff and was able to get a simple diff rendering. Here's what I did...

1. Installed 6 helm charts using helmfile.
2. Manually patched the image field on 1 of the deployments of 1 helm chart.
3. Output the manifest templates from our helmfile using helmfile template > test.yaml.
4. Ran kubectl diff -f test.yaml

The results were to show only 2 differences:

1. An increment of the generation field of the deployment.
2. The image field change as expected.

Seems to me that this suggestion could be a viable solution.

(btw - thank you for this package!)

[TJM]

helm/helm-www#521 <-- I filed this in the wrong place, guess you already know about this :)

[TJM]

For what its worth, I tried what @corradomatt mentioned above, but had pages of output, likely due to everything not being in one namespace. Apparently, to do this I would have to separate the comparison by namespace (somehow?).

[corradomatt]

I've been using the following to run diffs now...

# using helmfile
helmfile -e myenv template | kubectl diff -f - | bat -l diff -

# using helm directly
helm template releaseName chart | kubectl diff -f - | bat -l diff -

You could easily apply your namespace flags to these commands and/or supply a selector to helmfile to limit the output. This also uses the bat utility (which is amazing btw).

[sstarcher]

On this note now that helm 3 is doing a 3 way merge could we not look at the code they are using for the merge.

[max-rocket-internet]

I would just like to also mention the importance of having this feature 🙂

Before, with Helm2, showing a diff excluding manual changes in the cluster was fine because Helm2 would also ignore these. But now, with Helm3 and the new 3-way merge, we can't really be sure of what will be applied to the cluster when using this plugin. If the upgrade is going to revert some manual changes then this plugin won't show them.

[sstarcher]

Agreed this is much more important now. It is also needed to help debug why a 3 way merges in helm maybe not acting as we expect.

[TBBle]

To support helm/helm#7649, it'd be handy if the "actual state" diff side in this feature could be one not provided by Helm. Obviously, in such a case, only objects created by the Helm release would be considered, although it might be handy to explicitly flag objects that exist but are missing the appropriate annotations (although they'd show up in the diff anyway).

[max-rocket-internet]

@mumoshu any thoughts on this issue? Is it hard to fix? Or impossible?

[bergerx]

Same issue is impacting helmsman (and i believe helmfile) which used to use helm-diff plugin to decide if a "helm upgrade" will trigger any changes or not. And if helmsman doesnt see any changes to be applied it just skips the "helm upgrade" step. But in case of any out-of-helm changes are involved, it becomes blind due to the fact that helm-diff calculates the diff against "latest deployed version of a release". You can see some more context in Praqma/helmsman#487 (comment)

Here the issue is not how helmsman or helmfile uses helm-diff, its actually the users' expectation when they are using the helm-diff plugin. Most of the cases they do care about "which actual changes to be applied", rather than "what would be the change if there has been no manual changes in the cluster". But either may be confusing for the helm-diff users, so maybe introducing a new flag and providing both functionality is the best path forward.

[bergerx]

I've just looked into helm diff command usage a bit.

helm diff upgrade is designed in a way to imitate helm upgrade, most of the time i just add a diff keyword in between to see if helm upgrade will introduce changes or not. I know this is not really how the helm-diff is actually designed, but in most cases this will be the expectation.

So here is a sample case that helm diff upgrade command is clearly reporting a misleading output if we just forget the fact that helm-diff checks only against "latest deployed version of a release" rather than the "actual state". The current behaviour of helm-diff was aligned with helm 2's behaviour, but in helm 3 they implemented a 3-way-merge and not it works more like "kubectl apply". I believe "helm diff upgrade" subcommand should follow this design change in helm upstream.

Prepare a test environment

cd /tmp
kubectl create ns test
helm create test
helm install test ./test
kubectl delete deploy test

Now, here is what helm diff reports (it clearly reports as if no action will be taken by helm upgrade, true for helm 2, not for helm 3):

/tmp $ helm diff upgrade test ./test
/tmp $ 

And now, lets see if that's true (helm diff reported me a misleading update result):

/tmp $ helm upgrade test ./test --debug
upgrade.go:121: [debug] preparing upgrade for test
upgrade.go:129: [debug] performing update for test
upgrade.go:308: [debug] creating upgraded release for test
client.go:173: [debug] checking 3 resources for changes
client.go:436: [debug] Looks like there are no changes for ServiceAccount "test"
client.go:436: [debug] Looks like there are no changes for Service "test"
client.go:194: [debug] Created a new Deployment called "test" in bekir-test            -----------> Helm actually created the missing Deployment

upgrade.go:136: [debug] updating status for upgraded release for test
...

[dudicoco]

@bergerx I believe everyone here is already in agreement about the behaviour of helm-diff, but since it doesn't seem like @databus23 is working on this, someone else will have to take the initiative and open a PR to fix the behaviour.

[bergerx]

Ah, thanks for the clarification, so we just need a volunteer to work on the PR.

[mumoshu]

@sstarcher @max-rocket-internet Finally I could read some code related to how three-way merge patch is done in helm3: https://github.com/helm/helm/pull/6124/files but I'm not yet sure how we could leverage it. Maybe we need to add a some kind of --dry-run flag to helm upgrade so that it prints the generated patch, which could be used by helm-diff somehow to compute the diff?

[mumoshu]

@dudicoco @corradomatt Thanks a lot for suggesting the use of kubectl-diff!

It does seem to work in your examples, but I'm not entirely sure if there's any possible edge-case.

If helm template | kubectl diff always works as we might have expected, I believe that three-way merge patch among cluster-state/current-release/new-manifests(helm 3) must be theoretically equivalent to a two-way (strategic) merge patch between cluster-state/new-manifests(this is what I believe kubectl-apply is doing). I'm not aware of any information that backs the theory though.

My thought may distill into the same concern @databus23 has expressed earlier. kubectl-diff actually seem to show only meaningful diff. But that's not all. I'm not yet sure if the use of it in combination with helm template is theoretically correct.

[mumoshu]

I think the easiest path would be to send a pull request to the upstream as described in #176 (comment), and just use that instead of helm-diff. helm-diff will remain as the way to show diff between the current release and the new release, but tools like helmsman and helmfile should have an option or(sane default) to let the tool use the new helm command, so that the existence of the generated three-way merge patch can be used to detect if there's any "real" change to be applied or not.

[max-rocket-internet]

Maybe we need to add a some kind of --dry-run flag to helm upgrade so that it prints the generated patch

Should I create an issue in the Helm repo for this?

[mumoshu]

@max-rocket-internet That would be super helpful for all of us 👍

[max-rocket-internet]

I find the maintainers of helm to be terse at times but let's see how it goes: helm/helm#8718
Please add your support there 🤞

[luisdavim]

did anyone had a look at https://github.com/bacongobbler/helm-patchdiff from helm/helm#8718 (comment) ?

[R-omk]

Unfortunately I only now found out that this issue exists.
In January 2020, I was doing some research. Here's walkthrough I have left from that moment, maybe it will be useful to someone

helm repo add bitnami https://charts.bitnami.com/bitnami

helm install diff-example   --set db.host=mymariadb,db.port=3306  bitnami/phpmyadmin
helm upgrade diff-example   --set db.host=mysql,db.port=9999  bitnami/phpmyadmin

kubectl patch deployment diff-example-phpmyadmin -p '{"spec":{"template":{"spec":{"containers":[{"name":"phpmyadmin","image":"docker.io/bitnami/phpmyadmin:fake"}]}}}}'

# compare last release with actual
helm get  manifest diff-example | kubectl diff -f -

# what the plugin thinks about the changes
helm diff upgrade diff-example --reuse-values    --set db.host=mysql,db.port=9997  bitnami/phpmyadmin

#show what really changes after the update (yq v4)
helm upgrade diff-example --dry-run --reuse-values   --set db.host=mysql,db.port=9997  bitnami/phpmyadmin -o yaml | yq e '.manifest' -  | kubectl diff -f -

# this update silently overwrite the changes made through the patch (image)
helm upgrade diff-example --reuse-values   --set db.host=mysql,db.port=9997  bitnami/phpmyadmin

helm diff revision diff-example   2 3
helm uninstall diff-example

[luisdavim]

Any updates on this?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[colearendt]

Bump

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[colearendt]

Bump

[bilenkis]

Please, any updates on this?

[max-rocket-internet]

@mumoshu thanks for your work here.

I just have one question after testing with latest helm-diff release though: are annotations ignored?

I can install a random chart:

helm install test-release deliveryhero/toxiproxy

Make a manual change:

kubectl set image deployment/test-release-toxiproxy toxiproxy=shopify/toxiproxy:2.1.3

And helm-diff shows the change now 🎉

helm diff --three-way-merge upgrade test-release deliveryhero/toxiproxy

...
    - -config
    - /home/toxiproxy/proxies/config.json
-         image: shopify/toxiproxy:2.1.3
+         image: shopify/toxiproxy:2.1.4
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 2
...

But if I add a manual annotation:

kubectl annotate deployment test-release-toxiproxy test-annotation=yes

It is not shown in the diff output 🤔

[mumoshu]

@max-rocket-internet I think that's intended behavior / that's how three-way merge works. Annotations will be merged by keys among the current state, the latest manifest stored in the helm release, and the desired manifests rendered.
That means helm won't revert changes on test-annotation as long as you don't have test-annotation in your helm template.

[max-rocket-internet]

That means helm won't revert changes on test-annotation as long as you don't have test-annotation in your helm template.

OK yes, you are right.

Thanks again @mumoshu 💖

[jimbali]

Was there ever any actual practical solution or workaround to this?

[chadlwilson]

@jimbali --three-way-merge in v3.3.0+ of the plugin I believe? https://github.com/databus23/helm-diff#usage

[jimbali]

thanks @chadlwilson

[jimbali]

Still doesn't seem to pick up manual changes for me though! 🤷‍♂️

[chadlwilson]

If you think something isn't working perhaps it is best to open a separate issue with your scenario. I find folks are sometimes confused with how Helm three-way merge works as in the comment above. #176 (comment)

If Helm 3 is not going to revert your manual change on apply (which often it will not), it shouldn't show in the output of helm diff. At least if working as I understand was intended.

Are you sure your manual change is something Helm is going to revert if applied?

[jimbali]

I'd have thought that it would do, as the liveness and readiness probes were deleted manually, and they are explicitly present in the chart.

[chadlwilson]

Test it with Helm. I think you will find it will not reinstate such manual changes with Helm 3s default 3 way merge behavior.