ApplicationSet generators diff with CLI

[llavaud]

Summary

Beeing able to diff the ApplicationSet generators

Motivation

Today to push a modification on an application I can do an argocd app diff --revision ... in my CI to see the changes I will add.
I would love to have something similar to see the list of application that will be added or removed by the applicationset.

Proposal

Implement an argocd appset diff --revision ... ?

[bkrein-vertex]

Having something similar to what helm template does would be amazing for troubleshooting syntax! I'd love some sort of "dry-run" for AppSets.

[mabhi]

Hi @llavaud / @crenshaw-dev I am working on this issue, will submit a PR very soon.

[mabhi]

Hi @llavaud / @crenshaw-dev

Should the difference of every application resource, which is part of the ApplicationSet be displayed as output of the new command ? E.g.: the output shows the difference in the case of modification, addition and deletion respectively.

argocd appset diff guestbook-appset --revision xxxx

===== /Service default/guestbook-edit ======
52a53
>   hello: worldy

===== apps/Deployment default/guestbook-edit ======
102c102
<   replicas: 2
---
>   replicas: 1
106a107
>   hello: worldy

===== /Service default/guestbook-deploy-additional ======
52a53
>   hello: worldy

===== apps/Deployment default/guestbook-deploy-additional ======
0a1,23
> apiVersion: apps/v1
> kind: Deployment
> metadata:
>   name: guestbook-deploy-additional
>   namespace: default
> spec:
>   replicas: 1
>   revisionHistoryLimit: 3
>   ...
>     spec:
>       containers:
>       - image: gcr.io/heptio-images/ks-guestbook-demo:0.2
>       ...

===== apps/Deployment default/guestbook-deploy-delete ======
1,151d0
< apiVersion: apps/v1
< kind: Deployment
< metadata:
<   generation: 1
<   managedFields:
<   - apiVersion: apps/v1
<   ...
<     manager: kube-controller-manager
<     operation: Update
<     subresource: status
<     time: "2022-10-28T10:35:11Z"
<     ...
< spec:
<   progressDeadlineSeconds: 600
<   replicas: 1
<   ...
< status:
<   availableReplicas: 1
<   ...

Do let me know your thoughts on this.

[agaudreault]

I posted #9904 some time ago. Reposting here.

It could be useful to have a parameter so only the name of the Application are returned.

I assume that having diff in the application is expected if the applicationSet template was updated. But if only the generators were updated, then what is most important in my opinion is the result list of applications.

[llavaud]

I am more interested to get only the diff in the result list of applications like @agaudreault-jive said, but if we can have both with a parameter it would be great !

[allanger]

Hi @llavaud / @crenshaw-dev

Should the difference of every application resource, which is part of the ApplicationSet be displayed as output of the new command ? E.g.: the output shows the difference in the case of modification, addition and deletion respectively.

argocd appset diff guestbook-appset --revision xxxx

===== /Service default/guestbook-edit ======
52a53
>   hello: worldy

===== apps/Deployment default/guestbook-edit ======
102c102
<   replicas: 2
---
>   replicas: 1
106a107
>   hello: worldy

===== /Service default/guestbook-deploy-additional ======
52a53
>   hello: worldy

===== apps/Deployment default/guestbook-deploy-additional ======
0a1,23
> apiVersion: apps/v1
> kind: Deployment
> metadata:
>   name: guestbook-deploy-additional
>   namespace: default
> spec:
>   replicas: 1
>   revisionHistoryLimit: 3
>   ...
>     spec:
>       containers:
>       - image: gcr.io/heptio-images/ks-guestbook-demo:0.2
>       ...

===== apps/Deployment default/guestbook-deploy-delete ======
1,151d0
< apiVersion: apps/v1
< kind: Deployment
< metadata:
<   generation: 1
<   managedFields:
<   - apiVersion: apps/v1
<   ...
<     manager: kube-controller-manager
<     operation: Update
<     subresource: status
<     time: "2022-10-28T10:35:11Z"
<     ...
< spec:
<   progressDeadlineSeconds: 600
<   replicas: 1
<   ...
< status:
<   availableReplicas: 1
<   ...

Do let me know your thoughts on this.

I would say yes. It would be nice to have an ability to point to a source file with an application set and get a diff. For example, I am managing helm releases by the ApplicationSet. Generators are taken from a git, and they are generating helm releases that I'd like to install. But since Argo is watching only the main branch, I need to merge my application set to the main branch first, to be able to see a difference.

So it can be done on the ArgoCD "server" itself, like

argocd appset diff $APP_SET_NAME --revision $BRANC_NAME

But then it would require that the applicationset controller is rendering all the manifests on its side, and I'm afraid that if you add a lot of files to the generators folder it can have some unpleasant performance consequences. So I would rather teach a cli to generate everything on its side.

argocd appset diff $APP_SET_NAME --file $FILE_NAME

The base for comparing would be taker from the server, as it's already there, and probably rendered. But the wished state will be generated locally.

I would be happy to work on this feature myself, actually, but I'm not sure how long will it take then.

UPD: Actually, it would be possible to pass the file name without an AppSet name, because we already have a name in files. So like this

argocd appset diff --file $FILE_NAME

UPD2:
I'm sorry, haven't seen that it's already WIP.

[agaudreault]

IMO --revision does not work because if you specify a revision, how can argoCD knows where the appset is coming from? It might have been created in the UI or by kubectl. So the only option that makes sense, is to compare the current applicationset with an applicationSet manifest received in a parameter. I assume @allanger means that the --file parameter would send the file content to argoCD, and not the file path.

And obviously, both applicationSet names will need to match to be comparable.

[allanger]

@agaudreault-jive I'm not 100% sure how argoCD works here. But any ApplicationSet manifest has a name defined in it.

 apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: helm-releases
  namespace: argo-system

So if name and namespace can be taken from a file, then they could be used for diffing.

I think that the revision thing would be incredibly hard to implement and to use. For example, in my case, it would work like this:

Manifest:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: helm-releases
  namespace: argo-system
spec:
  syncPolicy:
    preserveResourcesOnDeletion: true
  generators:
    - git:
        repoURL: git@github.com:user/repo.git
        revision: main
        files:
          - path: "releases/*"

Exec argocd appset diff -f manifest.yaml --revision example-branch

And then application set controller would generate a manifest where spec.generators.git.revision is changed to one, I've set. But I don't know how it could work when you have several sources, and you need to use different revisions for them.

About the --file flag. I expect logic to be like this:

1. exec argocd appset diff --file manifest.yaml
2. argocd cli is connecting to the argo service and getting information about ApplicationSet with .meta.name and .meta.namespace
3. argocd cli tries to generate manifests from the current ApplicationSet (one provided with a flag) file
4. diff what argo server has and what cli could create.

But now I think that I'm dummy, because I still need to provide a revision somehow, otherwise generator files won't be taken from the needed branch, unless you substitute required values to all revisions in the yaml file.

[agaudreault]

@allanger I see a lot of problems if you try to generate the list of Application manifests that an applicationSet will create client-side, starting with authentication. This computing needs to be done server-side. The CLI can be used to "format" the results in different ways. What I have in mind is:

1. exec argocd appset diff --file manifest.yaml (you are right that we don't even need to specify an ApplicationSet name since it must be the same one)
2. The CLI fetches the content of --file.
3. The CLI does some validation to make sure the content is an ApplicationSet manifest. (support for a list of ApplicationSet could be possible)
4. The CLI sends the file content to argocd server as a parameter
5. Argocd server generates the Applications for the manifest received as a parameter. (it does not create resources in the cluster)
6. Argocd server get the current list of application for the ApplicationSet based on with .meta.name and .meta.namespace
7. Argocd server returns the current manifest (based on what is live in the cluster) and the generated manifest (based on the received manifest)
8. The CLI use the information to format the difference client-side

I think that this logic would work for any type of generators, and it would be easy to add different logic to show the diff without having to update the server.

So for your use case, what you would send to the server is

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: helm-releases
  namespace: argo-system
spec:
  syncPolicy:
    preserveResourcesOnDeletion: true
  generators:
    - git:
        repoURL: git@github.com:user/repo.git
        revision: <YOUR_REVISION>
        files:
          - path: "releases/*"

locally, you can use yq to update the revision field, create a new file, then use it as the value for --file. This way you get the diff with what is currently live on the server, and what it would be if <YOUR_REVISION> is merged.

[allanger]

@agaudreault-jive

locally, you can use yq to update the revision field

Aye, that's exactly what I meant by substituting, just didn't want to specify that hard :)

My only concern about server side rendering for diffs is performance.

I have about 20 generators files in the 'releases' folder, (I need to say, that it's only a fun project, so I don't have a very powerful machine), but was able to spot CPU peaks when the AppSet was reconciling.
I'm afraid what's going to happen if several guys in the same time are adding 1000 different generators files, and diffing.

Together with #12314 it may be even cooler.

Setting limits to amount of generators files seem not to be very flexible because in the end, I may want 2000 generators files, but I may not be ready for them, when someone is running diff.

[MeNsaaH]

Right now, what I'm doing is listing all the apps belonging to the appset and then running the diffs on those apps. Since argocd don't have a way of listing all the apps of an appset I'm doing it using the method stated #13309

[llavaud]

Right now, what I'm doing is listing all the apps belonging to the appset and then running the diffs on those apps. Since argocd don't have a way of listing all the apps of an appset I'm doing it using the method stated #13309

My problem is not really to have the diff of existing apps, but to know if the applicationSet will add or remove some of them after an update in the generator configuration.
If I make a mistake in the generator configuration that will remove all apps, I would like to see this before merging my PR ;)

[crenshaw-dev]

I have about 20 generators files in the 'releases' folder, (I need to say, that it's only a fun project, so I don't have a very powerful machine), but was able to spot CPU peaks when the AppSet was reconciling.

@allanger the ApplicationSet controller has efficiency problems in general. Folks have slowly chipped away at those, so I think things are improving.

Based on @agaudreault-jive's design above, I think the API server would be taking on most of the load, by running ApplicationSet code as library code. So you'd end up scaling the API server, not the ApplicationSet controller.

But either way, scalability is a concern. I'd suggest that we implement the feature, start with human users, and then knock out performance issues one by one as people start using the feature via automation (e.g. diff generation for PRs on git-hosted AppSet manifests).

[blakebarnett]

It seems like having the diff performed by the server is the barrier here. Would it be a better compromise to just add a appset render or appset template command first? That would probably solve most of the concerns and wouldn't need to involve the server. It would especially help with initial app-set creation and doing a client-side diff when needed seems reasonable.

[crenshaw-dev]

@blakebarnett what you anticipate the appset render or appset template commands doing? Without the input parameters (which only the server-side knows), I don't think you can do anything really useful.

[blakebarnett]

Ah true, I was only considering the git-files generator. But you're right I would love to have the matrix generator show cluster parameters, etc as well. Would it be less overhead on the cluster to only do a dry-run vs a diff or is that a negligible difference?

[benjamink]

Couldn't the CLI fetch the server-side data to generate a dry-run output if you're logged in & within the appropriate context/namespace? Seems like it would just gather the current cluster secrets & any other generator data then render the Application YAMLs to stdout.

[crenshaw-dev]

Would it be less overhead on the cluster to only do a dry-run vs a diff or is that a negligible difference?

I think the difference would be negligible.

Seems like it would just gather the current cluster secrets & any other generator data then render the Application YAMLs to stdout.

That's one way... personally I feel better about leaving the access secrets server-side and just generating the parameters and pass them back for templating.

[benjamink]

Yes, I wasn't suggesting the secret data be pulled/rendered (at least not without explicitly allowing/requesting them). I'd just want labels/annotations & the standard values like name, server, etc.

[DerekTBrown]

I am wondering if a lot of folks' usecases would be solved if we were able to pull out just the names of the apps:

#16334

[benjamink]

I just want it to dump the Application objects an ApplicationSet would produce. I can filter/parse from there.

[benjamink]

I am wondering if a lot of folks' usecases would be solved if we were able to pull out just the names of the apps:

#16334

This doesn't help troubleshoot templating logic too like when using paths or labels, etc. 75% of the time I want to test/verify my templating logic works with my generators, not just get app names.

[ForbiddenEra]

I am wondering if a lot of folks' usecases would be solved if we were able to pull out just the names of the apps:
#16334

This doesn't help troubleshoot templating logic too like when using paths or labels, etc. 75% of the time I want to test/verify my templating logic works with my generators, not just get app names.

I agree; I often just go to online template playgrounds for the appropriate templating implementation for things like this but even there I don't have full insight into the initial objects presented so it's not even that useful and definitely a hacky workaround when I should just be able to get the output from argo on the CLI or something easy so I don't have to constantly test on a live argo instance whether my templating logic and stuff even works :/