applicationset controller: git provider does not catch new changes when using webhook

[tomjohnburton]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I am using the git provider with an application set. The config is like so:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: devnet
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: REDACTED
        revision: HEAD
        files:
          - path: "networks/devnet/*.kubernetes.json"
  template:
    metadata:
      name: "{{network}}"
      namespace: argocd
    spec:
      project: "web-services"
      syncPolicy:
        automated:
          selfHeal: true
          prune: true
          allowEmpty: false
      sources:
        ...
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{network}}"

I have set up the webhook and it is correctly pinging the applicationset controller on every push to the repo.

argocd-applicationset-controller-5f948b9464-ww66g applicationset-controller {"level":"info","msg":"Received push event repo: REDACTED, revision: main, touchedHead: true","time":"2023-09-06T13:26:28Z"}

The problem is, is that it doesn't seem see the latest changes. Not until 3minutes later when the normal cycle completes.

To Reproduce

Create an application set that uses the git provider and sources variables from a json file.
Create a webhook
Push a new config file

Expected behavior

The application set should see that a new file has been added.

Screenshots

Version

{
    "Version": "v2.8.2+dbdfc71",
    "BuildDate": "2023-08-24T20:05:39Z",
    "GitCommit": "dbdfc712702ce2f781910a795d2e5385a4f5a0f9",
    "GitTreeState": "clean",
    "GoVersion": "go1.20.6",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KustomizeVersion": "v5.1.0 2023-06-19T16:58:18Z",
    "HelmVersion": "v3.12.1+gf32a527",
    "KubectlVersion": "v0.24.2",
    "JsonnetVersion": "v0.20.0"
}

Logs

As you can see the appset doesn't see new applications until some mins later

argocd-applicationset-controller-5f948b9464-ww66g applicationset-controller {"level":"info","msg":"Received push event repo: REDACTED, revision: main, touchedHead: true","time":"2023-09-06T13:26:28Z"}
argocd-applicationset-controller-5f948b9464-ww66g applicationset-controller {"generator":{"git":{"repoURL":"REDACTED","files":[{"path":"networks/devnet/*.kubernetes.json"}],"revision":"HEAD","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 0 applications","time":"2023-09-06T13:26:29Z"}
...
argocd-applicationset-controller-5f948b9464-ww66g applicationset-controller {"generator":{"git":{"repoURL":"REDACTED","files":[{"path":"networks/devnet/*.kubernetes.json"}],"revision":"HEAD","template":{"metadata":{},"spec":{"destination":{},"project":""}}}},"level":"info","msg":"generated 1 applications","time":"2023-09-06T13:27:09Z"}

[rumstead]

Can you see if this #16062 fixed your issue? It doesn't look to be in a tagged version of 2.8 or 2.9 yet.

[jgwest]

Appears resolved based on above, feel free to reopen if still present.

[pre]

@jgwest @rumstead The fix doesn't work, applicationset does not refresh applications after receiving the webhook. Could you help us figure this our for good? Issue is still in in 2.8, 2.9, 2.10 and 2.11.

There's a longer discussion in issue #15427

• Webhook not refreshing applications #15427 (comment)
• https://cloud-native.slack.com/archives/C04SURUPDL2/p1722330612869969