Inconsistent .Capabilities.APIVersions between helm install and argocd's helm template

[tevert]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

We're attempting to write template conditionals around resource upgrades, at the resource specific level - for example:

{{- if .Capabilities.APIVersions.Has "networking.k8s.io/v1/Ingress" -}}
apiVersion: networking.k8s.io/v1
{{- else if .Capabilities.APIVersions.Has "networking.k8s.io/v1beta1/Ingress" -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end -}}
kind: Ingress

However, these conditionals always resolve false, because the .Capabilities.APIVersions that is exposed when ArgoCD runs helm template only includes the API-level versions, and not the Resource-level versions.

Earlier github issues, and our own testing, has shown that conditionals against API-level checks work, so the following snippet is OK (for now):

{{- if .Capabilities.APIVersions.Has "networking.k8s.io/v1" -}}
apiVersion: networking.k8s.io/v1
{{- else if .Capabilities.APIVersions.Has "networking.k8s.io/v1beta1" -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end -}}
kind: Ingress

However, this isn't going to be OK forever, due to scenarios such as this one: https://gitea.com/gitea/helm-chart/issues/77

The inconsistency isn't great, but the specific problem of switching for resource API versions with fine-grained detail is really more important.

To Reproduce
Make a chart with a single small template:

helm create apiversiontest

apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ .Release.Name }}
data:
  apiVersions.txt: |
    {{ .Capabilities.APIVersions }}

Add to your ArgoCD by whatever method of your choosing, and also helm install -n testing apiversiontest-fromhelm ./apiversiontest/

$ kubectl get configmap -n testing apiversiontest-fromhelm -o yaml
apiVersion: v1
data:
  apiVersions.txt: |
    [cert-manager.io/v1/ClusterIssuer fleet.cattle.io/v1alpha1/GitRepoRestriction vpcresources.k8s.aws/v1beta1/SecurityGroupPolicy.... <truncated for brevity> 

$ kubectl get configmap -n testing apiversiontest-fromargocd -o yaml
apiVersion: v1
data:
  apiVersions.txt: |
    [v1 admissionregistration.k8s.io/v1 admissionregistration.k8s.io/v1beta1 internal.apiserver.k8s.io/v1alpha1.... <truncated for brevity> 

Note that the one created from plain ol' helm is much larger and contains entries for each individual resource.

Expected behavior

They don't need to be in the same order, but it would be cool to roughly the same values.

Screenshots

If applicable, add screenshots to help explain your problem.

Version

Don't have the CLI installed, but from the web UI: v2.0.0+f5119c0

Paste the output from `argocd version` here.

Logs

Paste any relevant application logs here.

[Lirt]

Hello, I had similar issue today and what I did to fix it was:

1. Open Web UI.
2. Go to Settings -> Clusters.
3. Open Cluster that has issue you described.
4. Hit button Invalidate Cache.
5. Open Application list.
6. Find Application that cannot be synced.
7. Hit Hard Refresh button (needs to be Hard).

If it doesn't help, then it's not the same issue (but symtoms identical).

[cilindrox]

Not following @Lirt - how does invalidating the cache result on the apiserver bundled with argo returning a newer version?

[Lirt]

Hmm I think my scenario was different, I apologise.

In my case I installed cert-manager to cluster. The helm chart I was installing was checking API version capabilities of cert-manager (for example cert-manager.io/v1 or cert-manager.io/v1alpha3). I encountered templating error because argo didn't see cert-manager.io/v1 in the capabilities (the api resource / CRD was installed in the cluster). I did the trick above and suddenly argo seen those capabilities.

[cilindrox]

got it - and agreed, seems those versions are coming from the different chart versions, and thus your suggestion for refreshing the dep cache makes sense. This issue however is related to the apiserver and the versions available there vs what argo's bundling and checking the helm template against.

[sspreitzer]

Same problem here with GitLab helm chart version 5.2.1.
The ingress apiVersion is included via a template that evaluates .Capabilities.APIVerisons and sets the ingress apiVersion accordingly.

https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/templates/_helpers.tpl#L518

[cilindrox]

Seems upgrading argo to v2.1.1 works on k8s v1.21 - still testing but the latest release might've updated the bundled deps.

edit: It did not help - issue's still present.

[sspreitzer]

I am still seeing this issue in the following combination:

• argo-cd helm chart version 3.17.5
• argo-cd version v2.1.1+aab9542
• GitLab helm chart version 5.2.1
• Kubernetes version 1.22.1

There is no avail with:

• Invalidating the cluster cache
• Hard refreshing the app

[Ragatzino]

I tried to add a feature to solve this kind of problem : helm/helm#10108. Yet it seems like it is not the good way around. Have you got another idea ? 🍵 🦖

[sathieu]

We need some ArgoCD devs input here. @alexmt, sorry to disturb you, but could you please take a look at helm/helm#10108 (comment)? Is the proposed path acceptable from the ArgoCD perspective?

[donovanmuller]

Same issue with the following combination with the Keycloak Chart:

• argo-cd Chart version 3.21.0
• keycloak Chart version 15.0.2
• Kubernetes version 1.21.2 (EKS)

{
    "Version": "v2.1.2+7af9dfb",
    "BuildDate": "2021-09-02T18:05:23Z",
    "GitCommit": "7af9dfb3524c13e941ab604e36e49a617fe47d2e",
    "GitTreeState": "clean",
    "GoVersion": "go1.16.5",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KsonnetVersion": "v0.13.1",
    "KustomizeVersion": "v4.2.0 2021-06-30T22:49:26Z",
    "HelmVersion": "v3.6.0+g7f2df64",
    "KubectlVersion": "v0.21.0",
    "JsonnetVersion": "v0.17.0"
}

Rendering templates locally using helm install keycloak . --values ... --dry-run correctly evaluates {{- if .Capabilities.APIVersions.Has "networking.k8s.io/v1/Ingress" -}}

[sathieu]

I've filled a new bug, with a summary of problems about APIVersions (and a new one). See #7291.