OCI helm charts hosted in AWS OCI broken after upgrade from v2.0.4 to v2.2.2 #8214

pthornton · 2022-01-18T22:35:39Z

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

[ x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
[ x] I've included steps to reproduce the bug.
[x ] I've pasted the output of argocd version.

Describe the bug
Application Chart:

apiVersion: v3
name: react-frontend-service
version: 0.2.7-latest-a6f42056
dependencies:
  - name: react-frontend-service
    repository: oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-react-frontend-service
    version: 0.2.7-latest-a6f42056

$ argocd --server argocd-test.falkor.rocks --grpc-web repo list
TYPE  NAME           REPO                                                              INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE  PROJECT
helm  us-west-2-oci  712053168757.dkr.ecr.us-west-2.amazonaws.com                      false     true   false  true   Successful

All applications referencing AWS oci charts fail after upgrade. It appears "helm registry login" is only called when add the helm repo using "helm repo add" from cronjob. The "helm dependency update" fails and no login messages appear in repoServer logs.

Error message from UI:

rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: could not download oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-react-frontend-service/react-frontend-service: pulling from host 712053168757.dkr.ecr.us-west-2.amazonaws.com failed with status code [manifests 0.2.7-latest-a6f42056]: 401 Unauthorized

Repo server log:

time="2022-01-18T22:25:22Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: could not download oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-react-frontend-service/react-frontend-service: pulling from host 712053168757.dkr.ecr.us-west-2.amazonaws.com failed with status code [manifests 0.2.7-latest-a6f42056]: 401 Unauthorized" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-01-18T22:30:22Z" grpc.service=repository.RepoServerService grpc.start_time="2022-01-18T22:25:22Z" grpc.time_ms=2.77 span.kind=server system=grpc

If I bash into the repo server and do "helm registry login" and then "help dependency build" in the /tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev/react-frontend-service/helm_base folder, it successfully pulls the OCI helm chart. So no permission issues. I'm using username/password/url from the argocd cli helm repo add created kubernetes secret repo-4159968007.

This also happens if I upgrade to 2.1.7 instead of 2.2.2. Post in argocd slack also: slack

To Reproduce

Add Application using chart with reference to AWS OCI helm chart.

Expected behavior

Helm charts should pull from AWS OCI after successful "helm registry login" executed before running "helm dependency update".

Screenshots

If applicable, add screenshots to help explain your problem.

Version

argocd: v2.2.2+03b17e0.dirty
  BuildDate: 2022-01-01T16:53:02Z
  GitCommit: 03b17e0233e64787ffb5fcf65c740cc2a20822ba
  GitTreeState: dirty
  GoVersion: go1.17.5
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.2.2+03b17e0
  BuildDate: 2022-01-01T06:27:52Z
  GitCommit: 03b17e0233e64787ffb5fcf65c740cc2a20822ba
  GitTreeState: clean
  GoVersion: go1.16.11
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
  Helm Version: v3.7.1+g1d11fcb
  Kubectl Version: v0.22.2
  Jsonnet Version: v0.17.0

Logs

time="2022-01-18T22:35:05Z" level=info msg=Trace args="[git fetch origin --tags --force]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev operation_name="exec git" time_ms=1232.725993
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[git checkout --force 4bf8341c9c979b45c443870bfff1328a49536dd1]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev operation_name="exec git" time_ms=4.994643
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[git clean -fdx]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev operation_name="exec git" time_ms=3.643158
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[git rev-parse HEAD]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev operation_name="exec git" time_ms=2.335853
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[helm template . --name-template react-frontend-service-a0000000027 --namespace example-test --kube-version 1.21 --values ../overlays/a0000000027/globals.yaml --values ../overlays/flags.yaml --values ../overlays/a0000000027/values.yaml --values ../overlays/a0000000027/hotfix.yaml --api-versions acme.cert-manager.io/v1 --api-versions acme.cert-manager.io/v1/Challenge --api-versions acme.cert-manager.io/v1/Order --api-versions acme.cert-manager.io/v1alpha2 --api-versions acme.cert-manager.io/v1alpha2/Challenge --api-versions acme.cert-manager.io/v1alpha2/Order --api-versions acme.cert-manager.io/v1alpha3 --api-versions acme.cert-manager.io/v1alpha3/Challenge --api-versions acme.cert-manager.io/v1alpha3/Order --api-versions acme.cert-manager.io/v1beta1 --api-versions acme.cert-manager.io/v1beta1/Challenge --api-versions acme.cert-manager.io/v1beta1/Order --api-versions admissionregistration.k8s.io/v1 --api-versions admissionregistration.k8s.io/v1/MutatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1beta1 --api-versions admissionregistration.k8s.io/v1beta1/MutatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1beta1/ValidatingWebhookConfiguration --api-versions apiextensions.k8s.io/v1 --api-versions apiextensions.k8s.io/v1/CustomResourceDefinition --api-versions apiextensions.k8s.io/v1beta1 --api-versions apiextensions.k8s.io/v1beta1/CustomResourceDefinition --api-versions apiregistration.k8s.io/v1 --api-versions apiregistration.k8s.io/v1/APIService --api-versions apiregistration.k8s.io/v1beta1 --api-versions apiregistration.k8s.io/v1beta1/APIService --api-versions apps/v1 --api-versions apps/v1/ControllerRevision --api-versions apps/v1/DaemonSet --api-versions apps/v1/Deployment --api-versions apps/v1/ReplicaSet --api-versions apps/v1/StatefulSet --api-versions argoproj.io/v1alpha1 --api-versions argoproj.io/v1alpha1/AppProject --api-versions argoproj.io/v1alpha1/Application --api-versions argoproj.io/v1alpha1/ApplicationSet --api-versions autoscaling/v1 --api-versions autoscaling/v1/HorizontalPodAutoscaler --api-versions autoscaling/v2beta1 --api-versions autoscaling/v2beta1/HorizontalPodAutoscaler --api-versions autoscaling/v2beta2 --api-versions autoscaling/v2beta2/HorizontalPodAutoscaler --api-versions batch/v1 --api-versions batch/v1/CronJob --api-versions batch/v1/Job --api-versions batch/v1beta1 --api-versions batch/v1beta1/CronJob --api-versions bitnami.com/v1alpha1 --api-versions bitnami.com/v1alpha1/SealedSecret --api-versions cert-manager.io/v1 --api-versions cert-manager.io/v1/Certificate --api-versions cert-manager.io/v1/CertificateRequest --api-versions cert-manager.io/v1/ClusterIssuer --api-versions cert-manager.io/v1/Issuer --api-versions cert-manager.io/v1alpha2 --api-versions cert-manager.io/v1alpha2/Certificate --api-versions cert-manager.io/v1alpha2/CertificateRequest --api-versions cert-manager.io/v1alpha2/ClusterIssuer --api-versions cert-manager.io/v1alpha2/Issuer --api-versions cert-manager.io/v1alpha3 --api-versions cert-manager.io/v1alpha3/Certificate --api-versions cert-manager.io/v1alpha3/CertificateRequest --api-versions cert-manager.io/v1alpha3/ClusterIssuer --api-versions cert-manager.io/v1alpha3/Issuer --api-versions cert-manager.io/v1beta1 --api-versions cert-manager.io/v1beta1/Certificate --api-versions cert-manager.io/v1beta1/CertificateRequest --api-versions cert-manager.io/v1beta1/ClusterIssuer --api-versions cert-manager.io/v1beta1/Issuer --api-versions certificates.k8s.io/v1 --api-versions certificates.k8s.io/v1/CertificateSigningRequest --api-versions certificates.k8s.io/v1beta1 --api-versions certificates.k8s.io/v1beta1/CertificateSigningRequest --api-versions config.gatekeeper.sh/v1alpha1 --api-versions config.gatekeeper.sh/v1alpha1/Config --api-versions constraints.gatekeeper.sh/v1alpha1 --api-versions constraints.gatekeeper.sh/v1alpha1/K8sContainerLimits --api-versions constraints.gatekeeper.sh/v1alpha1/K8sNodeSelectors --api-versions constraints.gatekeeper.sh/v1alpha1/K8sPSPPrivilegedContainer --api-versions constraints.gatekeeper.sh/v1alpha1/K8sPSPVolumeTypes --api-versions constraints.gatekeeper.sh/v1alpha1/K8sRequiredLabels --api-versions constraints.gatekeeper.sh/v1alpha1/K8sRequiredProbes --api-versions constraints.gatekeeper.sh/v1alpha1/K8sUniqueIngressHost --api-versions constraints.gatekeeper.sh/v1alpha1/K8sUniqueServiceSelector --api-versions constraints.gatekeeper.sh/v1beta1 --api-versions constraints.gatekeeper.sh/v1beta1/K8sContainerLimits --api-versions constraints.gatekeeper.sh/v1beta1/K8sNodeSelectors --api-versions constraints.gatekeeper.sh/v1beta1/K8sPSPPrivilegedContainer --api-versions constraints.gatekeeper.sh/v1beta1/K8sPSPVolumeTypes --api-versions constraints.gatekeeper.sh/v1beta1/K8sRequiredLabels --api-versions constraints.gatekeeper.sh/v1beta1/K8sRequiredProbes --api-versions constraints.gatekeeper.sh/v1beta1/K8sUniqueIngressHost --api-versions constraints.gatekeeper.sh/v1beta1/K8sUniqueServiceSelector --api-versions coordination.k8s.io/v1 --api-versions coordination.k8s.io/v1/Lease --api-versions coordination.k8s.io/v1beta1 --api-versions coordination.k8s.io/v1beta1/Lease --api-versions crd.k8s.amazonaws.com/v1alpha1 --api-versions crd.k8s.amazonaws.com/v1alpha1/ENIConfig --api-versions discovery.k8s.io/v1 --api-versions discovery.k8s.io/v1/EndpointSlice --api-versions discovery.k8s.io/v1beta1 --api-versions discovery.k8s.io/v1beta1/EndpointSlice --api-versions events.k8s.io/v1 --api-versions events.k8s.io/v1/Event --api-versions events.k8s.io/v1beta1 --api-versions events.k8s.io/v1beta1/Event --api-versions extensions/v1beta1 --api-versions extensions/v1beta1/Ingress --api-versions flowcontrol.apiserver.k8s.io/v1beta1 --api-versions flowcontrol.apiserver.k8s.io/v1beta1/FlowSchema --api-versions flowcontrol.apiserver.k8s.io/v1beta1/PriorityLevelConfiguration --api-versions koudingspawn.de/v1 --api-versions koudingspawn.de/v1/Vault --api-versions linkerd.io/v1alpha1 --api-versions linkerd.io/v1alpha1/ServiceProfile --api-versions linkerd.io/v1alpha2 --api-versions linkerd.io/v1alpha2/ServiceProfile --api-versions mutations.gatekeeper.sh/v1alpha1 --api-versions mutations.gatekeeper.sh/v1alpha1/Assign --api-versions mutations.gatekeeper.sh/v1alpha1/AssignMetadata --api-versions networking.k8s.io/v1 --api-versions networking.k8s.io/v1/Ingress --api-versions networking.k8s.io/v1/IngressClass --api-versions networking.k8s.io/v1/NetworkPolicy --api-versions networking.k8s.io/v1beta1 --api-versions networking.k8s.io/v1beta1/Ingress --api-versions networking.k8s.io/v1beta1/IngressClass --api-versions node.k8s.io/v1 --api-versions node.k8s.io/v1/RuntimeClass --api-versions node.k8s.io/v1beta1 --api-versions node.k8s.io/v1beta1/RuntimeClass --api-versions policy.linkerd.io/v1alpha1 --api-versions policy.linkerd.io/v1alpha1/Server --api-versions policy.linkerd.io/v1alpha1/ServerAuthorization --api-versions policy.linkerd.io/v1beta1 --api-versions policy.linkerd.io/v1beta1/Server --api-versions policy.linkerd.io/v1beta1/ServerAuthorization --api-versions policy/v1 --api-versions policy/v1/PodDisruptionBudget --api-versions policy/v1beta1 --api-versions policy/v1beta1/PodDisruptionBudget --api-versions policy/v1beta1/PodSecurityPolicy --api-versions rbac.authorization.k8s.io/v1 --api-versions rbac.authorization.k8s.io/v1/ClusterRole --api-versions rbac.authorization.k8s.io/v1/ClusterRoleBinding --api-versions rbac.authorization.k8s.io/v1/Role --api-versions rbac.authorization.k8s.io/v1/RoleBinding --api-versions rbac.authorization.k8s.io/v1beta1 --api-versions rbac.authorization.k8s.io/v1beta1/ClusterRole --api-versions rbac.authorization.k8s.io/v1beta1/ClusterRoleBinding --api-versions rbac.authorization.k8s.io/v1beta1/Role --api-versions rbac.authorization.k8s.io/v1beta1/RoleBinding --api-versions scheduling.k8s.io/v1 --api-versions scheduling.k8s.io/v1/PriorityClass --api-versions scheduling.k8s.io/v1beta1 --api-versions scheduling.k8s.io/v1beta1/PriorityClass --api-versions split.smi-spec.io/v1alpha1 --api-versions split.smi-spec.io/v1alpha1/TrafficSplit --api-versions split.smi-spec.io/v1alpha2 --api-versions split.smi-spec.io/v1alpha2/TrafficSplit --api-versions status.gatekeeper.sh/v1beta1 --api-versions status.gatekeeper.sh/v1beta1/ConstraintPodStatus --api-versions status.gatekeeper.sh/v1beta1/ConstraintTemplatePodStatus --api-versions status.gatekeeper.sh/v1beta1/MutatorPodStatus --api-versions storage.k8s.io/v1 --api-versions storage.k8s.io/v1/CSIDriver --api-versions storage.k8s.io/v1/CSINode --api-versions storage.k8s.io/v1/StorageClass --api-versions storage.k8s.io/v1/VolumeAttachment --api-versions storage.k8s.io/v1beta1 --api-versions storage.k8s.io/v1beta1/CSIDriver --api-versions storage.k8s.io/v1beta1/CSINode --api-versions storage.k8s.io/v1beta1/CSIStorageCapacity --api-versions storage.k8s.io/v1beta1/StorageClass --api-versions storage.k8s.io/v1beta1/VolumeAttachment --api-versions templates.gatekeeper.sh/v1 --api-versions templates.gatekeeper.sh/v1/ConstraintTemplate --api-versions templates.gatekeeper.sh/v1alpha1 --api-versions templates.gatekeeper.sh/v1alpha1/ConstraintTemplate --api-versions templates.gatekeeper.sh/v1beta1 --api-versions templates.gatekeeper.sh/v1beta1/ConstraintTemplate --api-versions v1 --api-versions v1/ConfigMap --api-versions v1/Endpoints --api-versions v1/Event --api-versions v1/LimitRange --api-versions v1/Namespace --api-versions v1/Node --api-versions v1/PersistentVolume --api-versions v1/PersistentVolumeClaim --api-versions v1/Pod --api-versions v1/PodTemplate --api-versions v1/ReplicationController --api-versions v1/ResourceQuota --api-versions v1/Secret --api-versions v1/Service --api-versions v1/ServiceAccount --api-versions vpcresources.k8s.aws/v1beta1 --api-versions vpcresources.k8s.aws/v1beta1/SecurityGroupPolicy --include-crds]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev/react-frontend-service/helm_base operation_name="exec helm" time_ms=38.575524
time="2022-01-18T22:35:05Z" level=info msg=Trace args="[helm dependency build]" dir=/tmp/https___gitlab.com_somewhere_prt_example-apps_mushu-dev/react-frontend-service/helm_base operation_name="exec helm" time_ms=96.118729

The text was updated successfully, but these errors were encountered:

pthornton · 2022-01-18T22:40:48Z

ApplicationSet that creates application for above chart:

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: react-frontend-service-test
  labels:
    ring: test
spec:
  generators:
    - git:
        repoURL: https://gitlab.com/somewhere/prt/example-apps/mushu-dev.git
        revision: HEAD
        files:
          - path: "config/**/config.json"
  template:
    metadata:
      name: 'react-frontend-service-{{cluster.name}}'
      labels:
        ring: test
        tenant: '{{cluster.name}}'
        domain: '{{cluster.domain}}'
    spec:
      project: '{{argocd.project}}'
      source:
        repoURL: https://gitlab.com/somwhere/prt/example-apps/mushu-dev.git
        targetRevision: HEAD
        path: "./react-frontend-service/helm_base"
        helm:
          version: v3
          releaseName: 'react-frontend-service-{{cluster.name}}'
          valueFiles:
            - ../overlays/{{cluster.name}}/globals.yaml
            - ../overlays/flags.yaml
            - ../overlays/{{cluster.name}}/values.yaml
            - ../overlays/{{cluster.name}}/hotfix.yaml
      destination:
        server: '{{cluster.address}}'
        namespace: 'example-test'
      ignoreDifferences:
      - group: apps
        jsonPointers:
        - /spec/replicas
        kind: Deployment
        namespace: 'example-test'

      # Sync policy
      syncPolicy:
        automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
          prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
          selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
          allowEmpty: false # Allows deleting all application resources during automatic syncing ( false by default ).
        syncOptions: # Sync options which modifies sync behavior
          - CreateNamespace=true # Namespace Auto-Creation ensures that namespace specified as the application destination exists in the destination cluster.
          - Validate=true # disables resource validation (equivalent to 'kubectl apply --validate=false') ( true by default ).

pthornton · 2022-01-19T15:04:49Z

Worked in v2.0.5 helm chart 3.12.1. Starts failing in v2.1.0 helm chart 3.17.1.

hammeryx · 2022-01-19T16:38:20Z

@AustinSmart t I noticed you had some comments in other threads around OCI chart deps being broken in 2.1.0.... Is this related?

AustinSmart · 2022-01-19T16:46:01Z

Since there is a oci chart as a dependency in the first comment, yes it’s likely related. I don’t think the oci chart dependency bug has been fixed yet.

hammeryx · 2022-01-19T16:49:04Z

Is there a workaround for that @AustinSmart ? Couldn't find any reference as to exactly why that functionality broke.

pthornton · 2022-01-19T16:57:19Z

time="2022-01-19T16:45:42Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add 712053168757.dkr.ecr.us-west-2.amazonaws.com oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-typescript-backend-service` failed exit status 1: Error: looks like \"oci://712053168757.dkr.ecr.us-west-2.amazonaws.com/helm/high-five-templates-typescript-backend-service\" is not a valid chart repository or cannot be reached: tag explicitly required" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-01-19T16:50:42Z" grpc.service=repository.RepoServerService grpc.start_time="2022-01-19T16:45:42Z" grpc.time_ms=1.839 span.kind=server system=grpc

Might be related to 7104

Helm 3.7 OCI

Support for the oci:// protocol is also available in various other subcommands. Here is a complete list:

helm pull
helm show
helm template
helm install
helm upgrade

Note that "helm repo" is missing....

mfbieber · 2022-01-26T23:04:48Z

I had the same issue (also AWS ECR repo that connected well but the same 401 error when adding an application).

The regression probably came in with (lines 195 to 197 in RepoAdd() in util/helm/cmd.go on October 30th 2021):

if c.helmPassCredentialsSupported && passCredentials {
	args = append(args, "--pass-credentials")
}

This is not what we need to login to the ECR, we need it without the --pass-credentials flag for helm. So for instance, this works to login:

aws ecr get-login-password --region us-east-1 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"

I haven't dug into why this is so, but I have a temporary workaround that is working for me with v2.2.3. You can simply set passCredentials to false in your Application definition:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: xxxx
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: xxxx
  source:
    repoURL: xxxx
    path: xxxx
    targetRevision: xxxx
    helm:
      passCredentials: false
...

pthornton · 2022-02-07T14:51:56Z

@mfbieber I'm setting this in the application but I'm not sure it's getting honored. I'm using ApplicationSet to create the application. The ApplicationSet shows:

      Source:
        Helm:
          Pass Credentials:  false
          Release Name:      typescript-backend-service-{{cluster.name}}
          Value Files:
            ../overlays/{{cluster.name}}/globals.yaml
            ../overlays/flags.yaml
            ../overlays/{{cluster.name}}/values.yaml
            ../overlays/{{cluster.name}}/hotfix.yaml

The created Application has no mention of passCredentials:

  Source:
    Helm:
      Release Name:  typescript-backend-service-a0000000027
      Value Files:
        ../overlays/a0000000027/globals.yaml
        ../overlays/flags.yaml
        ../overlays/a0000000027/values.yaml
        ../overlays/a0000000027/hotfix.yaml

Know of any way to verify passCredentials value on the Application? I see a --helm-pass-credentials on argocd cli set command, but nothing on get? Both get/set fail with no permission to access oci. Starting to think the ApplicationSet is not honoring the setting.

pthornton · 2022-02-07T15:48:40Z

@mfbieber What version are you on? I created the application directly and same failure on 3.29.5/2.2.2.

I created the Application directly and it still fails with same error:

  Source:
    Helm:
      Pass Credentials:  false
      Release Name:      typescript-backend-service-a0000000027
      Value Files:
        ../overlays/a0000000027/globals.yaml
        ../overlays/flags.yaml
        ../overlays/a0000000027/values.yaml
        ../overlays/a0000000027/hotfix.yaml

Are you calling this directly yourself?

aws ecr get-login-password --region us-east-1 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"

pthornton · 2022-02-10T20:37:50Z

Have a workaround using plugin.
In values.yaml:

  repoServer:
    volumes:
      - name: custom-tools
        emptyDir: {}
      - name: plugin-scripts
        configMap:
          name: plugin-scripts
          defaultMode: 0755
    initContainers:
    - name: helm-setup
      image: alpine:3
      command: [
        "/bin/sh",
        "-c",
        "apk add --no-cache aws-cli;
        which aws;
        aws --version;
        cp /usr/bin/aws /custom-tools/aws; "]
      volumeMounts:
      - mountPath: /custom-tools
        name: custom-tools
    volumeMounts:
    - mountPath: /usr/local/bin/aws
      name: custom-tools
      subPath: aws
    - mountPath: /usr/local/bin/helmsetup.sh
      name: plugin-scripts
      subPath: helmsetup.sh
      readOnly: true
    - mountPath: /usr/local/bin/helmtemplate.sh
      name: plugin-scripts
      subPath: helmtemplate.sh
      readOnly: true

  server:
    config:
      configManagementPlugins: |
        - name: helm_aws_oci
          init:
            command: ["/bin/sh", "-c", "/usr/local/bin/helmsetup.sh"]
          generate:
            command: ["/bin/sh", "-c", "/usr/local/bin/helmtemplate.sh"]

Add scripts configmap under templates/plugin-scripts.yaml:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: plugin-scripts
data:
  helmsetup.sh: |
    #!/bin/sh
    export HELM_EXPERIMENTAL_OCI=1
    aws ecr get-login-password --region us-west-2 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-west-2.amazonaws.com"
    helm dependency build
  helmtemplate.sh: |
    #!/bin/sh
    export HELM_EXPERIMENTAL_OCI=1
    helm template --api-versions ${KUBE_API_VERSIONS} -n ${ARGOCD_APP_NAMESPACE} ${ARGOCD_APP_NAME} . --include-crds ${HELM_VALUES_FILES} ${HELM_ARGS}

In application def, replace helm section with plugin:

apiVersion: argoproj.io/v1alpha1
kind: Application
*******
      source:
        path: ./lacework/helm_base
        plugin:
          env:
          - name: HELM_VALUES_FILES
            value: --values ../overlays/a0000000027/globals.yaml --values ../overlays/flags.yaml
              --values ../overlays/a0000000027/values.yaml --values ../overlays/a0000000027/hotfix.yaml
          - name: HELM_ARGS
            value: --set lacework.lacework-agent.laceworkConfig.accessToken='**************************'
              --set lacework.lacework-agent.laceworkConfig.env='mushu.falkor.rocks'
              --set lacework.lacework-agent.laceworkConfig.kubernetesCluster='a0000000027'
          name: helm_aws_oci

mfbieber · 2022-02-13T22:50:40Z

@pthornton, sorry for replying late.

I am working with 3.26.12 and yes, I called this myself:

aws ecr get-login-password --region us-east-1 | helm registry login --password-stdin --username AWS "$(aws sts get-caller-identity --query Account --output text).dkr.ecr.us-east-1.amazonaws.com"

I would try to check this week if something changed during the last two weeks in the code. I could also submit a PR to somehow fix this, but I am unsure of how we would like to solve this. I'll think about it and will try to make a proposal, if I can.

Good that you have the plugin working, it looks like a lot of work and I would rather have argo-cd support AWS OCI itself!

hedasaurabh · 2022-04-06T19:10:38Z

@mfbieber Any updates here? We are running into the same issue.

Version:

$ argocd version
argocd: v2.3.3+07ac038.dirty
  BuildDate: 2022-03-30T05:20:18Z
  GitCommit: 07ac038a8f97a93b401e824550f0505400a8c84e
  GitTreeState: dirty
  GoVersion: go1.18
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.2.5+8f981cc
  BuildDate: 2022-02-05T01:33:25Z
  GitCommit: 8f981ccfcf942a9eb00bc466649f8499ba0455f5
  GitTreeState: clean
  GoVersion: go1.16.11
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v4.2.0 2021-06-30T22:49:26Z
  Helm Version: v3.7.1+g1d11fcb
  Kubectl Version: v0.22.2
  Jsonnet Version: v0.17.0

Repo was added successfully.

$ argocd repo list
TYPE  NAME                  REPO                                                                  INSECURE  OCI    LFS    CREDS  STATUS      MESSAGE
helm  standard-service  xxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com     false     true   false  true   Successful

We are using this OCI repo as a subchart dependency, also our ArgoCD application is generated using ApplicationSet

name: sample-service
version: "0.1.0"
kubeVersion: ">= 1.19.0-0"
description: chart to deploy a ccp-sample-service
dependencies:
  - name: standard-service
    repository: oci://xxxxxxxxxx.dkr.ecr.us-east-1.amazonaws.com
    version: 1.0.0

This is the error I get

rpc error: code = Unknown desc = Manifest generation error (cached): `helm repo add --username ****** --password ****** standard-service oci://xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com` failed exit status 1: Error: looks like "oci://xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com" is not a valid chart repository or cannot be reached: object required

mfbieber · 2022-04-25T15:46:23Z

Hi guys, sorry, I don't manage to work on this efficiently at the moment (too much going on).

In general, we abandoned the OCI approach in our projects, because it didn't seem to be properly supported with ECR - at least when we checked. Also what we really disliked is the workaround around logging in every x hours with a workaround CronJob: #6062 (comment)

alexef · 2022-12-09T15:41:53Z

I believe this issue is now fixed in v2.5.4, by: #11327

@hedasaurabh / @pthornton can you confirm?

blakepettersson · 2023-09-19T20:49:07Z

Closing, we've been using ECR Helm charts in prod for quite some time.

lado936 · 2024-12-11T11:33:16Z

i still have this issue and cant find any workaround to make it work. i have added argo repo and its connected succesfully,
and have below dependency in chart.yaml

alias: ory-hydra-infra
name: base
version: ~0.0.4
repository: "oci://1111.dkr.ecr.eu-west-1.amazonaws.com/helm"

Argo CD
v2.13.1+af54ef8

time="2024-12-11T11:18:21Z" level=error msg="helm dependency build failed exit status 1: Error: could not download oci://1111.dkr.ecr.eu-west-1.amazonaws.com/helm/base: unexpected status from HEAD request to https://1111.dkr.ecr.eu-west-1.amazonaws.com/v2/helm/base/manifests/0.0.4: 403 Forbidden" execID=39991 │repo-server time="2024-12-11T11:18:21Z" level=error msg="helm registry logout 1111.dkr.ecr.eu-west-1.amazonaws.com/helmfailed exit status 1: Error: not logged in" execID=c875d repo-server time="2024-12-11T11:18:21Z" level=error msg="finished unary call with code Unknown" error="failed to build helm dependencies: failed to build dependencies: failed to get command args to log:helm dependency build faile d exit status 1: Error: could not download oci://1111.dkr.ecr.eu-west-1.amazonaws.com/helm/base: unexpected status from HEAD request to https://1111.dkr.ecr.eu-west-1.amazonaws.com/v2/helm/base/manifests/0.0.4: 403 Forbidden" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-12-11T11:18:20Z" grpc.time_ms=1002.078 span.kind=server system=grpc

pthornton added the bug Something isn't working label Jan 18, 2022

blakepettersson closed this as completed Sep 19, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCI helm charts hosted in AWS OCI broken after upgrade from v2.0.4 to v2.2.2 #8214

OCI helm charts hosted in AWS OCI broken after upgrade from v2.0.4 to v2.2.2 #8214

pthornton commented Jan 18, 2022 •

edited

Loading

pthornton commented Jan 18, 2022

pthornton commented Jan 19, 2022

hammeryx commented Jan 19, 2022

AustinSmart commented Jan 19, 2022

hammeryx commented Jan 19, 2022 •

edited

Loading

pthornton commented Jan 19, 2022 •

edited

Loading

mfbieber commented Jan 26, 2022 •

edited

Loading

pthornton commented Feb 7, 2022

pthornton commented Feb 7, 2022

pthornton commented Feb 10, 2022 •

edited

Loading

mfbieber commented Feb 13, 2022

hedasaurabh commented Apr 6, 2022

mfbieber commented Apr 25, 2022

alexef commented Dec 9, 2022

blakepettersson commented Sep 19, 2023

lado936 commented Dec 11, 2024

OCI helm charts hosted in AWS OCI broken after upgrade from v2.0.4 to v2.2.2 #8214

OCI helm charts hosted in AWS OCI broken after upgrade from v2.0.4 to v2.2.2 #8214

Comments

pthornton commented Jan 18, 2022 • edited Loading

pthornton commented Jan 18, 2022

pthornton commented Jan 19, 2022

hammeryx commented Jan 19, 2022

AustinSmart commented Jan 19, 2022

hammeryx commented Jan 19, 2022 • edited Loading

pthornton commented Jan 19, 2022 • edited Loading

mfbieber commented Jan 26, 2022 • edited Loading

pthornton commented Feb 7, 2022

pthornton commented Feb 7, 2022

pthornton commented Feb 10, 2022 • edited Loading

mfbieber commented Feb 13, 2022

hedasaurabh commented Apr 6, 2022

mfbieber commented Apr 25, 2022

alexef commented Dec 9, 2022

blakepettersson commented Sep 19, 2023

lado936 commented Dec 11, 2024

pthornton commented Jan 18, 2022 •

edited

Loading

hammeryx commented Jan 19, 2022 •

edited

Loading

pthornton commented Jan 19, 2022 •

edited

Loading

mfbieber commented Jan 26, 2022 •

edited

Loading

pthornton commented Feb 10, 2022 •

edited

Loading