2.6.0 RC1 Fails to find helms secrets file using multiple sources

[jete-vian]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

I'm using the new multiple sources functionality in 2.6 trying to read secrets via helm secrets. It seems $myRepo is undefined or empty, therefore it can't find the proper path to the secrets file.

I receive this error message
[helm-secrets] File does not exist: /helm/external-values/argo-workflows/dev.enc.values.yaml Error: plugin "scripts/run.sh downloader" exited with error

To Reproduce

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argo-workflows
  namespace: argocd

  labels:
    environment: dev
  annotations:
    argocd.argoproj.io/sync-wave: "-3"
spec:
  project: argo-projects

  revisionHistoryLimit: 3

  sources:
  - repoURL: git@github.com:company/repo.git
    targetRevision: main
    ref: myRepo
  - chart: argo-workflows
    repoURL: https://argoproj.github.io/argo-helm
    targetRevision: 0.22.6
    helm:
      valueFiles:
        - secrets+gpg-import:///helm-secrets-private-keys/key.asc?$myRepo/helm/external-values/argo-workflows/dev.enc.values.yaml

  destination:
    name: in-cluster
    namespace: argo

  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true

Expected behavior

I expect the $myRepo variable to contain the path and the secrets file to be located.
Instead, It's unable to find the secrets file.

Version

argocd-server: v2.6.0-rc1+81e40d5
  BuildDate: 2022-12-19T16:48:52Z
  GitCommit: 81e40d53fe8eee50b00ab38c4b07b34b3dcd6d25
  GitTreeState: clean
  GoVersion: go1.18.9
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
  Helm Version: v3.10.3+g835b733
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.19.1

Logs

rpc error: code = Unknown desc = `helm template . --name-template argo-workflows --namespace argo --kube-version 1.23 --values secrets+gpg-import:///helm-secrets-private-keys/key.asc?/helm/external-values/argo-workflows/dev.enc.values.yaml --api-versions acme.cert-manager.io/v1 --api-versions acme.cert-manager.io/v1/Challenge --api-versions acme.cert-manager.io/v1/Order --api-versions admissionregistration.k8s.io/v1 --api-versions admissionregistration.k8s.io/v1/MutatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration --api-versions apiextensions.k8s.io/v1 --api-versions apiextensions.k8s.io/v1/CustomResourceDefinition --api-versions apiregistration.k8s.io/v1 --api-versions apiregistration.k8s.io/v1/APIService --api-versions apps/v1 --api-versions apps/v1/ControllerRevision --api-versions apps/v1/DaemonSet --api-versions apps/v1/Deployment --api-versions apps/v1/ReplicaSet --api-versions apps/v1/StatefulSet --api-versions argoproj.io/v1alpha1 --api-versions argoproj.io/v1alpha1/AnalysisRun --api-versions argoproj.io/v1alpha1/AnalysisTemplate --api-versions argoproj.io/v1alpha1/AppProject --api-versions argoproj.io/v1alpha1/Application --api-versions argoproj.io/v1alpha1/ApplicationSet --api-versions argoproj.io/v1alpha1/ArgoCDExtension --api-versions argoproj.io/v1alpha1/ClusterAnalysisTemplate --api-versions argoproj.io/v1alpha1/ClusterWorkflowTemplate --api-versions argoproj.io/v1alpha1/CronWorkflow --api-versions argoproj.io/v1alpha1/EventBus --api-versions argoproj.io/v1alpha1/EventSource --api-versions argoproj.io/v1alpha1/Experiment --api-versions argoproj.io/v1alpha1/Rollout --api-versions argoproj.io/v1alpha1/Sensor --api-versions argoproj.io/v1alpha1/Workflow --api-versions argoproj.io/v1alpha1/WorkflowArtifactGCTask --api-versions argoproj.io/v1alpha1/WorkflowEventBinding --api-versions argoproj.io/v1alpha1/WorkflowTaskResult --api-versions argoproj.io/v1alpha1/WorkflowTaskSet --api-versions argoproj.io/v1alpha1/WorkflowTemplate --api-versions autoscaling/v1 --api-versions autoscaling/v1/HorizontalPodAutoscaler --api-versions autoscaling/v2 --api-versions autoscaling/v2/HorizontalPodAutoscaler --api-versions autoscaling/v2beta1 --api-versions autoscaling/v2beta1/HorizontalPodAutoscaler --api-versions autoscaling/v2beta2 --api-versions autoscaling/v2beta2/HorizontalPodAutoscaler --api-versions batch/v1 --api-versions batch/v1/CronJob --api-versions batch/v1/Job --api-versions batch/v1beta1 --api-versions batch/v1beta1/CronJob --api-versions cert-manager.io/v1 --api-versions cert-manager.io/v1/Certificate --api-versions cert-manager.io/v1/CertificateRequest --api-versions cert-manager.io/v1/ClusterIssuer --api-versions cert-manager.io/v1/Issuer --api-versions certificates.k8s.io/v1 --api-versions certificates.k8s.io/v1/CertificateSigningRequest --api-versions cloud.google.com/v1 --api-versions cloud.google.com/v1/BackendConfig --api-versions cloud.google.com/v1beta1 --api-versions cloud.google.com/v1beta1/BackendConfig --api-versions coordination.k8s.io/v1 --api-versions coordination.k8s.io/v1/Lease --api-versions discovery.k8s.io/v1 --api-versions discovery.k8s.io/v1/EndpointSlice --api-versions discovery.k8s.io/v1beta1 --api-versions discovery.k8s.io/v1beta1/EndpointSlice --api-versions events.k8s.io/v1 --api-versions events.k8s.io/v1/Event --api-versions flowcontrol.apiserver.k8s.io/v1beta1 --api-versions flowcontrol.apiserver.k8s.io/v1beta1/FlowSchema --api-versions flowcontrol.apiserver.k8s.io/v1beta1/PriorityLevelConfiguration --api-versions flowcontrol.apiserver.k8s.io/v1beta2 --api-versions flowcontrol.apiserver.k8s.io/v1beta2/FlowSchema --api-versions flowcontrol.apiserver.k8s.io/v1beta2/PriorityLevelConfiguration --api-versions hub.gke.io/v1 --api-versions hub.gke.io/v1/Membership --api-versions internal.autoscaling.gke.io/v1alpha1 --api-versions internal.autoscaling.gke.io/v1alpha1/CapacityRequest --api-versions migration.k8s.io/v1alpha1 --api-versions migration.k8s.io/v1alpha1/StorageState --api-versions migration.k8s.io/v1alpha1/StorageVersionMigration --api-versions monitoring.coreos.com/v1 --api-versions monitoring.coreos.com/v1/Alertmanager --api-versions monitoring.coreos.com/v1/PodMonitor --api-versions monitoring.coreos.com/v1/Probe --api-versions monitoring.coreos.com/v1/Prometheus --api-versions monitoring.coreos.com/v1/PrometheusRule --api-versions monitoring.coreos.com/v1/ServiceMonitor --api-versions monitoring.coreos.com/v1/ThanosRuler --api-versions monitoring.coreos.com/v1alpha1 --api-versions monitoring.coreos.com/v1alpha1/AlertmanagerConfig --api-versions networking.gke.io/v1 --api-versions networking.gke.io/v1/ManagedCertificate --api-versions networking.gke.io/v1/ServiceAttachment --api-versions networking.gke.io/v1beta1 --api-versions networking.gke.io/v1beta1/FrontendConfig --api-versions networking.gke.io/v1beta1/ManagedCertificate --api-versions networking.gke.io/v1beta1/ServiceAttachment --api-versions networking.gke.io/v1beta1/ServiceNetworkEndpointGroup --api-versions networking.gke.io/v1beta2 --api-versions networking.gke.io/v1beta2/ManagedCertificate --api-versions networking.k8s.io/v1 --api-versions networking.k8s.io/v1/Ingress --api-versions networking.k8s.io/v1/IngressClass --api-versions networking.k8s.io/v1/NetworkPolicy --api-versions node.k8s.io/v1 --api-versions node.k8s.io/v1/RuntimeClass --api-versions node.k8s.io/v1beta1 --api-versions node.k8s.io/v1beta1/RuntimeClass --api-versions nodemanagement.gke.io/v1alpha1 --api-versions nodemanagement.gke.io/v1alpha1/UpdateInfo --api-versions policy/v1 --api-versions policy/v1/PodDisruptionBudget --api-versions policy/v1beta1 --api-versions policy/v1beta1/PodDisruptionBudget --api-versions policy/v1beta1/PodSecurityPolicy --api-versions rbac.authorization.k8s.io/v1 --api-versions rbac.authorization.k8s.io/v1/ClusterRole --api-versions rbac.authorization.k8s.io/v1/ClusterRoleBinding --api-versions rbac.authorization.k8s.io/v1/Role --api-versions rbac.authorization.k8s.io/v1/RoleBinding --api-versions scheduling.k8s.io/v1 --api-versions scheduling.k8s.io/v1/PriorityClass --api-versions snapshot.storage.k8s.io/v1 --api-versions snapshot.storage.k8s.io/v1/VolumeSnapshot --api-versions snapshot.storage.k8s.io/v1/VolumeSnapshotClass --api-versions snapshot.storage.k8s.io/v1/VolumeSnapshotContent --api-versions snapshot.storage.k8s.io/v1beta1 --api-versions snapshot.storage.k8s.io/v1beta1/VolumeSnapshot --api-versions snapshot.storage.k8s.io/v1beta1/VolumeSnapshotClass --api-versions snapshot.storage.k8s.io/v1beta1/VolumeSnapshotContent --api-versions storage.k8s.io/v1 --api-versions storage.k8s.io/v1/CSIDriver --api-versions storage.k8s.io/v1/CSINode --api-versions storage.k8s.io/v1/StorageClass --api-versions storage.k8s.io/v1/VolumeAttachment --api-versions storage.k8s.io/v1beta1 --api-versions storage.k8s.io/v1beta1/CSIStorageCapacity --api-versions v1 --api-versions v1/ConfigMap --api-versions v1/Endpoints --api-versions v1/Event --api-versions v1/LimitRange --api-versions v1/Namespace --api-versions v1/Node --api-versions v1/PersistentVolume --api-versions v1/PersistentVolumeClaim --api-versions v1/Pod --api-versions v1/PodTemplate --api-versions v1/ReplicationController --api-versions v1/ResourceQuota --api-versions v1/Secret --api-versions v1/Service --api-versions v1/ServiceAccount --include-crds` failed exit status 1: [helm-secrets] File does not exist: /helm/external-values/argo-workflows/dev.enc.values.yaml Error: plugin "scripts/run.sh downloader" exited with error

[ishitasequeira]

Currently, the supported format for referenced valueFile from another source is $<ref_variable_name>/<path_to_file>. That is, the referenced value file needs to start with $<ref_variable_name>.

In this case, the format supported would be $myRepo/helm/external-values/argo-workflows/dev.enc.values.yaml.

[jete-vian]

Currently, the supported format for referenced valueFile from another source is $<ref_variable_name>/<path_to_file>. That is, the referenced value file needs to start with $<ref_variable_name>.

In this case, the format supported would be $myRepo/helm/external-values/argo-workflows/dev.enc.values.yaml.

I understand the format is the proposed and currently supported format. I shouldn't of labeled this a bug but it seems to render helm-secrets useless for multi-source apps. Will this be revisited in the future?

[almereyda]

The current proposal to rearrange a desired behaviour for multi-source applications here shows three implementation vectors:

• fix: Support source references in remote values #11966

1. The implemented and presented for discussion mimicing of Kubernetes-native string-interpolation syntax
   https://kubernetes.io/docs/tasks/inject-data-application/define-interdependent-environment-variables/#define-an-environment-dependent-variable-for-a-container
2. An in the longer term desireable argocd-multi-repo-server, which allows to pin certain projects to instances (slightly off-topic for this Helm Secrets case, but relevant for other new patterns emerging with multi-source applications)
3. An implementation idea by @crenshaw-dev in fix: Support source references in remote values #11966 (comment) which suggests not to

substitute the long-lived cache path of the referenced repo. Instead, copy the one referenced file out of the referenced repo to a new, randomized, temporary path. This has three advantages:

• we don't care so much about the possibility of leaking the path - I think we could arbitrarily substitute the path into the valuesFile string
• we don't have to hold a lock on the referenced source path as long - we release the lock immediately after copying the one file out
• we no longer have to prevent referencing the same repo at a different revision, because we're no longer holding a lock on the referenced repo while generating the referencing repo's sources - I've seen at least one person who wanted this restriction lifted