How to escape expression in Vault Injector annotation

[dbeilin]

I need help with escaping in my Argo Workflow:

    - name: faith-tmpl
      inputs:
        parameters:
          - name: run_name
          - name: repo_name
          - name: release_name
          - name: username
          - name: run_params
      metadata:
        labels:
          argo-workflows-e2e: "{{inputs.parameters.run_name}}"
        annotations:
          cluster-autoscaler.kubernetes.io/safe-to-evict: "false"
          vault.hashicorp.com/agent-inject: "true"
          vault.hashicorp.com/agent-pre-populate-only: "true"
          vault.hashicorp.com/agent-inject-secret-.env: "{{=sprig.fromJson(steps.composer.outputs.parameters.output).vault_ci_path}}"
          vault.hashicorp.com/role: home-ci
          vault.hashicorp.com/agent-inject-template-.env: |
            {{- with secret "{{=sprig.fromJson(steps.composer.outputs.parameters.output).vault_ci_path}}" }}
            {{- range $k, $v := .Data.data }}
            export {{ $k }}={{ $v }}
            {{- end }}
            {{- end }}

The sprig function in agent-inject-secret-.env works fine, but it doesn't work in agent-inject-template-.env.
I don't know how to escape the parameter here:

{{- with secret "{{=sprig.fromJson(steps.composer.outputs.parameters.output).vault_ci_path}}" }}

I tried a few things but none worked for me.

[agilgur5]

I tried a few things but none worked for me.

In general, it is good to list what you tried. For folks trying to help and for other users with similar problems.

 {{- with secret "{{=sprig.fromJson(steps.composer.outputs.parameters.output).vault_ci_path}}" }}

In this case you're using Argo templating within Vault templating, which uses consul-template under-the-hood which itself uses Go template.

But Argo's templating is evaluated first, so the string needs to be interpretable by Argo and then output a valid consul-template.
(For other readers, this is the opposite of the order of Helm template escaping).

Argo's expression engine is expr, which uses backticks `` for raw strings.
Putting this into the expr playground, you should be able to convert your template to:

{{= `{{- with secret ` + sprig.fromJson(steps.composer.outputs.parameters.output).vault_ci_path + ` }}` }}

The first {{= and last }} is Argo's expression notation. Then comes the raw string {{- with secret concatenated (+) with your expression and concatenated once more with the raw string }}.
So Argo should then output the YAML string: {{- with secret my_path }}

That should work in isolation, but you may need to further escape the entire vault.hashicorp.com/agent-inject-template-.env annotation though, as all of the {{ and }} may be confusing Argo (you did not provide an exact error message though. I've never tried this so I'm not sure).

Also you might not need to use sprig here, you might be able to use expr's own fromJSON function

[Joibel]

What does the annotation on the pod look like?

[agilgur5]

I got the following error in the vault container:

That would suggest that Argo didn't parse / template that annotation at all 🤔
That may be because the whole annotation needs escaping as I mentioned above, not just the one line (as it's all a string anyway). I would think that Argo would throw a validation error in this case though -- do your Controller logs mention anything?

[dbeilin]

@Joibel The original workflow I posted contains all of the annotations on this pod.
There are no errors in the controller. I tried to escape the rest of the lines like this:

          vault.hashicorp.com/agent-inject-template-.env: |
            {{= "`{{- with secret " + sprig.fromJson(steps.composer.outputs.parameters.output).vault_ci_path + " }}`" }}
            {{= "`{{- range $k, $v := .Data.data }}`" }}
            {{= "`export " + $k + "=" + $v + "`" }}
            {{= "`{{- end }}`" }}
            {{= "`{{- end }}`" }}

But the error remained the same:

│ 2024-03-15T16:12:07.481Z [ERROR] agent: runtime error encountered: error="template server: (dynamic): parse: template: :1: unexpected \"=\" in command" exitCode=1

Appreciate the help, thanks :)

[Joibel]

I was asking for the new annotations on the failing pod after trying to escape it and running the workflow. That'll show you the rendered template.
I'd have guessed @agilgur5 is right, that the template hasn't been rendered at all, but this would confirm it.

[dbeilin]

Oh sorry, my bad.
The generated annotations don't seem like the got templated at all:

      "annotations":
        {
          "cluster-autoscaler.kubernetes.io/safe-to-evict": "false",
          "vault.hashicorp.com/agent-inject": "true",
          "vault.hashicorp.com/agent-inject-secret-.env": "{{=sprig.fromJson(steps.composer.outputs.parameters.output).vault_ci_path}}",
          "vault.hashicorp.com/agent-inject-template-.env": "{{= `{{- with secret ` + sprig.fromJson(steps.composer.outputs.parameters.output).vault_ci_path + ` }}` }}\n{{- range $k, $v := .Data.data }}\nexport {{ $k }}={{ $v }}\n{{- end }}\n{{- end }}\n",
          "vault.hashicorp.com/agent-pre-populate-only": "true",

[dbeilin]

Eventually, this worked:

{{= "{"+"{- with secret \"" + fromJSON(inputs.parameters.run_params).vault_ci_path + "\" }"+"}" }

At the end of the line I concatenated two curly brackets because }} didn't work for some reason (also tried -}}). Now it works without escaping the rest of the lines.

Honestly this seems way to complicated, I might just pass this as a parameter and set the value in the template.
Also, thanks for the tip about not having to use sprig, fromJSON works as well :)
Also also, thanks for the explanation about the templating engines at work here.

[agilgur5]

So all the generated annotations got templated now? Could you provide those like you did in #12805 (reply in thread)? Since multiple didn't generate and there was no error message, that would be helpful for debugging

At the end of the line I concatenated two curly brackets because }} didn't work for some reason

I'm thinking this might be because Argo wasn't respecting the escaping. Since {{ is the Argo notation, not expr (expr just accepts a string, but Argo has to determine if it's a plain string or an embedded expression). That would also suggest Argo tries to interpret an expression in an expression 😵‍💫
I'm also still surprised there's no error anywhere