Adjust jinja2 runner to consume multiple input-parameters artifacts and generate single output

[mszostok]

Description

Jinja Runner engine can now accept multiple input data files. Now we need to add support for that in our workflows for jinja2.template Action execution to consume N input-parameters files.

Reason

Currently, there is no way to easily aggregate output artifacts into a single step.

For example, for RocketChat we need to executejinja.template 5 times to get final Helm Runner input. This leads to human copy-paste errors and is hard to define, read and maintain.

Ideas

• Maybe we can use some hacks from map-reduce example: docs: Add map-reduce example. Closes #4165  argoproj/argo-workflows#4175
• Maybe we can produce a .tgz file which will be a single input for jinja2.template, so we will be able to execute it as

                - - name: render
                    capact-action: jinja2.template
                    arguments:
                      artifacts:
                        - name: template
                          raw:
                            data: |
                              //...
                        - name: input-parameters
                          from: "{{steps.install-db.outputs.artifacts.mongodb}}"

  or

                - - name: zip
                     # our own step where we can define multiple inputs files to be zipped
                     # it's defined by Content Developer, so he can mention as many sources as he wants
                    template: zip-inputs
                    arguments:
                      artifacts:
                        - name: external-url
                          http:
                            url: https://raw.githubusercontent.com/foo/bar.yaml
                        - name: start-yaml
                          from: "{{steps.render-start-yaml.outputs.artifacts.render}}"
                        - name: default-values-yaml
                          from: "{{steps.render-values-yaml.outputs.artifacts.render}}"
                - - name: render
                    capact-action: jinja2.template
                    arguments:
                      artifacts:
                        - name: template
                          raw:
                            data: |
                              //...
                        - name: input-parameters-archive
                          from: "{{steps.zip-inputs.artifacts.output.archive}}"

• Maybe hack with passing s3 bucket name to jinja2.template where all inputs are located
• etc.

AC

• jinja2.template works in a single run on multiple input parameters.
• Current Implementation manifests are simplified based on implemented solution
  Testing:
  • Mock Helm Runner and Terraform Runner to print only input values instead of executing real action
  • Compare printed input values with those before changes. Output should be exactly the same.

[lukaszo]

I've prepared 4 versions to consider:

1. Change jinja2.template to accept 3 input-parameters https://gist.github.com/lukaszo/d1cd625e105fc8e7af90ffa1e76f4c71

• no boilerplate
• no additional steps

• requires changing jinja2.template interface
• required changing our jinja2 CLI to handle prefixes
• if there are less than 3 inputs then we need to pass empty files, similarly to config

2. Passing inputs using s3 bucket in Argo 2 - https://gist.github.com/lukaszo/4ddccf8d139255ddc2f0eef36ccce382

• (no pros :D)

• required changing our jinja2 CLI to handle prefixes(from file names?)
• for s3 artifact we need to pass everywhere [secret/access]key, endpoint etc. (We could inject it in our engine)
• requires extra step

3. Passing inputs using s3 bucket in Argo 3 https://gist.github.com/lukaszo/78454629c3158cc795b66942cdf9031c

• Only key is required for s3 artifact. It will use default artifacts configuration then

• requires changes in python
• requires extra step
• requires Argo 3

4. Use yq to add prefixes and merge into one bug file
   https://gist.github.com/lukaszo/d45cc0eaaf72aca4c2063b4cb21e6c71

• doesn't require any changes in our jinja2

• requires extra step