Define artifactRepositoryRef only once in spec

[hadim]

At the moment each new workflow must redefine an artifact repository:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: input-from-s3-

spec:
  entrypoint: input-from-s3
  serviceAccountName: argo
  templates:
  - name: input-from-s3
    container:
      image: alpine:3.12
      command: [sh, -c]
      args: ["cat /inputs/inputs.txt"]
    inputs:
      artifacts:
        - name: test-input
          path: /inputs
          s3:
            endpoint: s3.amazonaws.com
            region: us-east-2
            bucket: test-bucket
            key: inputs
            accessKeySecret:
              name: argo-s3-credentials
              key: accessKey
            secretKeySecret:
              name: argo-s3-credentials
              key: secretKey

Is that possible to define s3 once and reuse it in different workflows? Something like that:

apiVersion: argoproj.io/v1alpha1
kind: ArtifactRepo
metadata:
  name: my-input-artifact
spec:
  type: s3
  config:
    endpoint: s3.amazonaws.com
    region: us-east-1
    bucket: test-hadrien
    key: inputs
    accessKeySecret:
      name: argo-s3-credentials
      key: accessKey
    secretKeySecret:
      name: argo-s3-credentials
      key: secretKey
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: input-from-s3-

spec:
  entrypoint: input-from-s3
  serviceAccountName: argo
  templates:
  - name: input-from-s3
    container:
      image: alpine:3.12
      command: [sh, -c]
      args: ["cat /inputs/inputs.txt"]
    inputs:
      artifacts:
        - name: test-input
          path: /inputs
          artifactRepo: my-input-artifact

Or maybe this is already possible using workflow templates? If that's the case could you provide an example?

[sarabala1979]

I liked the above idea to decouple the (namespace scoped) artifact repo. Instead of new CRD, we can use Configmap which can have multiple artifact repo configuration which can be referred to in workflows.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: input-from-s3-

spec:
  entrypoint: input-from-s3
  serviceAccountName: argo
  templates:
  - name: input-from-s3
    container:
      image: alpine:3.12
      command: [sh, -c]
      args: ["cat /inputs/inputs.txt"]
    inputs:
      artifacts:
        - name: test-input
          path: /inputs
          configMapKeyRef:
              name: my-input-artifact
              key: repo1

@jessesuen comment on this.

Current Support:

you can configure the controller level artifact repository. All workflows on this controller can upload and download the artifacts to the central artifact repository.

https://github.com/argoproj/argo/blob/master/docs/configure-artifact-repository.md#s3-compatible-artifact-repository-bucket-such-as-aws-gcs-and-minio

# This example demonstrates the ability to pass artifacts
# from one step to the next.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: artifact-passing-
spec:
  entrypoint: artifact-example
  templates:
  - name: artifact-example
    steps:
    - - name: generate-artifact
        template: whalesay
    - - name: consume-artifact
        template: print-message
        arguments:
          artifacts:
          - name: message
            from: "{{steps.generate-artifact.outputs.artifacts.hello-art}}"

  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [sh, -c]
      args: ["sleep 1; cowsay hello world | tee /tmp/hello_world.txt"]
    outputs:
      artifacts:
      - name: hello-art
        path: /tmp/hello_world.txt

  - name: print-message
    inputs:
      artifacts:
      - name: message
        path: /tmp/message
    container:
      image: alpine:latest
      command: [sh, -c]
      args: ["cat /tmp/message"]

[hadim]

Thank you @sarabala1979. I am already using artifactRepository and it works well by I want the possibility to work with multiple artifacts at the same time.

About a new CRD or a configMap, whatever works best for you. I guess the configMap would be easier to implement but I think it could make sense to have a new CRD object too.

[alexec]

I'm pretty sure we already have an issue for this problem - which is a problem. I've tried to fix this already - hard to do unfortunatly.

[alexec]

How does this differ from #3307?

[alexec]

Work in progress (not trivial to solve) you only need to specify s3.key, everything else (e.g. s3.bucket) is taken from the artifactRepositoryRef

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: artifact-repository-ref
  labels:
    argo-e2e: "true"
spec:
  entrypoint: main
  artifactRepositoryRef:
    key: minio
  templates:
    - name: main
      dag:
        tasks:
          - name: generate
            template: generate
          - name: consume
            template: consume
            dependencies:
              - generate
    - name: generate
      container:
        image: argoproj/argosay:v2
        args:
          - echo
          - hello
          - /mnt/file
      outputs:
        artifacts:
          - name: file
            path: /mnt/file
            s3:
              key: my-file
    - name: consume
      container:
        image: argoproj/argosay:v2
        args:
          - cat
          - /tmp/file
      inputs:
        artifacts:
          - name: file
            path: /tmp/file
            s3:
              key: my-file