Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue when passing results in nested loops #5143

Closed
AdrianSkierniewski opened this issue Feb 18, 2021 · 3 comments
Closed

Issue when passing results in nested loops #5143

AdrianSkierniewski opened this issue Feb 18, 2021 · 3 comments

Comments

@AdrianSkierniewski
Copy link

Summary

First of all, I'd like to thank the Argo community for sharing such a nice tool with everyone. ❤️

I'm trying to run the workflow with nested loops based on the output of the previous steps. An outer loop should add more parameters to the inner loop.
At the first step, everything works as expected but in the next step, Argo doesn't recognize the output parameter as an array with JSON objects. Instead, I'm getting the whole string. I was looking at some other examples and I saw similar usage, but after spending some time trying to debug this I've stuck.

Could someone point me if I'm doing something wrong or is it a legit bug?

Diagnostics

What Kubernetes provider are you using?

  • Docker Desktop (Mac) v2.4.0.0 - k8s v1.18.8

Argo Workflows: v2.12.5

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  creationTimestamp: "2021-02-18T18:36:46Z"
  generateName: nested-loops-issue
  generation: 5
  labels:
    workflows.argoproj.io/completed: "true"
    workflows.argoproj.io/phase: Succeeded
  managedFields:
  - apiVersion: argoproj.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
   ...
    manager: argo
    operation: Update
    time: "2021-02-18T18:36:46Z"
  - apiVersion: argoproj.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      ...
    manager: workflow-controller
    operation: Update
    time: "2021-02-18T18:37:16Z"
  name: nested-loops-issuet6gpg
  namespace: default
  resourceVersion: "1537967"
  selfLink: /apis/argoproj.io/v1alpha1/namespaces/default/workflows/nested-loops-issuet6gpg
  uid: 2b77fa61-4c68-4c1d-8b0d-303f5560b160
spec:
  arguments: {}
  entrypoint: start
  templates:
  - arguments: {}
    inputs: {}
    metadata: {}
    name: start
    outputs: {}
    steps:
    - - arguments: {}
        name: return-first-array
        template: return-first-array
    - - arguments:
          parameters:
          - name: param
            value: '{{item}}'
        name: return-second-array
        template: return-second-array
        withParam: '{{steps.return-first-array.outputs.result}}'
    - - arguments:
          parameters:
          - name: message
            value: '{{item}}'
        name: print
        template: whalesay
        withParam: '{{steps.return-second-array.outputs.result}}'
  - arguments: {}
    inputs: {}
    metadata: {}
    name: return-first-array
    outputs: {}
    script:
      command:
      - python
      image: python:alpine3.6
      name: ""
      resources: {}
      source: |
        import json
        import sys
        json.dump(['11111111', '22222222', '33333333'], sys.stdout)
  - arguments: {}
    inputs:
      parameters:
      - name: param
    metadata: {}
    name: return-second-array
    outputs: {}
    script:
      command:
      - python
      image: python:alpine3.6
      name: ""
      resources: {}
      source: |
        import json
        import sys
        json.dump([{"name":"Foo - {{inputs.parameters.param}}"},{"name":"Doe - {{inputs.parameters.param}}"}], sys.stdout)
  - arguments: {}
    container:
      args:
      - '{{inputs.parameters.message}}'
      command:
      - cowsay
      image: docker/whalesay:latest
      name: ""
      resources: {}
    inputs:
      parameters:
      - name: message
    metadata: {}
    name: whalesay
    outputs: {}
status:
  conditions:
  - status: "True"
    type: Completed
  finishedAt: "2021-02-18T18:37:16Z"
  nodes:
    nested-loops-issuet6gpg:
      children:
      - nested-loops-issuet6gpg-3579028202
      displayName: nested-loops-issuet6gpg
      finishedAt: "2021-02-18T18:37:16Z"
      id: nested-loops-issuet6gpg
      name: nested-loops-issuet6gpg
      outboundNodes:
      - nested-loops-issuet6gpg-1329735744
      - nested-loops-issuet6gpg-2832660585
      - nested-loops-issuet6gpg-1833278630
      phase: Succeeded
      progress: 7/7
      resourcesDuration:
        cpu: 18
        memory: 18
      startedAt: "2021-02-18T18:36:46Z"
      templateName: start
      templateScope: local/nested-loops-issuet6gpg
      type: Steps
    nested-loops-issuet6gpg-1329735744:
      boundaryID: nested-loops-issuet6gpg
      displayName: print(0:[{"name":"Foo - 11111111"} {"name":"Doe - 11111111"}])
      finishedAt: "2021-02-18T18:37:10Z"
      id: nested-loops-issuet6gpg-1329735744
      inputs:
        parameters:
        - name: message
          value: '[{"name":"Foo - 11111111"},{"name":"Doe - 11111111"}]'
      name: nested-loops-issuet6gpg[2].print(0:[{"name":"Foo - 11111111"} {"name":"Doe
        - 11111111"}])
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 3
        memory: 3
      startedAt: "2021-02-18T18:37:06Z"
      templateName: whalesay
      templateScope: local/nested-loops-issuet6gpg
      type: Pod
    nested-loops-issuet6gpg-1552126305:
      boundaryID: nested-loops-issuet6gpg
      children:
      - nested-loops-issuet6gpg-2438547468
      displayName: return-second-array(0:11111111)
      finishedAt: "2021-02-18T18:36:59Z"
      id: nested-loops-issuet6gpg-1552126305
      inputs:
        parameters:
        - name: param
          value: "11111111"
      name: nested-loops-issuet6gpg[1].return-second-array(0:11111111)
      outputs:
        exitCode: "0"
        result: '[{"name": "Foo - 11111111"}, {"name": "Doe - 11111111"}]'
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2021-02-18T18:36:56Z"
      templateName: return-second-array
      templateScope: local/nested-loops-issuet6gpg
      type: Pod
    nested-loops-issuet6gpg-1833278630:
      boundaryID: nested-loops-issuet6gpg
      displayName: print(2:[{"name":"Foo - 33333333"} {"name":"Doe - 33333333"}])
      finishedAt: "2021-02-18T18:37:13Z"
      id: nested-loops-issuet6gpg-1833278630
      inputs:
        parameters:
        - name: message
          value: '[{"name":"Foo - 33333333"},{"name":"Doe - 33333333"}]'
      name: nested-loops-issuet6gpg[2].print(2:[{"name":"Foo - 33333333"} {"name":"Doe
        - 33333333"}])
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 6
        memory: 6
      startedAt: "2021-02-18T18:37:06Z"
      templateName: whalesay
      templateScope: local/nested-loops-issuet6gpg
      type: Pod
    nested-loops-issuet6gpg-2432375290:
      boundaryID: nested-loops-issuet6gpg
      children:
      - nested-loops-issuet6gpg-2438547468
      displayName: return-second-array(1:22222222)
      finishedAt: "2021-02-18T18:36:59Z"
      id: nested-loops-issuet6gpg-2432375290
      inputs:
        parameters:
        - name: param
          value: "22222222"
      name: nested-loops-issuet6gpg[1].return-second-array(1:22222222)
      outputs:
        exitCode: "0"
        result: '[{"name": "Foo - 22222222"}, {"name": "Doe - 22222222"}]'
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2021-02-18T18:36:56Z"
      templateName: return-second-array
      templateScope: local/nested-loops-issuet6gpg
      type: Pod
    nested-loops-issuet6gpg-2438547468:
      boundaryID: nested-loops-issuet6gpg
      children:
      - nested-loops-issuet6gpg-1329735744
      - nested-loops-issuet6gpg-2832660585
      - nested-loops-issuet6gpg-1833278630
      displayName: '[2]'
      finishedAt: "2021-02-18T18:37:16Z"
      id: nested-loops-issuet6gpg-2438547468
      name: nested-loops-issuet6gpg[2]
      phase: Succeeded
      progress: 3/3
      resourcesDuration:
        cpu: 13
        memory: 13
      startedAt: "2021-02-18T18:37:06Z"
      templateName: start
      templateScope: local/nested-loops-issuet6gpg
      type: StepGroup
    nested-loops-issuet6gpg-2832660585:
      boundaryID: nested-loops-issuet6gpg
      displayName: print(1:[{"name":"Foo - 22222222"} {"name":"Doe - 22222222"}])
      finishedAt: "2021-02-18T18:37:11Z"
      id: nested-loops-issuet6gpg-2832660585
      inputs:
        parameters:
        - name: message
          value: '[{"name":"Foo - 22222222"},{"name":"Doe - 22222222"}]'
      name: nested-loops-issuet6gpg[2].print(1:[{"name":"Foo - 22222222"} {"name":"Doe
        - 22222222"}])
      outputs:
        exitCode: "0"
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 4
        memory: 4
      startedAt: "2021-02-18T18:37:06Z"
      templateName: whalesay
      templateScope: local/nested-loops-issuet6gpg
      type: Pod
    nested-loops-issuet6gpg-3201289089:
      boundaryID: nested-loops-issuet6gpg
      children:
      - nested-loops-issuet6gpg-3511770631
      displayName: return-first-array
      finishedAt: "2021-02-18T18:36:49Z"
      id: nested-loops-issuet6gpg-3201289089
      name: nested-loops-issuet6gpg[0].return-first-array
      outputs:
        exitCode: "0"
        result: '["11111111", "22222222", "33333333"]'
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 1
        memory: 1
      startedAt: "2021-02-18T18:36:46Z"
      templateName: return-first-array
      templateScope: local/nested-loops-issuet6gpg
      type: Pod
    nested-loops-issuet6gpg-3511770631:
      boundaryID: nested-loops-issuet6gpg
      children:
      - nested-loops-issuet6gpg-1552126305
      - nested-loops-issuet6gpg-2432375290
      - nested-loops-issuet6gpg-4167115091
      displayName: '[1]'
      finishedAt: "2021-02-18T18:37:06Z"
      id: nested-loops-issuet6gpg-3511770631
      name: nested-loops-issuet6gpg[1]
      phase: Succeeded
      progress: 6/6
      resourcesDuration:
        cpu: 17
        memory: 17
      startedAt: "2021-02-18T18:36:56Z"
      templateName: start
      templateScope: local/nested-loops-issuet6gpg
      type: StepGroup
    nested-loops-issuet6gpg-3579028202:
      boundaryID: nested-loops-issuet6gpg
      children:
      - nested-loops-issuet6gpg-3201289089
      displayName: '[0]'
      finishedAt: "2021-02-18T18:36:56Z"
      id: nested-loops-issuet6gpg-3579028202
      name: nested-loops-issuet6gpg[0]
      phase: Succeeded
      progress: 7/7
      resourcesDuration:
        cpu: 18
        memory: 18
      startedAt: "2021-02-18T18:36:46Z"
      templateName: start
      templateScope: local/nested-loops-issuet6gpg
      type: StepGroup
    nested-loops-issuet6gpg-4167115091:
      boundaryID: nested-loops-issuet6gpg
      children:
      - nested-loops-issuet6gpg-2438547468
      displayName: return-second-array(2:33333333)
      finishedAt: "2021-02-18T18:37:00Z"
      id: nested-loops-issuet6gpg-4167115091
      inputs:
        parameters:
        - name: param
          value: "33333333"
      name: nested-loops-issuet6gpg[1].return-second-array(2:33333333)
      outputs:
        exitCode: "0"
        result: '[{"name": "Foo - 33333333"}, {"name": "Doe - 33333333"}]'
      phase: Succeeded
      progress: 1/1
      resourcesDuration:
        cpu: 2
        memory: 2
      startedAt: "2021-02-18T18:36:56Z"
      templateName: return-second-array
      templateScope: local/nested-loops-issuet6gpg
      type: Pod
  phase: Succeeded
  progress: 7/7
  resourcesDuration:
    cpu: 18
    memory: 18
  startedAt: "2021-02-18T18:36:46Z"
time="2021-02-18T18:36:46.704Z" level=info msg="Processing workflow" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:46.704Z" level=info msg="Updated phase  -> Running" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:46.705Z" level=info msg="Steps node nested-loops-issuet6gpg initialized Running" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:46.705Z" level=info msg="StepGroup node nested-loops-issuet6gpg-3579028202 initialized Running" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:46.705Z" level=info msg="Pod node nested-loops-issuet6gpg-3201289089 initialized Pending" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:46.713Z" level=info msg="Created pod: nested-loops-issuet6gpg[0].return-first-array (nested-loops-issuet6gpg-3201289089)" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:46.714Z" level=info msg="Workflow step group node nested-loops-issuet6gpg-3579028202 not yet completed" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:46.726Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=1537786 workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.718Z" level=info msg="Processing workflow" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.718Z" level=info msg="Setting node nested-loops-issuet6gpg-3201289089 outputs"
time="2021-02-18T18:36:56.718Z" level=info msg="Updating node nested-loops-issuet6gpg-3201289089 status Pending -> Succeeded"
time="2021-02-18T18:36:56.719Z" level=info msg="Step group node nested-loops-issuet6gpg-3579028202 successful" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.719Z" level=info msg="node nested-loops-issuet6gpg-3579028202 phase Running -> Succeeded" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.719Z" level=info msg="node nested-loops-issuet6gpg-3579028202 finished: 2021-02-18 18:36:56.719426533 +0000 UTC" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.719Z" level=info msg="StepGroup node nested-loops-issuet6gpg-3511770631 initialized Running" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.719Z" level=info msg="SG Outbound nodes of nested-loops-issuet6gpg-3201289089 are [nested-loops-issuet6gpg-3201289089]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.719Z" level=info msg="Pod node nested-loops-issuet6gpg-1552126305 initialized Pending" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.728Z" level=info msg="Created pod: nested-loops-issuet6gpg[1].return-second-array(0:11111111) (nested-loops-issuet6gpg-1552126305)" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.728Z" level=info msg="Pod node nested-loops-issuet6gpg-2432375290 initialized Pending" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.739Z" level=info msg="Created pod: nested-loops-issuet6gpg[1].return-second-array(1:22222222) (nested-loops-issuet6gpg-2432375290)" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.739Z" level=info msg="Pod node nested-loops-issuet6gpg-4167115091 initialized Pending" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.758Z" level=info msg="Created pod: nested-loops-issuet6gpg[1].return-second-array(2:33333333) (nested-loops-issuet6gpg-4167115091)" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.758Z" level=info msg="Workflow step group node nested-loops-issuet6gpg-3511770631 not yet completed" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.776Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=1537832 workflow=nested-loops-issuet6gpg
time="2021-02-18T18:36:56.805Z" level=info msg="Labeled pod completed" namespace=default pod=nested-loops-issuet6gpg-3201289089
time="2021-02-18T18:37:06.728Z" level=info msg="Processing workflow" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.728Z" level=info msg="Setting node nested-loops-issuet6gpg-2432375290 outputs"
time="2021-02-18T18:37:06.728Z" level=info msg="Updating node nested-loops-issuet6gpg-2432375290 status Pending -> Succeeded"
time="2021-02-18T18:37:06.728Z" level=info msg="Setting node nested-loops-issuet6gpg-1552126305 outputs"
time="2021-02-18T18:37:06.728Z" level=info msg="Updating node nested-loops-issuet6gpg-1552126305 status Pending -> Succeeded"
time="2021-02-18T18:37:06.728Z" level=info msg="Setting node nested-loops-issuet6gpg-4167115091 outputs"
time="2021-02-18T18:37:06.728Z" level=info msg="Updating node nested-loops-issuet6gpg-4167115091 status Pending -> Succeeded"
time="2021-02-18T18:37:06.730Z" level=info msg="SG Outbound nodes of nested-loops-issuet6gpg-3201289089 are [nested-loops-issuet6gpg-3201289089]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.731Z" level=info msg="Step group node nested-loops-issuet6gpg-3511770631 successful" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.731Z" level=info msg="node nested-loops-issuet6gpg-3511770631 phase Running -> Succeeded" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.731Z" level=info msg="node nested-loops-issuet6gpg-3511770631 finished: 2021-02-18 18:37:06.731484526 +0000 UTC" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.731Z" level=info msg="StepGroup node nested-loops-issuet6gpg-2438547468 initialized Running" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.731Z" level=info msg="SG Outbound nodes of nested-loops-issuet6gpg-1552126305 are [nested-loops-issuet6gpg-1552126305]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.731Z" level=info msg="SG Outbound nodes of nested-loops-issuet6gpg-2432375290 are [nested-loops-issuet6gpg-2432375290]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.732Z" level=info msg="SG Outbound nodes of nested-loops-issuet6gpg-4167115091 are [nested-loops-issuet6gpg-4167115091]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.732Z" level=info msg="Pod node nested-loops-issuet6gpg-1329735744 initialized Pending" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.736Z" level=info msg="Created pod: nested-loops-issuet6gpg[2].print(0:[{\"name\":\"Foo - 11111111\"} {\"name\":\"Doe - 11111111\"}]) (nested-loops-issuet6gpg-1329735744)" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.736Z" level=info msg="Pod node nested-loops-issuet6gpg-2832660585 initialized Pending" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.742Z" level=info msg="Created pod: nested-loops-issuet6gpg[2].print(1:[{\"name\":\"Foo - 22222222\"} {\"name\":\"Doe - 22222222\"}]) (nested-loops-issuet6gpg-2832660585)" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.743Z" level=info msg="Pod node nested-loops-issuet6gpg-1833278630 initialized Pending" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.750Z" level=info msg="Created pod: nested-loops-issuet6gpg[2].print(2:[{\"name\":\"Foo - 33333333\"} {\"name\":\"Doe - 33333333\"}]) (nested-loops-issuet6gpg-1833278630)" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.750Z" level=info msg="Workflow step group node nested-loops-issuet6gpg-2438547468 not yet completed" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.761Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=1537906 workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:06.770Z" level=info msg="Labeled pod completed" namespace=default pod=nested-loops-issuet6gpg-2432375290
time="2021-02-18T18:37:06.785Z" level=info msg="Labeled pod completed" namespace=default pod=nested-loops-issuet6gpg-1552126305
time="2021-02-18T18:37:06.808Z" level=info msg="Labeled pod completed" namespace=default pod=nested-loops-issuet6gpg-4167115091
time="2021-02-18T18:37:16.739Z" level=info msg="Processing workflow" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.739Z" level=info msg="Setting node nested-loops-issuet6gpg-2832660585 outputs"
time="2021-02-18T18:37:16.740Z" level=info msg="Updating node nested-loops-issuet6gpg-2832660585 status Pending -> Succeeded"
time="2021-02-18T18:37:16.740Z" level=info msg="Setting node nested-loops-issuet6gpg-1329735744 outputs"
time="2021-02-18T18:37:16.740Z" level=info msg="Updating node nested-loops-issuet6gpg-1329735744 status Pending -> Succeeded"
time="2021-02-18T18:37:16.740Z" level=info msg="Setting node nested-loops-issuet6gpg-1833278630 outputs"
time="2021-02-18T18:37:16.743Z" level=info msg="Updating node nested-loops-issuet6gpg-1833278630 status Pending -> Succeeded"
time="2021-02-18T18:37:16.744Z" level=info msg="SG Outbound nodes of nested-loops-issuet6gpg-3201289089 are [nested-loops-issuet6gpg-3201289089]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="SG Outbound nodes of nested-loops-issuet6gpg-1552126305 are [nested-loops-issuet6gpg-1552126305]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="SG Outbound nodes of nested-loops-issuet6gpg-2432375290 are [nested-loops-issuet6gpg-2432375290]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="SG Outbound nodes of nested-loops-issuet6gpg-4167115091 are [nested-loops-issuet6gpg-4167115091]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="Step group node nested-loops-issuet6gpg-2438547468 successful" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="node nested-loops-issuet6gpg-2438547468 phase Running -> Succeeded" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="node nested-loops-issuet6gpg-2438547468 finished: 2021-02-18 18:37:16.745714252 +0000 UTC" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="Outbound nodes of nested-loops-issuet6gpg-1329735744 is [nested-loops-issuet6gpg-1329735744]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="Outbound nodes of nested-loops-issuet6gpg-2832660585 is [nested-loops-issuet6gpg-2832660585]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="Outbound nodes of nested-loops-issuet6gpg-1833278630 is [nested-loops-issuet6gpg-1833278630]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="Outbound nodes of nested-loops-issuet6gpg is [nested-loops-issuet6gpg-1329735744 nested-loops-issuet6gpg-2832660585 nested-loops-issuet6gpg-1833278630]" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="node nested-loops-issuet6gpg phase Running -> Succeeded" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="node nested-loops-issuet6gpg finished: 2021-02-18 18:37:16.745883979 +0000 UTC" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="Checking daemoned children of nested-loops-issuet6gpg" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="Updated phase Running -> Succeeded" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="Marking workflow completed" namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.745Z" level=info msg="Checking daemoned children of " namespace=default workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.763Z" level=info msg="Workflow update successful" namespace=default phase=Succeeded resourceVersion=1537967 workflow=nested-loops-issuet6gpg
time="2021-02-18T18:37:16.774Z" level=info msg="Labeled pod completed" namespace=default pod=nested-loops-issuet6gpg-1329735744
time="2021-02-18T18:37:16.782Z" level=info msg="Labeled pod completed" namespace=default pod=nested-loops-issuet6gpg-1833278630
time="2021-02-18T18:37:16.789Z" level=info msg="Labeled pod completed" namespace=default pod=nested-loops-issuet6gpg-2832660585

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

@simster7
Copy link
Member

Will take a look

@simster7 simster7 self-assigned this Feb 19, 2021
@simster7
Copy link
Member

return-second-array returns a list of lists, one for each pod expanded with withParam. If you want a pod for each JSON object, you need to flatten the the list of lists into a simple list. Currently you have to do this yourself (see example below), but we're working on a data: template that is intended to cover use cases just as this (#4958), so you'll be able to do this natively in Argo in the future.

Example on how to flatten:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: nested-loops-issue-
spec:
  entrypoint: start
  templates:
  - name: start
    steps:
    - - name: return-first-array
        template: return-first-array
    - - arguments:
          parameters:
          - name: param
            value: '{{item}}'
        name: return-second-array
        template: return-second-array
        withParam: '{{steps.return-first-array.outputs.result}}'
    - - arguments:
          parameters:
          - name: param
            value: '{{steps.return-second-array.outputs.result}}'
        name: flatten
        template: flatten
    - - arguments:
          parameters:
          - name: message
            value: '{{item}}'
        name: print
        template: whalesay
        withParam: '{{steps.flatten.outputs.result}}'

  - name: return-first-array
    script:
      command:
      - python
      image: python:alpine3.6
      source: |
        import json
        import sys
        json.dump(['11111111', '22222222', '33333333'], sys.stdout)

  - inputs:
      parameters:
      - name: param
    name: flatten
    script:
      command:
      - python
      image: python:alpine3.6
      source: |
        import json
        import sys
        out = []
        for upper in {{inputs.parameters.param}}:
          for lower in upper:
            out.append(lower)
        json.dump(out, sys.stdout)

  - inputs:
      parameters:
      - name: param
    name: return-second-array
    script:
      command:
      - python
      image: python:alpine3.6
      source: |
        import json
        import sys
        json.dump([{"name":"Foo - {{inputs.parameters.param}}"},{"name":"Doe - {{inputs.parameters.param}}"}], sys.stdout)

  - container:
      args:
      - '{{inputs.parameters.message}}'
      command:
      - cowsay
      image: docker/whalesay:latest
    inputs:
      parameters:
      - name: message
    name: whalesay

@simster7 simster7 added type/feature Feature request works-for-me and removed type/bug type/feature Feature request labels Feb 19, 2021
@simster7 simster7 removed their assignment Feb 22, 2021
@AdrianSkierniewski
Copy link
Author

AdrianSkierniewski commented Feb 25, 2021

@simster7 Thank you for pointing me in the right direction. It would be really cool when we can do it natively in the Argo. After running your example I realized one thing. Flattening would require all previous jobs to finish which wasn't my intention. I'd like to generate multiple branches which could continue independently and aggregate state from previous jobs.
I think that I'd need to nest it instead of running one after another.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants