Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JSON script result vs JSON output parameters are treated differently for withParam item aggregation #13510

Closed
3 of 4 tasks
elliotgunton opened this issue Aug 27, 2024 · 1 comment · Fixed by #13513
Closed
3 of 4 tasks

JSON script result vs JSON output parameters are treated differently for withParam item aggregation #13510

elliotgunton opened this issue Aug 27, 2024 · 1 comment · Fixed by #13513
Assignees
@Joibel
Labels
area/controller Controller issues, panics area/looping `withParams`, `withItems`, and `withSequence` type/bug
Milestone
v3.5.x patches

Comments

@elliotgunton
Copy link

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

For a workflow where we want to have a nested fan-out, i.e. process a list-of-lists, using the .result of a script template is treated differently from using a named output parameter when collecting results, meaning we must use stdout script results to process a list-of-lists (which is not ideal).

This is due to the controller unmarshalling individual items of the results list here

argo-workflows/workflow/controller/operator.go

Line 3244 in f8f1893

err := json.Unmarshal([]byte(*node.Outputs.Result), &item)

And then re-marshalling the whole results list here

argo-workflows/workflow/controller/operator.go

Line 3252 in f8f1893

resultsJSON, err := json.Marshal(resultsList)

Which leads to a correct input parameter passed to the inner DAG:
image

Individual output parameters of the fan-out are not unmarshalled during processing here

argo-workflows/workflow/controller/operator.go

Lines 3232 to 3239 in f8f1893

param := make(map[string]string)
for _, p := range node.Outputs.Parameters {
param[p.Name] = p.Value.String()
outputParamValueList := outputParamValueLists[p.Name]
outputParamValueList = append(outputParamValueList, p.Value.String())
outputParamValueLists[p.Name] = outputParamValueList
}
paramList = append(paramList, param)

But are re-marshalled here

argo-workflows/workflow/controller/operator.go

Line 3259 in f8f1893

outputsJSON, err := json.Marshal(paramList)

Which leads to a serialized JSON string (with quotes escaped) passed to the inner DAG:
image

Resulting in the error message

withParam value could not be parsed as a JSON list: [{\"key1\": \"value1\"}, {\"key2\": \"value2\"}, {\"key3\": \"value3\"}]: invalid character '\\' looking for beginning of object key string

Investigated and confirmed with @Joibel, who tried patching the controller and got the output parameters version working, by attempting to unmarshal individual output parameters to then re-marshal them (and falling back to existing mechanism if it fails) - PR from Alan will be incoming 😄

Version(s)

v3.5.10

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: double-fan-out-using-result-
spec:
  entrypoint: main-dag
  templates:
    - inputs:
        parameters:
          - name: param1
      name: operation
      outputs:
        parameters:
          - name: output1
            valueFrom:
              path: /tmp/fan_out.json
      script:
        command:
          - python3
        image: python:3.11
        source: |-
          import os
          import sys
          sys.path.append(os.getcwd())
          import json
          try: param1 = json.loads(r'''{{inputs.parameters.param1}}''')
          except: param1 = r'''{{inputs.parameters.param1}}'''

          with open('/tmp/fan_out.json', 'w') as f:
              json.dump(param1, f)
          print(json.dumps(param1))
    - dag:
        tasks:
          - arguments:
              parameters:
                - name: param1
                  value: '{{item}}'
            name: task3
            template: operation
            withParam: '{{inputs.parameters.param1}}'
      inputs:
        parameters:
          - name: param1
      name: secondary-dag
    - dag:
        tasks:
          - arguments:
              parameters:
                - name: param1
                  value: '[[{"key1": "value1"}, {"key2": "value2"}, {"key3": "value3"}], [{"key4": "value4"}, {"key5": "value5"}]]'
            name: task1
            template: operation
          - arguments:
              parameters:
                - name: param1
                  value: '{{item}}'
            depends: task1
            name: task2
            template: operation
            withParam: '{{tasks.task1.outputs.result}}'
          - arguments:
              parameters:
                - name: param1
                  value: '{{item}}'
            depends: task2
            name: task3-dag
            template: secondary-dag
            withParam: '{{tasks.task2.outputs.result}}'
      name: main-dag
---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: double-fan-out-using-param-
spec:
  entrypoint: main-dag
  templates:
    - inputs:
        parameters:
          - name: param1
      name: operation
      outputs:
        parameters:
          - name: output1
            valueFrom:
              path: /tmp/fan_out.json
      script:
        command:
          - python3
        image: python:3.11
        source: |-
          import os
          import sys
          sys.path.append(os.getcwd())
          import json
          try: param1 = json.loads(r'''{{inputs.parameters.param1}}''')
          except: param1 = r'''{{inputs.parameters.param1}}'''

          with open('/tmp/fan_out.json', 'w') as f:
              json.dump(param1, f)
          print(json.dumps(param1))
    - dag:
        tasks:
          - arguments:
              parameters:
                - name: param1
                  value: '{{item}}'
            name: task3
            template: operation
            withParam: '{{inputs.parameters.param1}}'
      inputs:
        parameters:
          - name: param1
      name: secondary-dag
    - dag:
        tasks:
          - arguments:
              parameters:
                - name: param1
                  value: '[[{"key1": "value1"}, {"key2": "value2"}, {"key3": "value3"}], [{"key4": "value4"}, {"key5": "value5"}]]'
            name: task1
            template: operation
          - arguments:
              parameters:
                - name: param1
                  value: '{{item}}'
            depends: task1
            name: task2
            template: operation
            withParam: '{{tasks.task1.outputs.parameters.output1}}'
          - arguments:
              parameters:
                - name: param1
                  value: '{{item}}'
            depends: task2
            name: task3-dag
            template: secondary-dag
            withParam: '{{tasks.task2.outputs.parameters.output1}}'
      name: main-dag

Logs from the workflow controller

time="2024-08-27T10:02:37.702Z" level=info msg="Processing workflow" Phase= ResourceVersion=208400 namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:37.708Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=0 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:37.708Z" level=info msg="Updated phase  -> Running" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:37.708Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:37.708Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:37.708Z" level=info msg="DAG node double-fan-out-using-param-n2zxs initialized Running" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:37.708Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-3329134287, taskName task3-dag"
time="2024-08-27T10:02:37.708Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-2909490305, taskName task2"
time="2024-08-27T10:02:37.708Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-2859157448, taskName task1"
time="2024-08-27T10:02:37.708Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-2859157448, taskName task1"
time="2024-08-27T10:02:37.708Z" level=info msg="All of node double-fan-out-using-param-n2zxs.task1 dependencies [] completed" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:37.708Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:37.708Z" level=info msg="Pod node double-fan-out-using-param-n2zxs-2859157448 initialized Pending" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:37.713Z" level=info msg="Created pod: double-fan-out-using-param-n2zxs.task1 (double-fan-out-using-param-n2zxs-operation-2859157448)" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:37.713Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-2909490305, taskName task2"
time="2024-08-27T10:02:37.713Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-3329134287, taskName task3-dag"
time="2024-08-27T10:02:37.713Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-2909490305, taskName task2"
time="2024-08-27T10:02:37.713Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-3329134287, taskName task3-dag"
time="2024-08-27T10:02:37.713Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:37.713Z" level=info msg=reconcileAgentPod namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:37.722Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=208408 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.713Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=208408 namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.713Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=1 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.713Z" level=info msg="task-result changed" namespace=argo nodeID=double-fan-out-using-param-n2zxs-2859157448 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.713Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=double-fan-out-using-param-n2zxs-2859157448 old.message= old.phase=Pending old.progress=0/1 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.713Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-3329134287, taskName task3-dag"
time="2024-08-27T10:02:47.713Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-2909490305, taskName task2"
time="2024-08-27T10:02:47.713Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-2909490305, taskName task2"
time="2024-08-27T10:02:47.714Z" level=info msg="TaskGroup node double-fan-out-using-param-n2zxs-2909490305 initialized Running (message: )" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.714Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-2735842384, taskName task2(0:[{\"key1\":\"value1\"} {\"key2\":\"value2\"} {\"key3\":\"value3\"}])"
time="2024-08-27T10:02:47.714Z" level=info msg="All of node double-fan-out-using-param-n2zxs.task2(0:[{\"key1\":\"value1\"} {\"key2\":\"value2\"} {\"key3\":\"value3\"}]) dependencies [task1] completed" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.714Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.714Z" level=info msg="Pod node double-fan-out-using-param-n2zxs-2735842384 initialized Pending" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.720Z" level=info msg="Created pod: double-fan-out-using-param-n2zxs.task2(0:[{\"key1\":\"value1\"} {\"key2\":\"value2\"} {\"key3\":\"value3\"}]) (double-fan-out-using-param-n2zxs-operation-2735842384)" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.720Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-1865176775, taskName task2(1:[{\"key4\":\"value4\"} {\"key5\":\"value5\"}])"
time="2024-08-27T10:02:47.720Z" level=info msg="All of node double-fan-out-using-param-n2zxs.task2(1:[{\"key4\":\"value4\"} {\"key5\":\"value5\"}]) dependencies [task1] completed" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.721Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.721Z" level=info msg="Pod node double-fan-out-using-param-n2zxs-1865176775 initialized Pending" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.727Z" level=info msg="Created pod: double-fan-out-using-param-n2zxs.task2(1:[{\"key4\":\"value4\"} {\"key5\":\"value5\"}]) (double-fan-out-using-param-n2zxs-operation-1865176775)" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.727Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-3329134287, taskName task3-dag"
time="2024-08-27T10:02:47.727Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-3329134287, taskName task3-dag"
time="2024-08-27T10:02:47.727Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.727Z" level=info msg=reconcileAgentPod namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.739Z" level=info msg="Workflow update successful" namespace=argo phase=Running resourceVersion=208475 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:47.747Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/double-fan-out-using-param-n2zxs-operation-2859157448/labelPodCompleted
time="2024-08-27T10:02:57.722Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=208475 namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.723Z" level=info msg="Task-result reconciliation" namespace=argo numObjs=3 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.723Z" level=info msg="task-result changed" namespace=argo nodeID=double-fan-out-using-param-n2zxs-2859157448 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.723Z" level=info msg="task-result changed" namespace=argo nodeID=double-fan-out-using-param-n2zxs-1865176775 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.723Z" level=info msg="task-result changed" namespace=argo nodeID=double-fan-out-using-param-n2zxs-2735842384 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.723Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=double-fan-out-using-param-n2zxs-1865176775 old.message= old.phase=Pending old.progress=0/1 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.723Z" level=info msg="node changed" namespace=argo new.message= new.phase=Succeeded new.progress=0/1 nodeID=double-fan-out-using-param-n2zxs-2735842384 old.message= old.phase=Pending old.progress=0/1 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.724Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-3329134287, taskName task3-dag"
time="2024-08-27T10:02:57.724Z" level=info msg="node double-fan-out-using-param-n2zxs-2909490305 phase Running -> Succeeded" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.724Z" level=info msg="node double-fan-out-using-param-n2zxs-2909490305 finished: 2024-08-27 10:02:57.72440034 +0000 UTC" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.724Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-3329134287, taskName task3-dag"
time="2024-08-27T10:02:57.724Z" level=info msg="TaskGroup node double-fan-out-using-param-n2zxs-3329134287 initialized Running (message: )" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.724Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-109176038, taskName task3-dag(0:[{\\\"key1\\\": \\\"value1\\\"}, {\\\"key2\\\": \\\"value2\\\"}, {\\\"key3\\\": \\\"value3\\\"}])"
time="2024-08-27T10:02:57.724Z" level=info msg="All of node double-fan-out-using-param-n2zxs.task3-dag(0:[{\\\"key1\\\": \\\"value1\\\"}, {\\\"key2\\\": \\\"value2\\\"}, {\\\"key3\\\": \\\"value3\\\"}]) dependencies [task2] completed" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.724Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.724Z" level=info msg="DAG node double-fan-out-using-param-n2zxs-109176038 initialized Running" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.724Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-1233275690, taskName task3"
time="2024-08-27T10:02:57.724Z" level=info msg="Skipped node double-fan-out-using-param-n2zxs-1233275690 initialized Error (message: withParam value could not be parsed as a JSON list: [{\\\"key1\\\": \\\"value1\\\"}, {\\\"key2\\\": \\\"value2\\\"}, {\\\"key3\\\": \\\"value3\\\"}]: invalid character '\\\\' looking for beginning of object key string)" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.724Z" level=info msg="Outbound nodes of double-fan-out-using-param-n2zxs-109176038 set to [double-fan-out-using-param-n2zxs-1233275690]" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.724Z" level=info msg="node double-fan-out-using-param-n2zxs-109176038 phase Running -> Error" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.724Z" level=info msg="node double-fan-out-using-param-n2zxs-109176038 finished: 2024-08-27 10:02:57.724873674 +0000 UTC" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.724Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-2799279137, taskName task3-dag(1:[{\\\"key4\\\": \\\"value4\\\"}, {\\\"key5\\\": \\\"value5\\\"}])"
time="2024-08-27T10:02:57.724Z" level=info msg="All of node double-fan-out-using-param-n2zxs.task3-dag(1:[{\\\"key4\\\": \\\"value4\\\"}, {\\\"key5\\\": \\\"value5\\\"}]) dependencies [task2] completed" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.724Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg="DAG node double-fan-out-using-param-n2zxs-2799279137 initialized Running" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=warning msg="was unable to obtain the node for double-fan-out-using-param-n2zxs-3008809221, taskName task3"
time="2024-08-27T10:02:57.725Z" level=info msg="Skipped node double-fan-out-using-param-n2zxs-3008809221 initialized Error (message: withParam value could not be parsed as a JSON list: [{\\\"key4\\\": \\\"value4\\\"}, {\\\"key5\\\": \\\"value5\\\"}]: invalid character '\\\\' looking for beginning of object key string)" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg="Outbound nodes of double-fan-out-using-param-n2zxs-2799279137 set to [double-fan-out-using-param-n2zxs-3008809221]" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg="node double-fan-out-using-param-n2zxs-2799279137 phase Running -> Error" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg="node double-fan-out-using-param-n2zxs-2799279137 finished: 2024-08-27 10:02:57.725148549 +0000 UTC" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg="node double-fan-out-using-param-n2zxs-3329134287 phase Running -> Error" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg="node double-fan-out-using-param-n2zxs-3329134287 finished: 2024-08-27 10:02:57.725200757 +0000 UTC" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg="Outbound nodes of double-fan-out-using-param-n2zxs set to [double-fan-out-using-param-n2zxs-1233275690 double-fan-out-using-param-n2zxs-3008809221]" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg="node double-fan-out-using-param-n2zxs phase Running -> Error" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg="node double-fan-out-using-param-n2zxs finished: 2024-08-27 10:02:57.725286882 +0000 UTC" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg="TaskSet Reconciliation" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg=reconcileAgentPod namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg="Updated phase Running -> Error" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.725Z" level=info msg="Marking workflow completed" namespace=argo workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.735Z" level=info msg="Workflow update successful" namespace=argo phase=Error resourceVersion=208577 workflow=double-fan-out-using-param-n2zxs
time="2024-08-27T10:02:57.803Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/double-fan-out-using-param-n2zxs-operation-2735842384/labelPodCompleted
time="2024-08-27T10:02:57.803Z" level=info msg="cleaning up pod" action=labelPodCompleted key=argo/double-fan-out-using-param-n2zxs-operation-1865176775/labelPodCompleted

Logs from in your workflow's wait container

time="2024-08-27T10:02:50.896Z" level=info msg="Main container completed" error="<nil>"
time="2024-08-27T10:02:50.896Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-08-27T10:02:50.896Z" level=info msg="Saving output parameters"
time="2024-08-27T10:02:50.896Z" level=info msg="Saving path output parameter: output1"
time="2024-08-27T10:02:50.896Z" level=info msg="Copying /tmp/fan_out.json from base image layer"
time="2024-08-27T10:02:50.896Z" level=info msg="Successfully saved output parameter: output1"
time="2024-08-27T10:02:50.896Z" level=info msg="No output artifacts"
time="2024-08-27T10:02:50.904Z" level=info msg="Alloc=9382 TotalAlloc=13577 Sys=24677 NumGC=3 Goroutines=8"
time="2024-08-27T10:02:50.909Z" level=info msg="Deadline monitor stopped"
time="2024-08-27T10:02:50.909Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2024-08-27T10:02:48.887Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=argo podName=double-fan-out-using-param-n2zxs-operation-2735842384 templateName=operation version="&Version{Version:v3.5.10,BuildDate:2024-08-01T05:12:26Z,GitCommit:25829927431d9a0f46d17b72ae74aedb8d700884,GitTag:v3.5.10,GitTreeState:clean,GoVersion:go1.21.12,Compiler:gc,Platform:linux/arm64,}"
time="2024-08-27T10:02:48.895Z" level=info msg="Starting deadline monitor"
time="2024-08-27T10:02:50.897Z" level=info msg="Main container completed" error="<nil>"
time="2024-08-27T10:02:50.897Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-08-27T10:02:50.897Z" level=info msg="Saving output parameters"
time="2024-08-27T10:02:50.897Z" level=info msg="Saving path output parameter: output1"
time="2024-08-27T10:02:50.897Z" level=info msg="Copying /tmp/fan_out.json from base image layer"
time="2024-08-27T10:02:50.897Z" level=info msg="Successfully saved output parameter: output1"
time="2024-08-27T10:02:50.897Z" level=info msg="No output artifacts"
time="2024-08-27T10:02:50.904Z" level=info msg="Alloc=9269 TotalAlloc=13587 Sys=24165 NumGC=3 Goroutines=8"
time="2024-08-27T10:02:38.989Z" level=info msg="Starting deadline monitor"
time="2024-08-27T10:02:40.992Z" level=info msg="Main container completed" error="<nil>"
time="2024-08-27T10:02:40.992Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-08-27T10:02:40.992Z" level=info msg="Saving output parameters"
time="2024-08-27T10:02:40.992Z" level=info msg="Saving path output parameter: output1"
time="2024-08-27T10:02:40.992Z" level=info msg="Copying /tmp/fan_out.json from base image layer"
time="2024-08-27T10:02:40.992Z" level=info msg="Successfully saved output parameter: output1"
time="2024-08-27T10:02:40.992Z" level=info msg="No output artifacts"
time="2024-08-27T10:02:41.007Z" level=info msg="Alloc=9787 TotalAlloc=13590 Sys=23909 NumGC=3 Goroutines=8"
time="2024-08-27T10:02:41.012Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
@Joibel Joibel self-assigned this Aug 27, 2024
@Joibel Joibel added area/controller Controller issues, panics area/looping `withParams`, `withItems`, and `withSequence` labels Aug 27, 2024
Joibel added a commit to pipekit/argo-workflows that referenced this issue Aug 27, 2024
@Joibel
fix: aggregate JSON output parameters correctly
76c5e3e 
Fixes argoproj#13510

`outputs.result` of a `withItems`/`withParams` will be conditionally
`json.Unmarshal`ed.

Other output parameters are always json.Unmarshalled, which leads to
inconsitency and a complete inability to pass JSON out of withItems
outputs straight back in as `withItems`

Conditionally unmarshal the outputs.params
* avoiding it if the parameter isn't enclosed in `{` or `[`, primarily
to avoid bare numbers as they're valid JSON but need quoting here
* falling back to old behaviour if this fails

Additional e2e test tests the example case.

Signed-off-by: Alan Clucas <alan@clucas.org>
@Joibel Joibel mentioned this issue Aug 27, 2024
Merged
@Joibel
Copy link
Member

Joibel commented Aug 27, 2024

Looks to be similar/same as #5143.

@Joibel Joibel mentioned this issue Aug 30, 2024
Open
4 tasks
Joibel added a commit to pipekit/argo-workflows that referenced this issue Aug 30, 2024
@Joibel
fix: aggregate JSON output parameters correctly
80cbf2d 
Fixes argoproj#13510

`outputs.result` of a `withItems`/`withParams` will be conditionally
`json.Unmarshal`ed.

Other output parameters are always json.Unmarshalled, which leads to
inconsitency and a complete inability to pass JSON out of withItems
outputs straight back in as `withItems`

Conditionally unmarshal the outputs.params
* avoiding it if the parameter isn't enclosed in `{` or `[`, primarily
to avoid bare numbers as they're valid JSON but need quoting here
* falling back to old behaviour if this fails

Additional e2e test tests the example case.

Signed-off-by: Alan Clucas <alan@clucas.org>
@agilgur5 agilgur5 changed the title JSON-serialized script result vs JSON-serialized output parameters are treated differently for withParam item collection JSON script result vs JSON output parameters are treated differently for withParam item aggregation Aug 30, 2024
@elliotgunton elliotgunton mentioned this issue Sep 5, 2024
Closed
6 tasks
@agilgur5 agilgur5 added this to the v3.5.x patches milestone Sep 20, 2024
@elliotgunton elliotgunton mentioned this issue Oct 1, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Joibel Joibel

Labels
area/controller Controller issues, panics area/looping `withParams`, `withItems`, and `withSequence` type/bug
Projects
None yet
Milestone
v3.5.x patches
Development

Successfully merging a pull request may close this issue.

fix: aggregate JSON output parameters correctly pipekit/argo-workflows
3 participants
@Joibel @agilgur5 @elliotgunton