Path to enable execution of PWD via CWL

[GeigerJ2]

It would be nice if we, eventually, could make it possible to execute a workflow defined through our PWD via CWL. ¹ We think this shouldn't be too difficult, and a possible approach could look like this:

1. Having a converter from the PWD graph representation to CWL
2. All processes of the workflow (executions of python code) are achieved via CWL’s CommandLineTool
3. This CommandLineTool runs a python script (we provide) that takes inputs as pickled objects, unpickles them, runs the python code (our workflow functions, also pickled), and pickles the resulting outputs
4. To achieve this, all pickled python objects could live in one directory and are named based on the respective IDs in the PWD (or similar identifiers that map the objects to the graph), such that the Python script can do the execution

This is more of a long-term vision, but we should at least add a section on comparison to CWL, or this proposed approach, in the paper.

Footnotes

1. Naturally, the other direction, CWL -> PWD would be too complicated, as we would need to support all features of CWL ↩

[jan-janssen]

I had a discussion with @elinscott in MADICES/MADICES-2025#16 about this topic. As you already pointed out the challenging part is the data transfer between the individual tasks. Still I think this is fine as the Python Workflow Definition and CWL address two rather different challenges, CWL is for interoperability of file-based workflow engines and PWD for interoperability of Python based workflow engines. So I would add a comment in the supplementary material but would not make it a part of the main manuscript.

[GeigerJ2]

Given by Gemini from JSON:

{
  "nodes": [
    { "id": 0, "type": "function", "value": "workflow.get_prod_and_div" },
    { "id": 1, "type": "function", "value": "workflow.get_sum" },
    { "id": 2, "type": "input", "name": "a", "defaultValue": 1 },
    { "id": 3, "type": "input", "name": "b", "defaultValue": 2 },
    { "id": 4, "type": "output", "name": "c" }
  ],
  "edges": [
    { "target": 0, "targetPort": "x", "source": 2, "sourcePort": null },
    { "target": 0, "targetPort": "y", "source": 3, "sourcePort": null },
    { "target": 1, "targetPort": "x", "source": 0, "sourcePort": "prod" },
    { "target": 1, "targetPort": "y", "source": 0, "sourcePort": "div" },
    { "target": 4, "targetPort": null, "source": 1, "sourcePort": "result" }
  ]
}

to CWL:

cwlVersion: v1.0
class: Workflow
id: my_workflow

inputs:
  a:
    type: int
    default: 1
  b:
    type: int
    default: 2

steps:
  get_prod_and_div_step:
    run: "#get_prod_and_div_tool"
    in:
      x: a
      y: b
    out: [prod, div]

  get_sum_step:
    run: "#get_sum_tool"
    in:
      x: get_prod_and_div_step/prod
      y: get_prod_and_div_step/div
    out: [result]

outputs:
  c:
    type: int
    outputSource: get_sum_step/result

$schemas:
- https://raw.githubusercontent.com/common-workflow-language/cwltool/main/schemas/v1.0/CommandLineTool.yml

# Dummy CommandLineTool definitions (replace with actual tool descriptions)
tools:
  get_prod_and_div_tool:
    id: "#get_prod_and_div_tool"
    class: CommandLineTool
    baseCommand: ["python", "get_prod_and_div.py"]
    inputs:
      x: int
      y: int
    outputs:
      prod:
        type: int
        outputBinding:
          glob: "prod.txt"
      div:
        type: int
        outputBinding:
          glob: "div.txt"
    requirements:
      - class: InlineJavascriptRequirement

  get_sum_tool:
    id: "#get_sum_tool"
    class: CommandLineTool
    baseCommand: ["python", "get_sum.py"]
    inputs:
      x: int
      y: int
    outputs:
      result:
        type: int
        outputBinding:
          glob: "result.txt"
    requirements:
      - class: InlineJavascriptRequirement

[jan-janssen]

I close this issue as this was implemented in #98