Add a slurm workflow manager

[linsword13]

Example usage:

ramble:
  variants:
    workflow_manager: slurm
  variables:
    processes_per_node: 1
  applications:
    hostname:
      workloads:
        local:
          experiments:
            test:
              variables:
                n_nodes: 1

  # The batch_submit is defined by the workflow object
ramble on

  # Executors for query and cancel are available
ramble on --executor "{batch_query}"
ramble on --executor "{batch_cancel}"

  # The status report is also available in analyze
ramble workspace analyze -p

[linsword13]

Marking this as a draft, given that this PR will need to be updated if #804 is in.

[douglasjacobsen]

Something I just found out, if a job is completed and you run ramble on --executor='{query_job}' then it'll say the job failed (even if you haven't analyzed it.

We might want to see if there's a way to use something like scontrol to get the status of completed jobs.

[linsword13]

Something I just found out, if a job is completed and you run ramble on --executor='{query_job}' then it'll say the job failed (even if you haven't analyzed it.

We might want to see if there's a way to use something like scontrol to get the status of completed jobs.

Hm that's unexpected. I did have a sacct command to try to query for status of completed jobs, if squeue comes up empty. I will look into it.

[douglasjacobsen]

Let me try it again, it might be that analyzing the job failed for some reason.

[douglasjacobsen]

Yeah, I think I just had a weird issue with my experiments.

Can we also add default -o and -e pragmas to the generated submission script? This can place the slurm logs into the experiment directory.

[linsword13]

Yeah, I think I just had a weird issue with my experiments.

Can we also add default -o and -e pragmas to the generated submission script? This can place the slurm logs into the experiment directory.

Thanks for trying out! Also added the -o and -e.

[linsword13]

Marking this as a draft, given that this PR will need to be updated if #804 is in.

This is now updated. It currently uses changes in #814.

[douglasjacobsen]

This is looking great. One small request. :)

The merge-base changed after approval.