Maestro/Flux Interaction, Interfacing, and Behavior

[FrankD412]

When working with @Jmast on GMD related Maestro configuration, we dove fairly deep into how the Maestro/Flux adapter behaves. We have a few things to discuss/sort out, see below:

Flux Broker (use_broker) Behavior

In the case of use_broker, it can be either set to True or False. This setting is meant to control whether or not a nested instance is used to execute a Maestro step. If we use the following step as an example, we see the following behavior.

Using a nested instance

The YAML specification for setting the nested instance for running LULESH is as follows:

- name: run-lulesh
      description: Run LULESH.
      run:
          cmd: |
            $(LAUNCHER) $(LULESH)/build/lulesh2.0 -s $(SIZE) -i $(ITERATIONS) -p > $(outfile)
          depends: [make-lulesh]
          nodes: 1
          procs: 1
          cores per task: 1
          use_broker: True
          walltime: "00:10:00"

The resulting script is run under a nested instance, where $(LAUNCHER) is replaced by a flux mini -c 1 -n 1 -N 1 call, which bubbles up to the newly created sub-instance of Flux. This code path in the adapter utilizes the from_batch_command code API and results in a job listing that looks like:

> flux jobs
       JOBID USER     NAME       ST NTASKS NNODES  RUNTIME NODELIST
   ƒVzAdfAMd fdinatal run-lulesh  R      1      1   1.675m lassen9
   ƒVz2gb1iP fdinatal run-lulesh  R      1      1    1.68m lassen9
   ƒVyNxMHmR fdinatal run-lulesh  R      1      1   1.705m lassen9
   ƒVyNGKdA7 fdinatal run-lulesh  R      1      1   1.706m lassen9
   ƒVyMUN1RR fdinatal run-lulesh  R      1      1   1.706m lassen9
   ƒVyLopL6T fdinatal run-lulesh  R      1      1   1.706m lassen9
   ƒVyL6JgCo fdinatal run-lulesh  R      1      1   1.707m lassen10
   ƒVyKQH1bV fdinatal run-lulesh  R      1      1   1.707m lassen10

Forcing steps to use the global instance

When running a step under the global instance, the current version of the specification sets use_broker to False and the step looks as follows:

- name: run-lulesh
      description: Run LULESH.
      run:
          cmd: |
            $(LAUNCHER) $(LULESH)/build/lulesh2.0 -s $(SIZE) -i $(ITERATIONS) -p > $(outfile)
          depends: [make-lulesh]
          nodes: 1
          procs: 1
          cores per task: 1
          use_broker: False
          walltime: "00:10:00"

The expectation is that this would result in run-lulesh jobs running in the top level, but we see duplicate entries. These duplicates are because the from_command API is called which allocates resources, and then even further the $(LAUNCHER) in the step is replaced with a flux mini call that calls into the global instance with the end result looking like:

> flux jobs
       JOBID USER     NAME       ST NTASKS NNODES  RUNTIME NODELIST
   ƒVzAdfAMd fdinatal run-lulesh  R      1      1   1.675m lassen9
   ƒVz2gb1iP fdinatal run-lulesh  R      1      1    1.68m lassen9
   ƒVyNxMHmR fdinatal run-lulesh  R      1      1   1.705m lassen9
   ƒVyNGKdA7 fdinatal run-lulesh  R      1      1   1.706m lassen9
   ƒVyMUN1RR fdinatal run-lulesh  R      1      1   1.706m lassen9
   ƒVyLopL6T fdinatal run-lulesh  R      1      1   1.706m lassen9
   ƒVyL6JgCo fdinatal run-lulesh  R      1      1   1.707m lassen10
   ƒVyKQH1bV fdinatal run-lulesh  R      1      1   1.707m lassen10
   ƒPZyraVps fdinatal lulesh2.0   R      1      1   15.75m lassen7
   ƒPZghnoRq fdinatal lulesh2.0   R      1      1   15.76m lassen10
   ƒPZgepps9 fdinatal lulesh2.0   R      1      1   15.76m lassen10
   ƒPZgbrrJT fdinatal lulesh2.0   R      1      1   15.76m lassen10

We end up with two jobs per Maestro step since flux mini is a blocking call which sequesters a new set of resources on top of keeping the original job mapped to the run-lulesh step alive.

What we expect to see

For the two cases, we would expect to see:

use_broker	Expected Behavior	Why it's beneficial
True	A new broker with a unique URI	Useful for scheduling workers or large steps that are meant to run many smaller MPI enabled jobs. Useful for writing custom managers that reside over a subset of resources
False	A single job executed in the global instance	Allows users to run jobs as they normal would on SLURM or other conventional schedulers

Questions

1. Do the expected behaviors line up with use cases?
2. What other functionality is missing? Is there a gap in the current expectations?
3. Is Flux meant to operate this way? Does Maestro align with the best practices for Flux?

@Jmast -- This is meant to tee up our conversation next week. Feel free to chime in.
@SteVwonder @dongahn -- Just point you here since we've been discussing interfacing between Maestro and Flux.

[FrankD412]

For reference, the full specification that runs on Lassen:

description:
    name: lulesh_sample1
    description: A sample LULESH study that downloads, builds, and runs a parameter study of varying problem sizes and iterations on SLURM.

env:
    variables:
        OUTPUT_PATH: ./sample_output/lulesh

    labels:
        outfile: $(SIZE.label).$(ITERATIONS.label).log

    dependencies:
      git:
        - name: LULESH
          path: $(OUTPUT_PATH)
          url: https://github.com/LLNL/LULESH.git

batch:
    type        : flux
    host        : quartz
    bank        : baasic
    queue       : pbatch

study:
    - name: make-lulesh
      description: Build the MPI enabled version of LULESH.
      run:
          cmd: |
            cd $(LULESH)
            mkdir build
            cd build
            cmake -WITH_MPI=Off -WITH_OPENMP=Off ..
            make
          depends: []

    - name: run-lulesh
      description: Run LULESH.
      run:
          cmd: |
            $(LAUNCHER) $(LULESH)/build/lulesh2.0 -s $(SIZE) -i $(ITERATIONS) -p > $(outfile)
            #$(LULESH)/build/lulesh2.0 -s $(SIZE) -i $(ITERATIONS) -p > $(outfile)
          depends: [make-lulesh]
          nodes: 1
          procs: 1
          cores per task: 1
          use_broker: True
          walltime: "00:10:00"

global.parameters:
    SIZE:
        values  : [100, 100, 100, 200, 200, 200, 300, 300, 300]
        label   : SIZE.%%
    ITERATIONS:
        values  : [10, 20, 30, 10, 20, 30, 10, 20, 30]
        label   : ITER.%%

[SteVwonder]

the current version of the specification sets use_broker to False and the step looks as follows:

The expectation is that this would result in run-lulesh jobs running in the top level, but we see duplicate entries. These duplicates are because the from_command API is called which allocates resources, and then even further the $(LAUNCHER) in the step is replaced with a flux mini call that calls into the global instance with the end result looking like:

Now that the force_broker defaults to True, what are your thoughts on modifying the behavior of force_broker=False so that the cmd is just run "locally" as a subprocess of maestro. That would avoid the duplicate job issue under Flux. More generally, this would allow users to write a cmd with the $(LAUNCHER) present and have it work portably under Slurm, under LSF, under Flux with force_broker=True and under Flux with force_broker=False. And then the only difference (from the users perspective) between force_broker=True and force_broker=False is whether or not they get a nested Flux instance.

[FrankD412]

I was thinking about that, but isn't flux mini run a blocking call? I was wondering if in these cases, we'd need to detect that we're dealing with a force_broker=False case and change the $(LAUNCHER) to a flux mini submit so that the subprocess can return.

In the case of force_broker=True, it's my understanding that the from_nested_command(script, ...) equates to flux start script -- in those cases should we just launch the nested instance with the script and default the $(LAUNCHER) to an empty string?