Added initial script to bootstrap Flux on CORAL/IBM machine

[lpottier]

Added a script that bootstraps FLux on IBM machines (CORAL like Lassen). The script takes as input: number of nodes and JSON configuration file. This script starts Flux and create 3 sub allocations and add their respective Flux URI to the JSON file.

The JSON file given as input should contains some fields, at least the number of nodes for each sub-allocation (ML, Physics and containers):

{
  "ams_app": {
    "executable": "/build_mpi/examples/ams_example",
    "modelpath": "/trained_models/debug_model.pt",
    "mpi_ranks": 2,
    "use_gpu": true,
    "use_db": true,
    "dbtype": "rmq"
  },
  "physics": {
    "nodes": 1,
    "cores": 20,
    "gpus": 2
  },
  "ml": {
    "nodes": 1,
    "cores": 10,
    "gpus": 2
  },
  "containers": {
    "nodes": 1,
    "cores": 1,
    "gpus": 0
  },
  "daemon": {
    "queue-training-init": "training-init",
    "queue-training-start": "training-start",
    "queue-training-done": "training-done"
  },
  "rabbitmq": {
    "rabbitmq-name": "testamsrabbitmq",
    "rabbitmq-password": "passwd",
    "rabbitmq-user": "lpottier",
    "rabbitmq-vhost": "/",
    "service-port": 1234,
    "service-host": "url-for-rabbitmq",
    "rabbitmq-cert": "rmq-pds.crt",
    "rabbitmq-inbound-queue": "test4",
    "rabbitmq-outbound-queue": "test3"
  }
}

The script will update the JSON file to add a block flux block:

{
  "flux": {
    "uid": "12345",
    "global_uri": "ssh://...",
    "physics_uri": "ssh://...",
    "ml_uri": "ssh://...",
    "container_uri": "ssh://..."
  }
}

To test these scripts with two nodes:

./bootstrap_flux.sh 3 config.json
./launch_flux.sh config.json

Script tested on Ruby (Slurm) with flux-core==0.50 and on Lassen (LSF) with flux-core==0.45.
Warning: on Lassen only flux-core<=0.45 will work, modern Flux fails for unknown reasons.

Updates:

• Added a general script to run everything launch_ams.sh
• One just need to first export where AMS is located with export AMS_ROOT=<AMS repo path>

Important:

• Note that this script does not load any Flux installation from Spack. This must be done before (or added into this script).
• The physics and ML allocation are exclusive so you will need at least 3 nodes to run.