Split MPMD stdout into tasks on slurm

It can be difficult to debug MPMD jobs because their logs are all written
concurrently to a single file. While the use of tags to designate which
task via the preamble and PS4 can help identify which line is from which
task, it is still difficult to follow a single task through the log,
particularly for larger MPMD jobs with dozens of tasks.

Individual stdout files are now created by using the `srun` `--output`
option. These files are written to the working directory (in `$DATA`).

Fixes: NOAA-EMC#1468

env/HERA.env

@@ -17,7 +17,7 @@ step=$1
 
 export npe_node_max=40
 export launcher="srun -l --export=ALL"
-export mpmd_opt="--multi-prog"
+export mpmd_opt="--multi-prog --output=${step}.%J.%t.out"
 
 # Configure MPI environment
 #export I_MPI_ADJUST_ALLREDUCE=5

env/JET.env

@@ -20,7 +20,7 @@ elif [[ "${PARTITION_BATCH}" = "vjet" ]]; then
   export npe_node_max=16
 fi
 export launcher="srun -l --export=ALL"
-export mpmd_opt="--multi-prog"
+export mpmd_opt="--multi-prog --output=${step}.%J.%t.out"
 
 # Configure STACK
 export OMP_STACKSIZE=2048000

env/ORION.env

@@ -17,7 +17,7 @@ step=$1
 
 export npe_node_max=40
 export launcher="srun -l --export=ALL"
-export mpmd_opt="--multi-prog"
+export mpmd_opt="--multi-prog --output=${step}.%J.%t.out"
 
 # Configure MPI environment
 export MPI_BUFS_PER_PROC=2048

env/S4.env

@@ -22,7 +22,7 @@ elif [[ ${PARTITION_BATCH} = "ivy" ]]; then
    export npe_node_max=20
 fi
 export launcher="srun -l --export=ALL"
-export mpmd_opt="--multi-prog"
+export mpmd_opt="--multi-prog --output=${step}.%J.%t.out"
 
 # Configure MPI environment
 export OMP_STACKSIZE=2048000