Job submission issue on Summit

[dqwu]

On Summit, case.submit fails with the following error message:
ERROR: Command: '-env "all, ARGS_FOR_SCRIPT=--resubmit" bsub -q batch -W 02:00 -P cli115 .case.run' failed with error '/bin/sh: - : invalid option'

This issue seems to be caused by PR #3092

Before that PR was merged, preview_run shows:

    SUBMIT CMD:
      bsub -q batch -W 02:00 -P cli115 -env "all, ARGS_FOR_SCRIPT=--resubmit" .case.run

After that PR was merged, preview_run shows:

    SUBMIT CMD:
      -env "all, ARGS_FOR_SCRIPT=--resubmit" bsub -q batch -W 02:00 -P cli115 .case.run

[jgfouca]

Looking at it now.

[jgfouca]

@dqwu , does this only happen for resubmissions? I was unable to reproduce the error for a simple case.

[dqwu]

@jgfouca
Below are detailed steps to reproduce it with a simple F case.
Use GNU compiler as there are some known build errors with default XL compiler.

git clone https://github.com/E3SM-Project/E3SM.git

cd E3SM

git submodule update --init

cd cime/scripts

./create_newcase --compiler=gnu --case FC5AV1C-H01B_ne4_ne4 --compset FC5AV1C-H01B --res ne4_ne4

cd FC5AV1C-H01B_ne4_ne4

./case.setup

./preview_run

./case.build

./case.submit

preview_run shows:

SUBMIT CMD:
  -env "all, ARGS_FOR_SCRIPT=--resubmit" bsub -q batch -W 02:00 -P cli115 .case.run

case.submit shows:

...
Check case OK
submit_jobs case.run
Submit job case.run
Submitting job script -env "all, ARGS_FOR_SCRIPT=--resubmit" bsub -q batch -W 02:00 -P cli115 .case.run
ERROR: Command: '-env "all, ARGS_FOR_SCRIPT=--resubmit" bsub -q batch -W 02:00 -P cli115 .case.run' failed with error '/bin/sh: - : invalid option'

@sarats , could you please try to reproduce it on Summit? Thanks.

[sarats]

Yes, I tried preview_run after setting up a case and it shows the extraneous stuff before bsub invocation.

[jgfouca]

@dqwu , ok, i see the problem now

[amametjanov]

@jgfouca there is a similar issue on Theta with the latest master: env-vars are not getting resolved in preview_run output:

    MPIRUN (job=case.test):
      aprun -n 9280 -N 64 --cc depth -d 1 -j 1 $ENV{SMP_VARS} $ENV{labeling} /projects/ClimateEnergy_3/azamatm/SMS.ne120_oRRS18v3_ICG.A_WCYCL2000_H01AS.theta_intel.cam-cosplite.20190816_174816_6yenqk/bld/e3sm.exe   >> e3sm.log.$LID 2>&1

[jgfouca]

@amametjanov , ok, looking at it

[jgfouca]

@amametjanov , fixed.

[amametjanov]

Worked, thanks.