Job submission issue on theta

[ndkeen]

With a master of July 31st, I tried SMS.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6 and the job failed with:

Submit job case.test
ERROR: Command: '/projects/ccsm/acme/tools/cobalt/dsub -A ClimateEnergy_3 -t 24:00:00 -n 128 -q default --mode script --env ARGS_FOR_SCRIPT='--skip-preview-namelist' .case.test' 
failed with error 'Job routed to queue "default".
Jobs smaller than 256 nodes are limited to 3 hours.
Filter /usr/bin/reroute_jobs failed' from dir '/lus/theta-fs0/projects/ClimateEnergy_3/ndkeen/SMS.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.theta_intel.20190731_181646_sb65f2'

[ndkeen]

I'm also seeing a problem where an xmlchange on the job wallclock time was not accepted due to

job is case.run USER_REQUESTED_WALLTIME None USER_REQUESTED_QUEUE None
WARNING: No queue on this system met the requirements for this job. Falling back to defaults

Not sure this is related to above, but is anyone else seeing issues submitting jobs on theta?

[amametjanov]

@jgfouca is there a recommended way to set max wall-clock times according to job node-counts? The queueing policy from https://www.alcf.anl.gov/user-guides/job-scheduling-policy-xc40-systems

Wall-clock limits are a step-wise function designed to encourage scaling:
* node count >= 128 nodes (minimum allocation): maximum 3:00:00 hours
* node count >= 256 nodes : maximum 6:00:00 hours
* node count >= 384 nodes : maximum 9:00:00 hours
* node count >= 640 nodes : maximum 12:00:00 hours
* node count >= 802 nodes : maximum 24:00:00 hours

[jgfouca]

@amametjanov , this would have to be described in the config_batch.xml entry for theta. A combination of walltimemin, walltimemax, nodemin, and nodemax should work.

[amametjanov]

I'm also seeing a problem where an xmlchange on the job wallclock time was not accepted

To avoid a reset of requested walltime, both master and maint-1.0 require

$ ./xmlchange USER_REQUESTED_WALLTIME=$new_time
$ ./xmlchange USER_REQUESTED_QUEUE=$new_queue

If only one or none of these are set prior to case.setup, cime can pick unexpected defaults.

[jgfouca]

@amametjanov , that shouldn't be necessary (from xmlchange):

    # Subtle: If the user is making specific requests for walltime and queue, we need to                                                                                                                                                       
    # store these choices in USER_REQUESTED_WALLTIME and/or USER_REQUESTED_QUEUE so they                                                                                                                                                       
    # are not lost if the user does a case.setup --reset.                                                                                                                                                                                      
    if xmlid == "JOB_WALLCLOCK_TIME":
        case.set_value("USER_REQUESTED_WALLTIME", xmlval, subgroup)

[amametjanov]

@jgfouca please give it a try on theta with maint-1.0. Noel had three different jobs launched with run_e3sm time-out because xmlchange JOB_WALLCLOCK_TIME=08:00:00 had no effect.

[jgfouca]

@ndkeen , @amametjanov, this is for SMS.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.theta_intel (both master and maint-1.0 seem to work):

% ./xmlquery JOB_WALLCLOCK_TIME

Results in group case.st_archive
	JOB_WALLCLOCK_TIME: 0:20:00

Results in group case.test
	JOB_WALLCLOCK_TIME: 24:00:00

% ./xmlchange JOB_WALLCLOCK_TIME=08:00:00 --subgroup case.test

% ./xmlquery JOB_WALLCLOCK_TIME

Results in group case.st_archive
	JOB_WALLCLOCK_TIME: 0:20:00

Results in group case.test
	JOB_WALLCLOCK_TIME: 08:00:00

% ./xmlquery USER_REQUESTED_WALLTIME

Results in group case.st_archive
	USER_REQUESTED_WALLTIME: 

Results in group case.test
	USER_REQUESTED_WALLTIME: 08:00:00

[amametjanov]

A reproducer:

azamatm@thetalogin1:~/repos/E3SM-maint> git branch
  azamat/theta-maint-1.0/update-queue-policies
* maint-1.0
  master

azamatm@thetalogin1:~/repos/E3SM-maint> ./cime/scripts/create_newcase --case /projects/ClimateEnergy_3/azamatm/SMS_P51328x1.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.th
eta_intel.20190820_newcase --res ne30_oECv3_ICG --compset A_WCYCL1850S_CMIP6 --machine theta --compiler intel --project ClimateEnergy_3 --pecount 51328x1

azamatm@thetalogin1:~/repos/E3SM-maint> cd /projects/ClimateEnergy_3/azamatm/SMS_P51328x1.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.theta_intel.20190820_newcase/

azamatm@thetalogin1:/projects/ClimateEnergy_3/azamatm/SMS_P51328x1.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.theta_intel.20190820_newcase> ./preview_run
BATCH SUBMIT: (nodes 802)
module command is /opt/modules/default/bin/modulecmd python rm craype-mic-knl PrgEnv-intel PrgEnv-cray PrgEnv-gnu intel cce cray-mpich cray-parallel-netcdf cray-hdf5-parallel cray-hdf5 cray-netcdf cray-netcdf-hdf5parallel cray-libsci craype
module command is /opt/modules/default/bin/modulecmd python load craype/2.5.12 intel/18.0.0.128 PrgEnv-intel/6.0.4 craype-mic-knl cray-mpich/7.6.2 cray-netcdf-hdf5parallel/4.4.1.1.3 cray-hdf5-parallel/1.10.0.3 cray-parallel-netcdf/1.8.1.3
Setting Environment MPICH_ENV_DISPLAY=1 
Setting Environment MPICH_VERSION_DISPLAY=1
Setting Environment MPAS_TOOL_DIR=/projects/ccsm/acme/tools/mpas
Setting Environment labeling=
Setting Environment SMP_VARS=
    case.run -> /projects/ccsm/acme/tools/cobalt/dsub   -A ClimateEnergy_3 -t 02:00:00 -n 0 -q default --mode script -M azamat@anl.gov .case.run

MPIRUN: aprun -n 0 -N 0 --cc depth -d 2 -j 2     /projects/ClimateEnergy_3/azamatm/SMS_P51328x1.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.theta_intel.20190820_newcase/bld/e3sm.exe  >> e3sm.log.$LID 2>&1

azamatm@thetalogin1:/projects/ClimateEnergy_3/azamatm/SMS_P51328x1.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.theta_intel.20190820_newcase> ./xmlchange JOB_WALLCLOCK_TIME=08:00:00
azamatm@thetalogin1:/projects/ClimateEnergy_3/azamatm/SMS_P51328x1.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.theta_intel.20190820_newcase> ./xmlquery JOB_WALLCLOCK_TIME 
Results in group case.run
        JOB_WALLCLOCK_TIME: 08:00:00

Results in group case.st_archive
        JOB_WALLCLOCK_TIME: 08:00:00

azamatm@thetalogin1:/projects/ClimateEnergy_3/azamatm/SMS_P51328x1.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.theta_intel.20190820_newcase> ./case.setup

azamatm@thetalogin1:/projects/ClimateEnergy_3/azamatm/SMS_P51328x1.ne30_oECv3_ICG.A_WCYCL1850S_CMIP6.theta_intel.20190820_newcase> ./xmlquery JOB_WALLCLOCK_TIME 
Results in group case.run
        JOB_WALLCLOCK_TIME: 02:00:00

Results in group case.st_archive
        JOB_WALLCLOCK_TIME: 08:00:00

[jgfouca]

@amametjanov , thanks. Looking at it now.

[jgfouca]

@jhkennedy , do you have any spare cycles to look at this?

[jhkennedy]

@jgfouca I can look at it on Monday.

[jhkennedy]

@amametjanov I can reproduce that behavior for maint-1.0 but not for master; are you experiencing it with master as well? Still trying to pinpoint where the problem is...

[amametjanov]

Yes, I can reproduce on master after pulling in changes in #3145 that are needed to avoid the error in the initial comment of this issue.

[amametjanov]

Fixed by #3144 (maint-1.0), #3145 (master). To keep custom wall-time after case.setup, it's recommended to either set --walltime $new_time --queue $new_queue on create_newcase launch or call

./xmlchange USER_REQUESTED_WALLTIME=$new_time &&
./xmlchange USER_REQUESTED_QUEUE=$new_queue

between create_newcase and case.setup.