Refactor access to components which are paused
Pause works with the driver and at least one active component (POP).
Active component (POP) able to correctly process resume signal.
PRE test refactored to be more flexible.
Find good restart variable to change for POP PRE test (PSURF_CUR).
Modify code and scripts to error if NINST_ESP > 1.
Fixed NCK test to not set NINST_ESP > 1
Added PRE test back to cime_developer. Fails with CIME_MODEL=acme.

… buildnml

Move utility to turn time interval into seconds from driver buildnml to utils.
Change PRE test to coarse grid for speedup.

We no longer pass rearranger options via pio_init(). We have only
implemented (for now) passing rearranger options via pio_init()
for non-async cases, but since we no longer pass rearranger opts
via pio_init (and pio_set_rearr_opts() works for both async and
non-async cases) we can get rid of the conditional ("#ifdef PIO1")
code.

Promote to warning if machine doesn't match probed machine

1) Can run from any directory
2) Pass through arguments from test-specific cs.status to general cs.status
3) Support for multiple test-ids
4) Add --summary option

Now --user-mods on the command line (including testmods) will take
precedence over the user_mods set by the compset - for user_nl files,
shell_commands and SourceMods.

I have tested this with this diff to the A compset

diff --git a/src/drivers/mct/cime_config/config_compsets.xml b/src/drivers/mct/cime_config/config_compsets.xml
index c11354e..7e6c2c9 100644
--- a/src/drivers/mct/cime_config/config_compsets.xml
+++ b/src/drivers/mct/cime_config/config_compsets.xml
@@ -40,6 +40,7 @@
   <compset>
     <alias>A</alias>
     <lname>2000_DATM%NYF_SLND_DICE%SSMI_DOCN%DOM_DROF%NYF_SGLC_SWAV</lname>
+    <user_mods>/Users/sacks/temporary/user_mods_compset</user_mods>
   </compset>

   <compset>

Along with this create_newcase command:

./create_newcase -case test_0414m -compset A -res f45_g37 \
--run-unsupported \
--user-mods-dir /Users/sacks/temporary/user_mods_command_line

where the contents of the two relevant user_mods directories are:

--- user_mods_compset/shell_commands ---
./xmlchange STOP_N=101
--- user_mods_compset/SourceMods/src.drv/mysrc.F90 ---
user_mods_compset
--- user_mods_compset/user_nl_cpl ---
user_mods_compset

--- user_mods_command_line/shell_commands ---
./xmlchange STOP_N=102
--- user_mods_command_line/SourceMods/src.drv/mysrc.F90 ---
user_mods_command_line
--- user_mods_command_line/user_nl_cpl ---
user_mods_command_line

The final contents are:

--- user_nl_cpl ---
user_mods_compset
user_mods_command_line
--- shell_commands ---
./xmlchange --force STOP_N=102
--- SourceMods/src.drv/mysrc.F90 ---
user_mods_command_line

And

$ ./xmlquery STOP_N
	STOP_N: 102

thus demonstrating that the user_mods on the command-line takes
precedence over the compset's user_mods.

This was already the policy, but was not being enforced correctly.

The previous implementation had two problems:

1. If you specified a user_mods on the command-line along with a compset
that has its own user_mods, then the compset's user_mods get applied
twice.

2. The new place where there was a call to apply_user_mods happened too
early: xmlchange commands can not be done at that point.

This fixes these problems.

I have tested this with the same changes described in
9cc7740. I tested create_newcase with
no user_mods, user_mods just from the command line, user_mods just from
the compset, and user_mods from the command line and the compset.

Fixes for PRE test

Remove conditional pio1 code

Make Machines.get_value more likely to return values of the correct type

Inform user of machine

Force user to always go through case.submit

Big cs.status upgrade

Now --user-mods on the command line (including testmods) will take
precedence over the user_mods set by the compset - for user_nl files,
shell_commands and SourceMods.

I have tested this with this diff to the A compset

diff --git a/src/drivers/mct/cime_config/config_compsets.xml b/src/drivers/mct/cime_config/config_compsets.xml
index c11354e..7e6c2c9 100644
--- a/src/drivers/mct/cime_config/config_compsets.xml
+++ b/src/drivers/mct/cime_config/config_compsets.xml
@@ -40,6 +40,7 @@
   <compset>
     <alias>A</alias>
     <lname>2000_DATM%NYF_SLND_DICE%SSMI_DOCN%DOM_DROF%NYF_SGLC_SWAV</lname>
+    <user_mods>/Users/sacks/temporary/user_mods_compset</user_mods>
   </compset>

   <compset>

Along with this create_newcase command:

./create_newcase -case test_0414m -compset A -res f45_g37 \
--run-unsupported \
--user-mods-dir /Users/sacks/temporary/user_mods_command_line

where the contents of the two relevant user_mods directories are:

--- user_mods_compset/shell_commands ---
./xmlchange STOP_N=101
--- user_mods_compset/SourceMods/src.drv/mysrc.F90 ---
user_mods_compset
--- user_mods_compset/user_nl_cpl ---
user_mods_compset

--- user_mods_command_line/shell_commands ---
./xmlchange STOP_N=102
--- user_mods_command_line/SourceMods/src.drv/mysrc.F90 ---
user_mods_command_line
--- user_mods_command_line/user_nl_cpl ---
user_mods_command_line

The final contents are:

--- user_nl_cpl ---
user_mods_compset
user_mods_command_line
--- shell_commands ---
./xmlchange --force STOP_N=102
--- SourceMods/src.drv/mysrc.F90 ---
user_mods_command_line

And

$ ./xmlquery STOP_N
	STOP_N: 102

thus demonstrating that the user_mods on the command-line takes
precedence over the compset's user_mods.

The previous implementation had two problems:

1. If you specified a user_mods on the command-line along with a compset
that has its own user_mods, then the compset's user_mods get applied
twice.

2. The new place where there was a call to apply_user_mods happened too
early: xmlchange commands can not be done at that point.

This fixes these problems.

I have tested this with the same changes described in
9cc7740. I tested create_newcase with
no user_mods, user_mods just from the command line, user_mods just from
the compset, and user_mods from the command line and the compset.

Point is: I want to make it hard to miss

…into mvertens/usermods_in_compset