ERS test failure leaves namelist in unexpected state

[mnlevy1981]

I had the following happen with an ERS test:

1. Test failed in the first leg of the restart
2. I made necessary SourceMods and rebuilt / resubmitted the test
3. TestStatus reported a failure in the namelist comparison:

2016-12-09 08:24:57: Differences in namelist 'seq_timemgr_inparm':
  BASE: history_n = -999
  COMP: history_n = 11
  BASE: history_option = 'never'
  COMP: history_option = 'ndays'
  BASE: restart_n = 11
  COMP: restart_n = 6

case.build actually threw a warning about this failure

Found difference in HIST_N: case: 11 original value -999
 Use option --force to run the test with this value or --reset to reset to original

But (as @jedwards4b pointed out in Slack) that warning message is out of date; the --force and --reset options are not available in the build script (or submit, for that matter).

I think the main cause of this failure is that case.test does not abort if the first half of a restart test fails so it makes the necessary namelist changes to launch the restart run as well; if that's an accurate assessment of the issue, throwing an exception if the first run fails and aborting before setting up the second run would be a sufficient fix.

For cases where the ERS test aborts in the second run, though (i.e. if a bug caused a variable to be omitted from a restart file but a component tried to read it anyway), there needs to be a way to reset the namelist variables... ideally case.test would call this script in case of run failures but maybe make it available to users to allow re-running of these tests. Going off the original warning I got from case.build, I think it would be nice to be able to run

$ ./case.test --reset

to reset the XML variables to their expected values at the beginning of a test. (This approach would make it easy for case.test to call this routine when a run fails as well.)

[mvertens]

That's an excellent suggestion. In CIME2 the test always reset itself on a resubmission. However, the implementation to do that was not an optimal one. I thought that a better logic had been implemented in CIME5 to do the same thing. @JEdwards - what can we expect at this point in terms of resubmissions of failed tests?

…

On Fri, Dec 9, 2016 at 8:58 AM, Michael Levy ***@***.***> wrote: I had the following happen with an ERS test: 1. Test failed in the first leg of the restart 2. I made necessary SourceMods and rebuilt / resubmitted the test 3. TestStatus reported a failure in the namelist comparison: 2016-12-09 08:24:57: Differences in namelist 'seq_timemgr_inparm': BASE: history_n = -999 COMP: history_n = 11 BASE: history_option = 'never' COMP: history_option = 'ndays' BASE: restart_n = 11 COMP: restart_n = 6 case.build actually threw a warning about this failure Found difference in HIST_N: case: 11 original value -999 Use option --force to run the test with this value or --reset to reset to original But (as @jedwards4b <https://github.com/jedwards4b> pointed out in Slack) that warning message is out of date; the --force and --reset options are not available in the build script (or submit, for that matter). I think the main cause of this failure is that case.test does not abort if the first half of a restart test fails so it makes the necessary namelist changes to launch the restart run as well; if that's an accurate assessment of the issue, throwing an exception if the first run fails and aborting before setting up the second run would be a sufficient fix. For cases where the ERS test aborts in the second run, though (i.e. if a bug caused a variable to be omitted from a restart file but a component tried to read it anyway), there needs to be a way to reset the namelist variables... ideally case.test would call this script in case of run failures but maybe make it available to users to allow re-running of these tests. Going off the original warning I got from case.build, I think it would be nice to be able to run $ ./case.test --reset to reset the XML variables to their expected values at the beginning of a test. (This approach would make it easy for case.test to call this routine when a run fails as well.) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#901>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHlxEwaa3YM3skYsg50ErJHU_K_XS8fAks5rGXqzgaJpZM4LJFJU> .

[jedwards4b]

I believe that all of the issues of this PR have been addressed as of the current master.