Add controlled stop when detect timing problems with lateral boundary data

[davegill]

TYPE: bug fix

KEYWORDS: LBC, valid time

SOURCE: identified by Michael Duda (NCAR/MMM), fixed internally

DESCRIPTION OF CHANGES:
Problem:

1. If a user tried to start a simulation after the last LBC valid period, the
   WRF model would get into a nearly infinite loop and print out repeated statements:

 THIS TIME 2000-01-24_18:00:00, NEXT TIME 2000-01-25_00:00:00
d01 2000-01-25_06:00:00  Input data is acceptable to use: wrfbdy_d01
           2  input_wrf: wrf_get_next_time current_date: 2000-01-24_18:00:00 Status =           -4
d01 2000-01-25_06:00:00  ---- ERROR: Ran out of valid boundary conditions in file wrfbdy_d01

2. If a user tries to extend the model simulation beyond that valid times of the LBC, the code
   behavior is not controlled (nearly infinite loops on some machines, or runtime errors with a backtrace
   on other machines).

Solution:
In another routine, the lateral boundary condition is read to get to the
correct time. Once inside of share/input_wrf.F, we should be at the
correct time. There is no need to try to get to the next time. In this
particular case, the effort to get to the next time fails, but we try
again (and again and again). This solution fixes both problems identified
above.

ISSUE:
Fixes #769 "WRF doesn't halt when beginning LBC time is not in wrfbdy_d01 file"

LIST OF MODIFIED FILES:
M share/input_wrf.F

TESTS CONDUCTED:

1. Without fix, start the model after the last valid time of the LBC file => lots of repeated messages

 THIS TIME 2000-01-24_18:00:00, NEXT TIME 2000-01-25_00:00:00
d01 2000-01-25_06:00:00  Input data is acceptable to use: wrfbdy_d01
           2  input_wrf: wrf_get_next_time current_date: 2000-01-24_18:00:00 Status =           -4
d01 2000-01-25_06:00:00  ---- ERROR: Ran out of valid boundary conditions in file wrfbdy_d01

2. With this fix, when LBC stops at 2000 01 25 00, and WRF starts at 2000 01 25 06

d01 2000-01-25_06:00:00  Input data is acceptable to use: wrfbdy_d01
 THIS TIME 2000-01-24_12:00:00, NEXT TIME 2000-01-24_18:00:00
d01 2000-01-25_06:00:00  Input data is acceptable to use: wrfbdy_d01
 THIS TIME 2000-01-24_18:00:00, NEXT TIME 2000-01-25_00:00:00
d01 2000-01-25_06:00:00  Input data is acceptable to use: wrfbdy_d01
           2  input_wrf: wrf_get_next_time current_date: 2000-01-24_18:00:00 Status =           -4
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:    1134
 ---- ERROR: Ran out of valid boundary conditions in file wrfbdy_d01
-------------------------------------------

3. Without this fix, if we try to extend the module simulation beyond the valid lateral boundary times

Timing for main: time 2000-01-24_23:54:00 on domain   1:    0.53782 elapsed seconds
Timing for main: time 2000-01-24_23:57:00 on domain   1:    0.51111 elapsed seconds
Timing for main: time 2000-01-25_00:00:00 on domain   1:    0.54507 elapsed seconds
Timing for Writing wrfout_d01_2000-01-25_00:00:00 for domain        1:    0.03793 elapsed seconds
d01 2000-01-25_00:00:00  Input data is acceptable to use: wrfbdy_d01
           2  input_wrf: wrf_get_next_time current_date: 2000-01-25_00:00:00 Status =           -4
d01 2000-01-25_00:00:00  ---- ERROR: Ran out of valid boundary conditions in file wrfbdy_d01
At line 777 of file module_date_time.f90
Fortran runtime error: Bad value during integer read

Error termination. Backtrace:
#0  0x10e67c36c
#1  0x10e67d075
#2  0x10e67d7e9

4. With this fix, if we try to extend the module simulation beyond the valid lateral boundary times

Timing for main: time 2000-01-24_23:54:00 on domain   1:    0.60755 elapsed seconds
Timing for main: time 2000-01-24_23:57:00 on domain   1:    0.57641 elapsed seconds
Timing for main: time 2000-01-25_00:00:00 on domain   1:    0.60817 elapsed seconds
Timing for Writing wrfout_d01_2000-01-25_00:00:00 for domain        1:    0.04499 elapsed seconds
d01 2000-01-25_00:00:00  Input data is acceptable to use: wrfbdy_d01
           2  input_wrf: wrf_get_next_time current_date: 2000-01-25_00:00:00 Status =           -4
-------------- FATAL CALLED ---------------
FATAL CALLED FROM FILE:  <stdin>  LINE:    1134
 ---- ERROR: Ran out of valid boundary conditions in file wrfbdy_d01
-------------------------------------------

MMM Classroom regtest; em_real, nmm, em_chem; GNU only

Individual tests:
SUCCESS_RUN_WRF_d01_em_real_32_em_chem_1
SUCCESS_RUN_WRF_d01_em_real_32_em_chem_2
SUCCESS_RUN_WRF_d01_em_real_32_em_chem_5
SUCCESS_RUN_WRF_d01_em_real_32_em_real_03FD
SUCCESS_RUN_WRF_d01_em_real_32_em_real_07NE
SUCCESS_RUN_WRF_d01_em_real_32_em_real_10
SUCCESS_RUN_WRF_d01_em_real_32_em_real_11
SUCCESS_RUN_WRF_d01_em_real_33_em_real_03FD
SUCCESS_RUN_WRF_d01_em_real_33_em_real_07NE
SUCCESS_RUN_WRF_d01_em_real_33_em_real_10
SUCCESS_RUN_WRF_d01_em_real_33_em_real_11
SUCCESS_RUN_WRF_d01_em_real_34_em_chem_1
SUCCESS_RUN_WRF_d01_em_real_34_em_chem_2
SUCCESS_RUN_WRF_d01_em_real_34_em_chem_5
SUCCESS_RUN_WRF_d01_em_real_34_em_real_03FD
SUCCESS_RUN_WRF_d01_em_real_34_em_real_07NE
SUCCESS_RUN_WRF_d01_em_real_34_em_real_10
SUCCESS_RUN_WRF_d01_em_real_34_em_real_11
SUCCESS_RUN_WRF_d01_nmm_real_32_nmm_nest_01
SUCCESS_RUN_WRF_d01_nmm_real_32_nmm_nest_03
SUCCESS_RUN_WRF_d01_nmm_real_32_nmm_nest_04a
SUCCESS_RUN_WRF_d01_nmm_real_32_nmm_nest_06
SUCCESS_RUN_WRF_d01_nmm_real_34_nmm_nest_01
SUCCESS_RUN_WRF_d01_nmm_real_34_nmm_nest_03
SUCCESS_RUN_WRF_d01_nmm_real_34_nmm_nest_04a
SUCCESS_RUN_WRF_d01_nmm_real_34_nmm_nest_06

Comparison tests:
SUCCESS_RUN_WRF_d01_em_real_32_em_real_03FD vs SUCCESS_RUN_WRF_d01_em_real_33_em_real_03FD status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_03FD vs SUCCESS_RUN_WRF_d01_em_real_34_em_real_03FD status = 0
Files SUCCESS_RUN_WRF_d01_em_real_32_em_real_07NE and SUCCESS_RUN_WRF_d01_em_real_33_em_real_07NE differ
SUCCESS_RUN_WRF_d01_em_real_32_em_real_07NE vs SUCCESS_RUN_WRF_d01_em_real_33_em_real_07NE status = 1
SUCCESS_RUN_WRF_d01_em_real_32_em_real_07NE vs SUCCESS_RUN_WRF_d01_em_real_34_em_real_07NE status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_10 vs SUCCESS_RUN_WRF_d01_em_real_33_em_real_10 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_10 vs SUCCESS_RUN_WRF_d01_em_real_34_em_real_10 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_11 vs SUCCESS_RUN_WRF_d01_em_real_33_em_real_11 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_real_11 vs SUCCESS_RUN_WRF_d01_em_real_34_em_real_11 status = 0
SUCCESS_RUN_WRF_d01_nmm_real_32_nmm_nest_01 vs SUCCESS_RUN_WRF_d01_nmm_real_34_nmm_nest_01 status = 0
SUCCESS_RUN_WRF_d01_nmm_real_32_nmm_nest_03 vs SUCCESS_RUN_WRF_d01_nmm_real_34_nmm_nest_03 status = 0
SUCCESS_RUN_WRF_d01_nmm_real_32_nmm_nest_04a vs SUCCESS_RUN_WRF_d01_nmm_real_34_nmm_nest_04a status = 0
SUCCESS_RUN_WRF_d01_nmm_real_32_nmm_nest_06 vs SUCCESS_RUN_WRF_d01_nmm_real_34_nmm_nest_06 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_chem_1 vs SUCCESS_RUN_WRF_d01_em_real_34_em_chem_1 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_chem_2 vs SUCCESS_RUN_WRF_d01_em_real_34_em_chem_2 status = 0
SUCCESS_RUN_WRF_d01_em_real_32_em_chem_5 vs SUCCESS_RUN_WRF_d01_em_real_34_em_chem_5 status = 0

[davegill]

@kkeene44 @dudhia @smileMchen
Folks,
Again, we can put these into release-v4.1.1, so no rush.

[jamiebresch]

Just a note that the infinite print also happens when the specified forecast range is longer than the available bdy length. For example, having wrfbdy out to 36-h, but asks the model to run to 48-h.

[dudhia]

maybe it can be added to Known Problems for V4.1

[jamiebresch]

I think the problem exists since V3.9.

[kkeene44]

I'll check that it started with V3.9 and will take care of putting the problem on all of the known problems pages.

[davegill]

@jamiebresch

Just a note that the infinite print also happens when the specified forecast range is longer than the available bdy length. For example, having wrfbdy out to 36-h, but asks the model to run to 48-h.

Jamie,
I just tried your suggested test. If I extend the simulation period beyond the valid times covered by the LBC file, I get different behaviors.

1. With the new fix, there is a controlled stop (which is good).
2. With the original code, I get an error that looks like it was trying to read more data that did not
   exist. This is the bad behavior.

So, it looks like this PR fixes the problem that you mention. I will update the PR commit message to include that information.