Apply stream reference_time to individual output records

[matthewhoffman]

Previously, a stream's reference_time only applied to determining the
filename breaks. Right now, output is always relative to the time of
the start of the current simulation (even if it is a restart). There is
no way to force, say, monthly output (i.e. on the 1st of every month) if
your restart does not start on the first of a month.

This commit extends reference_time to also apply to
determining when individual records should be written. Thus, output
stream behavior will not change after a restart.

This is done by adding a new routine to mpas_timekeeping that adjusts
the prevRingTime of an alarm to be an integer multiple of the stream's
output_interval away from the stream's reference_time. This routine is
called for each output stream created by the stream manager.

Fixes Issue #1307.

[matthewhoffman]

To test, I set up a restart test that does a cold start followed by a restart.
The cold start has the following namelist and streams settings. The key thing is to make the output and restart intervals indivisible (here using 5 and 7 days).

    config_dt = '0000-00-01_00:00:00'
    config_do_restart = .false.
    config_start_time = 0000-01-01_00:00:00
    config_stop_time = '0000-01-08_00:00:00'

<immutable_stream name="restart"
                  filename_interval="output_interval"
                  clobber_mode="truncate"
                  reference_time="0000-01-01_00:00:00"
                  precision="double"
                  output_interval="0000-00-07_00:00:00"
                  filename_template="rst.$Y.$M.$D.nc"
                  input_interval="initial_only"
                  type="input;output"
                  input_interal="initial_only"/>

<stream name="output"
        clobber_mode="overwrite"
        reference_time="0000-01-01_00:00:00"
        output_interval="0000-00-05_00:00:00"
        filename_template="output.nc"
        type="output">
	<var name="xtime"/>
</stream>

The restart is identical except for:

    config_do_restart = .true.
    config_start_time = 'file'
    config_stop_time = '0000-01-20_00:00:00'

With this configuration the output file contains:

 xtime =
  "0000-01-01_00:00:00                                             ",
  "0000-01-06_00:00:00                                             ",
  "0000-01-13_00:00:00                                             ",
  "0000-01-18_00:00:00                                             " ;

This is incorrect because we asked for output to occur every 5 days, so we should have gotten it on day 11 and 16 instead of day 13 and 18 - the restart on day 8 threw it off.

Rerunning the test with this branch, the output is correct:

 xtime =
  "0000-01-01_00:00:00                                             ",
  "0000-01-06_00:00:00                                             ",
  "0000-01-11_00:00:00                                             ",
  "0000-01-16_00:00:00                                             " ;

I tried a second test where the stream reference_time is in the future, but I get an unrelated error that exists prior to this branch:

At line 1096 of file mpas_timekeeping.F
Fortran runtime error: Bad integer for item 1 in list input

We could try to fix this in this branch if anyone is willing to figure it out. I took a quick look but was unable to get a usable backtrace.

Note that I included code for reverse-running clocks but I did not test it.

[mark-petersen]

@matthewhoffman I added this I8 kind in an extra commit to get it to compile.

[mark-petersen]

Yes! I was able to mimic your exact test, with success, when I merged this into ocean/develop. Thank you @matthewhoffman, that is one more annoying item off our list. This is a bit like swatting flies.

[mark-petersen]

An important note for ACME is that the restart and all output streams should have
reference_time="0000-01-01_00:00:00"
added to it to get this to work. @wolfe, I made a note of that.

[mark-petersen]

@matthewhoffman I'm ready to take off in progress and don't merge if you are.

[mark-petersen]

@mgduda do you have time to look at this today? We are making a big push to merge these framework PRs and update framework in all the ACME cores.

[mark-petersen]

I ran a test on the original issue, which is monthly output and sub-monthly restarts. I ran the ocean core for 70 days with 10 day restarts, then continue for another 70 days with ten day restarts. Output file is monthly. It worked perfectly.

head of ocean/develop

 xtime =
  "0001-01-01_00:00:00                                             ",
  "0001-02-01_00:00:00                                             ",
  "0001-03-01_00:00:00                                             ",
  "0001-03-12_00:00:00                                             ",
  "0001-04-12_00:00:00                                             ",
  "0001-05-12_00:00:00                                             " ;

This PR merged into ocean/develop

 xtime =
  "0001-01-01_00:00:00                                             ",
  "0001-02-01_00:00:00                                             ",
  "0001-03-01_00:00:00                                             ",
  "0001-04-01_00:00:00                                             ",
  "0001-05-01_00:00:00                                             " ;

Note you still need
config_write_output_on_startup = .false
or you will get an extra entry on restart, which is expected.

For my reference, runs at:

/lustre/scratch3/turquoise/mpeterse/runs/t51p/ocean/global_ocean/QU480/default/forward
/lustre/scratch3/turquoise/mpeterse/runs/t51q/ocean/global_ocean/QU480/default/forward

[mark-petersen]

@mgduda I know you are busy. Any chance I can merge this in today? This is the last framework PR we need, then we can update all our cores and test.

[mark-petersen]

@mgduda Sorry to jump ahead on this, but we have a big ACME code update with framework updates today. This PR worked perfectly in my testing, and solves a problem that has been a long-standing annoyance in ACME. I merged locally and test compiled the atmosphere core with gnu and intel.

[matthewhoffman]

@mark-petersen , thanks for sponsoring this PR. BTW, regarding the comment you made about still needing config_write_output_on_startup = .false, you could consider if a change like this:
eaa703f
would be appropriate for the ocean core to eliminate that.