Short term archiver not copying MPAS history files.

[rljacob]

In a one year simulation of a low-res fully coupled ACME case with monthly output, the short term archiver is not moving the mpas history files.

The entry in config_archive (just for ocean) is:

  <comp_archive_spec compname="mpaso" compclass="ocn">
    <rest_file_extension>\.rst.*</rest_file_extension>
    <hist_file_extension>\.hist.*</hist_file_extension>
    <rest_history_varname>unset</rest_history_varname>
    <rpointer>
      <rpointer_file>rpointer.ocn</rpointer_file>
    </rpointer>
  </comp_archive_spec>

mpas outputs a lot of files. Here is a sample:

mpaso.hist.am.globalStats.0001-01-01.nc
mpaso.hist.am.globalStats.0001-02-01.nc
...
mpaso.hist.am.layerVolumeWeightedAverage.0001-01-01.nc
mpaso.hist.am.layerVolumeWeightedAverage.0001-02-01.nc
...
mpaso.hist.am.meridionalHeatTransport.0001-01-01.nc
mpaso.hist.am.meridionalHeatTransport.0001-02-01.nc
...

Full list: https://gist.github.com/rljacob/5ef9f4b0e6d08d9f1ac3e32f0b517622

The batch system output from running the archive script isn't very helpful:

-------------------------------------------
doing short term archiving for mpaso (ocn)
-------------------------------------------
histfiles_savein_rundir [] 

Full output: https://gist.github.com/rljacob/9597f76408973802210f0a6bd72b254b

What are we missing?

[bertinia]

You can try defining a more specific regex in the XML. For example,
change this line:
<hist_file_extension>\.hist.*</hist_file_extension>
to be:
<hist_file_extension>\.hist.*\.*\.*\.nc</hist_file_extension>

[rljacob]

Still not working. From the debug output of st_archive, there's:
"short term archiving suffix is Wlowrnext.mpaso..hist....nc"

Is the archiver expecting the history name to start with the case name (Wlowrnext in this case)? Because they don't and that would explain the behavior.

[mvertens]

Yes - that is the problem. The short term archiver has some hard-wired assumptions that are valid for CESM but clearly are not extensible. The following is assumed as a naming convention and is in the _archive_history_files routine in case_st_archive.py: rundir = case.get_value("RUNDIR") for suffix in archive.get_hist_file_extensions(archive_entry): for i in range(ninst): if compname == 'dart': newsuffix = casename + suffix else: if ninst_string: newsuffix = casename + '.' + compname + ".*" + ninst_string[i] + suffix else: newsuffix = casename + '.' + compname + ".*" + suffix logger.debug("short term archiving suffix is %s " %newsuffix) pfile = re.compile(newsuffix) histfiles = [f for f in os.listdir(rundir) if pfile.search(f)] So short term archiving is assuming that all history files start with the casename - as you point out. We really need to rethink the entire strategy. But as a short term fix - until we do so - you could add a new if block that would be if compname == 'dart': newsuffix = casename + suffix elif compname == "mpas": newsuffix = "xxxxxx" <====== else: if ninst_string: newsuffix = casename + '.' + compname + ".*" + ninst_string[i] + suffix else: newsuffix = casename + '.' + compname + ".*" + suffix

[rljacob]

That worked! What do I do for the restart files? They also don't start with the casename but the archiver assumes they do.

Here's what the restarts look like:

mpaso.rst.0002-01-01_00000.nc
mpaso.rst.am.timeSeriesStatsClimatology.0002-01-01_00000.nc
mpaso.rst.am.timeSeriesStatsCustom.0002-01-01_00000.nc
mpascice.rst.0002-01-01_00000.nc
mpascice.rst.am.timeSeriesStatsMonthly.0002-01-01_00000.nc

[mvertens]

In the _archive_restarts routine you need to add a similar block for mpas - see below That said, I don't see how the pattern would ever equal "dart" in what we is there now - but that is another issue. # get file_extension suffixes for suffix in archive.get_rest_file_extensions(archive_entry): for i in range(ninst): restfiles = "" if compname == "mpas": files = xxxxx <============== else: pattern = r"%s\.%s\d*.*" % (casename, compname) if pattern != "dart" and pfile = re.compile(pattern) files = [f for f in os.listdir(rundir) if pfile.search(f)] if ninst_strings: pattern = ninst_strings[i] + suffix + datename pfile = re.compile(pattern) restfiles = [f for f in files if pfile.search(f)] else: pattern = suffix + datename pfile = re.compile(pattern) restfiles = [f for f in files if pfile.search(f)] else: pattern = suffix pfile = re.compile(pattern) restfiles = [f for f in os.listdir(rundir) if pfile.search(f)]

…

On Sat, Mar 18, 2017 at 4:03 PM, Robert Jacob ***@***.***> wrote: That worked! What do I do for the restart files? They also don't start with the casename but the archiver assumes they do. Here's what the restarts look like: mpaso.rst.0002-01-01_00000.ncmpaso.rst.am.timeSeriesStatsClimatology.0002-01-01_00000.ncmpaso.rst.am.timeSeriesStatsCustom.0002-01-01_00000.ncmpascice.rst.0002-01-01_00000.ncmpascice.rst.am.timeSeriesStatsMonthly.0002-01-01_00000.nc — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1252 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHlxEx6ctt3sHc1zTTpy5dqF3rqnIVEdks5rnFS_gaJpZM4MfoUw> .

[mvertens]

@rljacob - has this been fixed. I thought it was.

[rljacob]

Only on the ACME side. Hasn't been brought over here yet so leave open.

[rljacob]

Added in PR #1634