problem with st_archive and short runs (< 1 month) #594

bertinia · 2016-09-27T17:50:51Z

If the short term archiver is run prior to the end of month boundary then it errors out because it can't find the component monthly history file corresponding to the component restart history file.

The st_archive log errors from a F-WACCM case are:

==> f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002.st_archive.570708 <==
out=
err=ncdump: nhfil: No such variable
histfiles_savein_rundir ['f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002.cam.h0.1979-01.nc', 'f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002.cam.h1.1979-01-01-00000.nc', 'f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002.cam.h2.1979-01-01-00000.nc']
doing short term archiving for clm (lnd)
removing interim restart file /glade/scratch/mmills/f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002/run/f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002.clm2.r.1979-01-06-00000.nc
WARNING: ncdump -v locfnh /glade/scratch/mmills/f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002/run/f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002.clm2.rh0.1979-01-06-00000.nc failed rc=1
out=
err=ncdump: locfnh: No such variable
removing interim restart file /glade/scratch/mmills/f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002/run/f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002.clm2.rh0.1979-01-06-00000.nc
ERROR: restart file /glade/scratch/mmills/f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002/run/f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002.clm2.h0.1979-01.nc does not exist

rljacob · 2016-10-07T18:09:51Z

Can this be reproduced using CIME only?

drmikemills · 2016-10-27T17:34:17Z

I am still having this issue, and it also occurs when the run ends on an end-of-month boundary. I am using my copy of Francis’ Phase 3 sandbox, which I have here:

/glade/u/home/mmills/cesm/phase3_n03_cam5_4_90_compsets/

This is the latest case to fail:

/glade/p/work/mmills/case/f.e20.FSDW.f09_f09.phase3_n03_cam5_4_90.02

See f.e20.FSDW.f09_f09.phase3_n03_cam5_4_90.02.st_archive.259967

ERROR: restart file /glade/scratch/mmills/f.e20.FSDW.f09_f09.phase3_n03_cam5_4_90.02/run/f.e20.FSDW.f09_f09.phase3_n03_cam5_4_90.02.clm2.h0.1990-09.nc does not exist

I am not seeing any .clm2.h0. files in the run directory or the short-term archive:

/glade/scratch/mmills/f.e20.FSDW.f09_f09.phase3_n03_cam5_4_90.02/run

/glade/scratch/mmills/archive/f.e20.FSDW.f09_f09.phase3_n03_cam5_4_90.02

drmikemills · 2016-10-27T18:56:29Z

I have located the CLM history output. They are in the short-term archive under atm/hist, instead of under lnd/hist. So are the history files for CICE and MOSART.

See:

/glade/scratch/mmills/archive/f.e15.FWscAMIP.f19_f19.misc08_cam5_4_81.002/atm/hist
/glade/scratch/mmills/archive/f.e20.FSDW.f09_f09.phase3_n03_cam5_4_90.02/atm/hist
/glade/scratch/mmills/archive/f.e20.FWAMIP.f09_f09.phase3_n03_cam5_4_90.03/atm/hist

bertinia · 2017-08-11T17:59:42Z

@drmikemills came across another problem with the STA where a cam restart history
file for a 1 day run was not copied correctly into the restart set. The file did remain
in the rundir. Here's the specifics for the case in order to reproduce the problem:

casedir: /glade/p/work/mmills/case/f.e20.FWHIST.f09_f09_mg17.20thC.190_02

I ran for 1 day. The short-term archive is here:

/glade2/scratch2/mmills/archive/f.e20.FWHIST.f09_f09_mg17.20thC.190_02/rest/1970-01-02-00000

This file did not get put in the short-term archive, but it is required for continuing the run:

f.e20.FWHIST.f09_f09_mg17.20thC.190_02.cam.rh0.1970-01-02-00000.nc

Source code: /glade/p/work/hannay/cesm_tags/cesm2_0_alpha07b_clm4_5_16_r251

jedwards4b · 2017-08-11T18:59:56Z

In config_archive.xml I see <rest_file_extension>\.r[sh]\.*</rest_file_extension>
which would not match a file*.cam.rh0.*- should this include a \d?

mvertens · 2017-08-11T19:24:23Z

@jedwards4 - thanks for looking into this. I think this is exactly the problem. Alice - can you have Mike rerun his case with this change to the config_archive.xml file?

…

On Fri, Aug 11, 2017 at 12:59 PM, jedwards4b ***@***.***> wrote: In config_archive.xml I see <rest_file_extension>.r[sh].*< /rest_file_extension> which would not match a file *.cam.rh0.* - should this include a \d? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#594 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHlxE50Y943OvBNmWucztUBcTk3dzAZqks5sXKSsgaJpZM4KH9lZ> .

jedwards4b · 2017-08-11T19:26:22Z

To be clear I think that this needs to be two seperate entries:

 <rest_file_extension>\.rh\d\.*</rest_file_extension>
 <rest_file_extension>\.rs\.*</rest_file_extension>

mvertens · 2017-08-11T20:00:31Z

Can you please outline what you thing these two entries should be.

…

On Fri, Aug 11, 2017 at 1:26 PM, jedwards4b ***@***.***> wrote: To be clear I think that this needs to be two seperate entries: <rest_file_extension>\.r[sh]\.*</rest_file_extension> <rest_file_extension>\.r[sh]\.*</rest_file_extension> — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#594 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHlxEx_ZIYUg_dfEMK1EIccFp7rGEEAVks5sXKrfgaJpZM4KH9lZ> .

jedwards4b · 2017-08-11T20:02:24Z

See my previous comment.

<rest_file_extension>\.rh\d\.*</rest_file_extension>
<rest_file_extension>\.rs\.*</rest_file_extension>

bertinia · 2017-08-12T04:53:42Z

I tested this change and it worked correctly. I'll issue a PR with the update to config_archive.xml.

…

On Fri, Aug 11, 2017 at 2:02 PM, jedwards4b ***@***.***> wrote: See my previous comment. <rest_file_extension>\.rh\d\.*</rest_file_extension> <rest_file_extension>\.rs\.*</rest_file_extension> — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#594 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKO3mcqImY728cXaXNJ6GjYD9L7YtLiGks5sXLNRgaJpZM4KH9lZ> .

-- Alice Bertini NCAR - CSEG Software Engineer

Add irt test Adds IRT (Interum Restart Test) using the system_test_compare_two methodology This test revealed a bug in the driver config_archive.xml file which I also fixed. Changed cime_developer tests from ERS to IRT Test suite: scripts_regression_tests.py Test baseline: Test namelist changes: Test status: bit for bit Fixes #594 Fixes #1731 User interface changes?: Update gh-pages html (Y/N)?: Code review:

bertinia added ty: Bug CRITICAL user scripts labels Sep 27, 2016

rljacob changed the title ~~problem with st_archive and short runs (< 1 month)~~ problem with st_archive and short runs (< 1 month) Sep 28, 2016

rljacob added the ready label Sep 28, 2016

rljacob assigned mvertens Dec 9, 2016

rljacob added in progress and removed ready labels Dec 9, 2016

mvertens pushed a commit that referenced this issue Dec 9, 2016

fixed bug #594

688d1d4

This was referenced Dec 9, 2016

Starchive fixes #910

Closed

short term archiving bugfix #912

Merged

rljacob closed this as completed Dec 16, 2016

rljacob removed the in progress label Dec 16, 2016

bertinia reopened this Aug 11, 2017

bertinia self-assigned this Aug 11, 2017

bertinia added this to the cesm2 milestone Aug 11, 2017

bertinia mentioned this issue Aug 25, 2017

Documentation tool updates and st_archive fix #1838

Merged

jedwards4b mentioned this issue Aug 25, 2017

Add irt test #1837

Merged

jgfouca closed this as completed in #1837 Aug 25, 2017

ekluzek mentioned this issue Dec 16, 2017

Automatic restarts will fail for CLM if the last restarts are written before the end of the simulation ESCOMP/CTSM#154

Closed

whannah1 mentioned this issue Dec 7, 2023

Warnings from case.st_archive -- "nhfil: No such variable and locfnh: No such variable" E3SM-Project/E3SM#4615

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

problem with st_archive and short runs (< 1 month) #594

problem with st_archive and short runs (< 1 month) #594

bertinia commented Sep 27, 2016 •

edited by rljacob

Loading

rljacob commented Oct 7, 2016

drmikemills commented Oct 27, 2016 •

edited

Loading

drmikemills commented Oct 27, 2016

bertinia commented Aug 11, 2017

jedwards4b commented Aug 11, 2017 •

edited

Loading

mvertens commented Aug 11, 2017 via email

jedwards4b commented Aug 11, 2017 •

edited

Loading

mvertens commented Aug 11, 2017 via email

jedwards4b commented Aug 11, 2017

bertinia commented Aug 12, 2017 via email

problem with st_archive and short runs (< 1 month) #594

problem with st_archive and short runs (< 1 month) #594

Comments

bertinia commented Sep 27, 2016 • edited by rljacob Loading

rljacob commented Oct 7, 2016

drmikemills commented Oct 27, 2016 • edited Loading

drmikemills commented Oct 27, 2016

bertinia commented Aug 11, 2017

jedwards4b commented Aug 11, 2017 • edited Loading

mvertens commented Aug 11, 2017 via email

jedwards4b commented Aug 11, 2017 • edited Loading

mvertens commented Aug 11, 2017 via email

jedwards4b commented Aug 11, 2017

bertinia commented Aug 12, 2017 via email

bertinia commented Sep 27, 2016 •

edited by rljacob

Loading

drmikemills commented Oct 27, 2016 •

edited

Loading

jedwards4b commented Aug 11, 2017 •

edited

Loading

jedwards4b commented Aug 11, 2017 •

edited

Loading