history write optimization

[jedwards4b]

Description of changes

optimize history write

Specific notes

Same as #2114
Contributors other than yourself, if any:

CTSM Issues Fixed (include github issue #):

Are answers expected to change (and if so in what way)? No

Any User Interface Changes (namelist or namelist defaults changes)? No

Testing performed, if any: None, but normal testing will be done

[wwieder]

Hi Jim,

Thanks for making this PR, I'm assuming this makes writing out history files more efficiently?
Can you provide an example how it will change variable names in the history files? How may this change post processing workflows?

[ekluzek]

@wwieder this is the same as the earlier one on sorting, just to the cesm2.2 release branch.

It shouldn't change post processing workflows, if you check for variables by name. It would if you assume a certain order, which would be a horrible assumption to make and something you shouldn't do.

@jedwards4b is the cesm2.2 release branch important? I thought we had mostly abandoned it? I suppose it might be important to have the release version working for Derecho though.

[jedwards4b]

@ekluzek this release branch is supporting the ASD run from Louisa Emmons

[ekluzek]

I sent testing on Cheyenne, but basically all tests failed with issues in the changed file. I'll see if I can get it working.

[ekluzek]

The issue is that the sorting that came in on ctsm5.1.dev018 wasn't in the much older cesm2.2.0 release version. So the sorting should be added in first and then the changes from #2114 applied to that.

[ekluzek]

@billsacks helped me to see this was far easier than I had originally thought it was. So I've made a simple change to get it to compile. So I'll move forward on the testing here. Hopefully, that is the only issue.

[ekluzek]

Got it to compile and the following test PASSes...

SMS_D_Ld5_Mmpi-serial.1x1_mexicocityMEX.I1PtClm50SpRsGs.cheyenne_intel.clm-default

[ekluzek]

Testing on cheyenne has the nuopc test failing because of an issue in CDEPS.

The following tests fail because of problems the restart history files don't compare exactly...

FAIL ERP_D.f10_f10_musgs.IHistClm50Bgc.cheyenne_gnu.clm-decStart COMPARE_base_rest
FAIL ERP_D.f10_f10_musgs.IHistClm50Bgc.cheyenne_intel.clm-decStart COMPARE_base_rest
FAIL ERP_D.f10_f10_musgs.IHistClm50Bgc.cheyenne_intel.clm-decStart MEMCOMP Error: Memory usage increase > 10% from baseline
FAIL ERP_D_Ld10_P36x2.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-ciso_decStart COMPARE_base_rest
FAIL ERP_D_Ld5.f10_f10_musgs.I2000Clm50Sp.cheyenne_intel.clm-decStart COMPARE_base_rest
FAIL ERP_D_Ld5.f10_f10_musgs.I2000Clm50Sp.cheyenne_intel.clm-decStart MEMCOMP Error: Memory usage increase > 10% from baseline
FAIL ERS_Ln9.ne0ARCTICne30x4_ne0ARCTICne30x4_mt12.IHistClm50SpGs.cheyenne_intel.clm-clm50cam6LndTuningMode_1979Start COMPARE_base_rest

[ekluzek]

On izumi:

The following test fails at run

ERP_D_Ld5_P48x1.f10_f10_musgs.I2000Clm50Sp.izumi_nag.clm-o3

and this test fails at compare like the ones on cheyenne:

ERP_D_P48x1.f10_f10_musgs.IHistClm50Bgc.izumi_nag.clm-decStart

[ekluzek]

Tests that are failing are ERP tests with BGC on, transient, and a December Start date that goes over the year boundary.

[ekluzek]

Note, that these tests PASSed for the previous tag, and I just ran one again on that tag and it passes for me again. So it isn't something that comes in from machine updates.

ERP_D_Ld10_P36x2.f10_f10_musgs.IHistClm50BgcCrop.cheyenne_intel.clm-ciso_decStart

The only things that have changed are the history sorting, and a minor change to the cime and mosart tags.

[ekluzek]

It looks like the problem is with the MOSART tag. So I'll back it out to make the tag. The issue is that it can't find the comparison file for the Restart case. The restart files look like they compare identically. So it's more of a glitch in testing than in the capability itself.

[ekluzek]

The NUOPC test passes when I unload python and conda, so that python2 is the default. Otherwise it has an error in preview_namelists in DATM buildnml because it uses python2 syntax rather than python3.

[ekluzek]

OK, the test

ERS_Ln9.ne0ARCTICne30x4_ne0ARCTICne30x4_mt12.IHistClm50SpGs.cheyenne_intel.clm-clm50cam6LndTuningMode_1979Start

Is still failing COMPARE base_rest, and I don't know why