add timestamp to rpointer files

[jedwards4b]

Description of changes

Adds a timestamp to rpointer files in a backward compatible manor

Specific notes

Contributors other than yourself, if any:

CTSM Issues Fixed (include github issue #):

Are answers expected to change (and if so in what way)?
no
Any User Interface Changes (namelist or namelist defaults changes)?

Does this create a need to change or add documentation? Did you do so?

Submodules updated: Requires all of the following updates to work
cime6.1.49
share1.1.6
cmeps1.0.32

Testing performed, if any: will do regular

Things to do:

• Get externals and testing working
• Bring in Update cdeps with cam7 nextsw cday changes #2900 first as it changes answers
• Add a user writeup about this to ChangeLog

[wwieder]

Thanks Jim. Can this go onto b4bdev, @ekluzek ?

[ekluzek]

Thanks @jedwards4b.

@wwieder yes this totally makes sense as something coming into b4b-dev. Since, it has backwards compatibility it doesn't need to be coordinated with other CESM tags or externals. So bringing it into b4b-dev and having it go into CTSM main-dev the next time a b4b-dev tag is made (in two weeks) makes a lot of sense.

[jedwards4b]

I've run into an issue here. The clm_timemgr reads its clock information from the restart file on restart - which makes it hard to read the clock to read the restart file. It's also not a requirement to get this from the restart file as the driver has already set the clock.

[jedwards4b]

I have tested with ERS.ne30pg3_t232.BLT1850.derecho_intel.allactive-defaultio
and plan to do a complete set of cesm prealpha tests.

[samsrabin]

We discussed this at the CTSM SE meeting this morning and decided it would be in our cesm3_0_beta04 tag, which fits with @jedwards4b's timeline.

[ekluzek]

We are going to do this as a standalone tag to master, so I'll rebase to master.

[ekluzek]

@jedwards4b this is great, thanks for getting this out there for us. There's some nice improvements I saw you add (only reading the rpointer file on masterproc and catching some typos) which is great.

There are some changes that are required, and some I think would be good to do as they should be easy. They are outlined in the code changes. Right now I'm planning on just doing those changes. Feel free to comment on any of it though.

The required change is to move the updates into lnd_comp_esmf.F90 for LILAC.

Thanks again for the PR.

[ekluzek]

Running aux_clm on Derecho I'm seeing tons of tests passing 199, with only 12 pending, but 23 failing. LILAC fails as I expected, but a bunch of ERI, the SSP tests, and one ERP, a few ERS, one REP, and a few SMS tests fail at the RUN phase.

[jedwards4b]

@ekluzek I haven't yet merged the cime PR that you will need for these tests, are you using the branch?

[jedwards4b]

I just merged it - try updating to cime6.1.47

[ekluzek]

Ahh, OK, thanks @jedwards4b! I'll update to that and see how it goes.

[peverwhee]

@ekluzek i haven't run the tests on izumi but will do that in an hour or so. Here are my tests on derecho: /glade/derecho/scratch/courtneyp/aux_cam_intel_20241216142407

Here's the PR in CAM: ESCOMP/CAM#1147

[ekluzek]

It looks like the problem I'm having with the ERP tests on Izumi is due to a need to update CDEPS. Using the latest CDEPS I'm getting those tests to work now.

@peverwhee as such never mind on trying ERP tests on Izumi. But, I'll still be interested in what your testing shows on Izumi.

[ekluzek]

Note, we talked about this yesterday at the standup as something to try. I tried some testing with backing off on the CMEPS update, and although some tests worked that way, ERI tests didn't because they require a coordination of CIME and CMEPS tags. So I can't back off any of the submodule updates as they are dependent on each other.

[ekluzek]

The tests that I still have problems with:

ERP_P64x2_Ld765.f10_f10_mg37.I2000Clm60BgcCrop.derecho_intel.clm-monthly (NLCOMP RUN)
ERS_P128x1_Ld765.f10_f10_mg37.I2000Clm60Fates.derecho_intel.clm-FatesColdNoComp (NLCOMP RUN)

The test ERS_D_Ld20.f45_f45_mg37.I2000Clm50FatesRs.derecho_intel.clm-FatesColdTwoStream fails at build. But, you just need to go into the case and build by hand, and then it's fine.

In addition to the SSP spinup tests which I'm working on.

[peverwhee]

@ekluzek I kicked off the tests before I saw your message about updating CDEPS. But the CAM tests do pass on izumi with an older version (cdeps1.0.53) of CDEPS.

[ekluzek]

OK, oddly enough it's looking like the CDEPS update solved all the problems, both the SSP tests and the two outstanding broken tests. I'll rerun all of the tests and verify that this is correct, but that's what it looks like right now. I thought the tests that were dying were in the Driver and NOT DATM, but maybe it really was in DATM and that's why the update is important. What I don't get is why the SSP tests suddenly start working with this update. Improving some of the error checking may help with this for the future.

@billsacks @briandobbins and @peverwhee this is looking like really good news for our current plan.

[billsacks]

That's great - thanks @ekluzek ! Let me know if things look different and you'd like me to help look at anything.

[ekluzek]

Testing underway again. I am seeing issues with a few tests that I thought had been fixed with the CDEPS update. So we'll see we'll see what works and what doesn't after the testing is finished. From the CSEG meeting yesterday for practical reasons we may make the tag with a few things failing in the end.

[ekluzek]

I was able to resubmit many of the tests that failed at the build step due to the new issue #2915. I think that's likely a problem with our ctsm_pylib environment, so might be easy to fix.

But, the mpi-serial tests are still failing to build in trying to build mpi-serial. One problem is the mismatch in autoconf version on Izumi (1.16.1) vs. the build that likely came from Derecho (1.16.5).

ERS_D_Ld5_Mmpi-serial.1x1_vancouverCAN.I1PtClm50SpRs.izumi_nag.clm-CLM1PTStartDate.GC.ctsm5316acl_nag
ERS_D_Ld7_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropRs.izumi_intel.clm-decStart1851_noinitial.GC.ctsm5316acl_int
ERS_D_Mmpi-serial_Ld5.1x1_brazil.I2000Clm50FatesRs.izumi_nag.clm-FatesCold.GC.ctsm5316acl_nag
ERS_Ld1200_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_gnu.clm-cropMonthlyNoinitial.GC.ctsm5316acl_gnu
ERS_Ld1600_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-cropMonthlyNoinitial.GC.ctsm5316acl_int
ERS_Ld600_Mmpi-serial.1x1_smallvilleIA.I1850Clm50BgcCrop.izumi_gnu.clm-cropMonthlyNoinitial.GC.ctsm5316acl_gnu
ERS_Ly20_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-cropMonthlyNoinitial.GC.ctsm5316acl_int
ERS_Ly20_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-cropMonthlyNoinitial--clm-matrixcnOn.GC.ctsm5316acl_int
ERS_Ly3_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropQianRs.izumi_gnu.clm-cropMonthOutput.GC.ctsm5316acl_gnu
ERS_Ly5_Mmpi-serial.1x1_smallvilleIA.I1850Clm50BgcCrop.izumi_gnu.clm-ciso_monthly.GC.ctsm5316acl_gnu
ERS_Ly5_Mmpi-serial.1x1_smallvilleIA.I1850Clm50BgcCrop.izumi_gnu.clm-ciso_monthly--clm-matrixcnOn.GC.ctsm5316acl_gnu
ERS_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropQianRs.izumi_intel.clm-cropMonthOutput.GC.ctsm5316acl_int
ERS_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropQianRs.izumi_intel.clm-cropMonthOutput--clm-matrixcnOn_ignore_warnings.GC.ctsm5316acl_int
SMS_D_Ld1_Mmpi-serial.f45_f45_mg37.I2000Clm50SpRs.izumi_gnu.clm-ptsRLA.GC.ctsm5316acl_gnu
SMS_D_Ld1_Mmpi-serial.f45_f45_mg37.I2000Clm50SpRs.izumi_gnu.clm-ptsROA.GC.ctsm5316acl_gnu
SMS_D_Ld1_Mmpi-serial.f45_f45_mg37.I2000Clm50SpRs.izumi_nag.clm-ptsRLA.GC.ctsm5316acl_nag
SMS_D_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm45BgcCropQianRs.izumi_intel.clm-cropMonthOutput.GC.ctsm5316acl_int
SMS_D_Mmpi-serial_Ld5.5x5_amazon.I2000Clm60FatesRs.izumi_nag.clm-FatesCold.GC.ctsm5316acl_nag
SMS_Ld10_D_Mmpi-serial.CLM_USRDAT.I1PtClm60SpRs.izumi_nag.clm-default--clm-NEON-TOOL.GC.ctsm5316acl_nag
SMS_Ld5_Mmpi-serial.1x1_brazil.IHistClm60Bgc.izumi_gnu.clm-mimics.GC.ctsm5316acl_gnu
SMS_Ly3_Mmpi-serial.1x1_numaIA.I2000Clm50BgcDvCropQianRs.izumi_gnu.clm-ignor_warn_cropMonthOutputColdStart.GC.ctsm5316acl_gnu
SMS_Ly5_Mmpi-serial.1x1_brazil.IHistClm50BgcQianRs.izumi_intel.clm-newton_krylov_spinup.GC.ctsm5316acl_int
SMS_Ly5_Mmpi-serial.1x1_smallvilleIA.IHistClm60BgcCropQianRs.izumi_gnu.clm-gregorian_cropMonthOutput.GC.ctsm5316acl_gnu

This was working just yesterday, so I'm puzzling out what's going on here.

[jedwards4b]

@ekluzek I just tried this on izume and built case SMS_D_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm45BgcCropQianRs.izumi_intel.clm-cropMonthOutput.20241219_131416_z62hfl without any issues. Could it be something in your environment or perhaps the node you tried to build on is not configured correctly?

[ekluzek]

Ahhh, @jedwards4b good question. Thanks for trying it yourself. I suppose it could be an environment thing, but I haven't done anything with my environment. I'll try some different logins and/or a different node. Good suggestions here.

[ekluzek]

I tried it for ctsm5.3.014 and it seemed to work -- but maybe I just need to try these cases in that same window? I'll try that...

[ekluzek]

Hmmm, trying the same window where ctsm5.3.014 worked -- still didn't work. I also compared the software_environment.txt file in the CASE between @jedwards4b case and mine, and didn't see anything obvious. I sent a ticket to NRIT to see if they have ideas.

However, now all of a sudden Izumi scratch is read-only so there are other problems.

[ekluzek]

Izumi was having some readonly problems with disks and I thought that could be the problem, so I tried again. But, I got the same problem again.

HOWEVER, I DID FIGURE OUT A WORKAROUND:
I noticed this because @jedwards4b had a case that was working for him. And what I noticed is that as part of him building his case the case build modified some files in his mpi-serial directory: /home/jedwards/CTSM/libraries/mpi-serial with automake and configure. So I figured if I ran autoconf to do some of the setup in the mpi-serial directory I might get it to work. A catch to this is that the configure is going to be specific to the compiler settings for the case you are trying to build.

What worked for me (in tcshell):

cd $SRCDIR
cd libraries/mpi-serial
aclocal
autoconf
automake
source $CASEDIR/.env_mach_specific.csh
./configure
\rm -rf Makefile autom4te.cache/ config.log config.status stamp-h1 tests/.deps/ tests/Makefile
cd $CASEDIR
./case.build

Where: SRCDIR is the top level CTSM checkout, and CASEDIR is the case or test directory you are trying to build.
NOTE:

• The remove step is needed as otherwise it tells you that the mpi-serial directory is already configured and you need to run "make distclean" to clean it. Doing that does clean up the local configuration, but then for me it went back to the original error I saw. But, still I might be removing more files than I really need to, but at least something in the list asked for the clean.
• There might be extra steps in the above that aren't needed, but the above is what I got to work.
• Suggestions on how to simplify above would be great to hear about.

[ekluzek]

Oh, put that here in the PR rather than in mpi-serial space. I'll copy it to there.