FATES API Updates to facilitate refactor

[adrifoster]

Description of changes

updates, mostly to module uses, to facilitate a FATES PR that involves a heavy refactor of the patch and cohort types.

Specific notes

All changes should be b4b. I also updated to the latest ccsconfig branch to bring in an MPI bug fix.

CTSM Issues Fixed (include github issue #):

Are answers expected to change (and if so in what way)?

No answer changes, everything is B4B

Any User Interface Changes (namelist or namelist defaults changes)?

No

Testing performed, if any:
fates test suite on Cheyenne - all B4B

Will test aux_clm and then test on izumi as well to ensure everything is B4B

[glemieux]

There is one minor typo in a comment.

[ekluzek]

I just have a few comments and questions. Obviously most of the work for this is inside FATES. The FATES version does need to be updated in Externals_CLM.cfg.

There isn't much at this level, and it all looks good. I'm happy for the refactoring going on. The principles of the changes make sense to me.

[glemieux]

Regression testing with aux_clm on izumi and cheyenne are underway.

[glemieux]

aux_clm regression testing on izumi is complete. All non-fates tests are b4b as expected. The fates tests result in DIFFs against ctsm5.1.dev132 as we are updating the fates pointer from sci.1.66.1_api.25.5.0 to sci.1.67.1_api.26.0.0, which includes intermediate non-b4b science updates. The fates suite tests against the last fates tag are indeed b4b as can be seen in NGEET/fates#1024 (comment).

Still waiting on aux_clm testing on cheyenne to complete.

Izumi results location: /home/glemieux/scratch/ctsm-tests/tests_pr1024-aux_clm

[glemieux]

I'm seeing all the nvhpc tests fail on cheyenne (including the expected failure) during the aux_clm regression testing: Otherwise all other expected tests are b4b:

    FAIL SMS_D.f10_f10_mg37.I2000Clm51BgcCrop.cheyenne_nvhpc.clm-crop RUN time=15 (EXPECTED FAILURE)
    FAIL SMS.f10_f10_mg37.I2000Clm50BgcCrop.cheyenne_nvhpc.clm-crop RUN time=16
    FAIL SMS.f45_f45_mg37.I2000Clm51FatesSpRsGs.cheyenne_nvhpc.clm-FatesColdSatPhen RUN time=11

The cesm logs for the unexpected failures appear to be failing in a manner similar to the expected failure.

[glemieux]

It looks like the other nvhpc tests are failing due to the externals update that incremented the ccs_config_cesm version. Retesting one of the nvhpc testmods from ctsm5.1dev132 resulted in the test passing. Upon seeing this, I manually reverted the ccs_config_cesm change on the local version of this pr branch and retested, which resulted in the test passing. Results can be found here:
/glade/u/home/glemieux/scratch/ctsm-tests/tests_pr2000-ccs_config_cesm_revert

@ekluzek given this, should I push the commit to revert the change and retest or should I mark these as expected failures? Regardless, I'm assuming I should make an issue, correct?

[ekluzek]

@glemieux leave the ccs_config version as it is, and file an issue. I think this will be resolved with an update of the externals. We do want to update externals to cesm2_3_beta15. I'm thinking that is likely to fix that. We updated the ccs_config because there was a slow down that we think this version resolves. So ust adding two expected fails isn't that bad...

[glemieux]

Roger that @ekluzek. Given that the failure mode looks very similar to #1733, I'll update that issue and reference it in the expected failures list. Once that is set, this PR should be good to go.