Many tests fail to build on Izumi and have to be resent

[ekluzek]

Brief summary of bug

In testing for ctsm5.3.016 a bunch of the tests initially fail with a build problem, and then I had to go back into each one and run:

cd $TESTDIR
./case.build
./case.submit

So it works in the end, but you have to manually go in and get the tests to work again.

This is with:

cime6.1.49

and likely a fix either needs to go into cime and/or ccs_config? Or is it we just need an update of the ctsm_pylib environment?

General bug information

CTSM version you are using: ctsm5.3.016

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: Some tests on Izumi in the aux_clm testlist

Details of bug

Using the parse cs.status script I see things like this...

parse_cime.cs.status cs.status.ctsm5316acl_* -s

ERS_D_Ld5_Mmpi-serial.1x1_vancouverCAN.I1PtClm50SpRs.izumi_nag.clm-CLM1PTStartDate	(SHAREDLIB_BUILD NLCOMP SUBMIT)		
ERS_D_Ld7_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropRs.izumi_intel.clm-decStart1851_noinitial	(SHAREDLIB_BUILD NLCOMP SUBMIT)		
ERS_D_Mmpi-serial_Ld5.1x1_brazil.I2000Clm50FatesRs.izumi_nag.clm-FatesCold	(SHAREDLIB_BUILD NLCOMP SUBMIT)		
ERS_Ld1200_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_gnu.clm-cropMonthlyNoinitial	(SHAREDLIB_BUILD NLCOMP SUBMIT)		
ERS_Ld1600_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-cropMonthlyNoinitial	(SHAREDLIB_BUILD NLCOMP SUBMIT)		
ERS_Ld600_Mmpi-serial.1x1_smallvilleIA.I1850Clm50BgcCrop.izumi_gnu.clm-cropMonthlyNoinitial	(SHAREDLIB_BUILD NLCOMP SUBMIT)		
ERS_Ly20_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-cropMonthlyNoinitial	(SHAREDLIB_BUILD NLCOMP SUBMIT)		
ERS_Ly20_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-cropMonthlyNoinitial--clm-matrixcnOn	(SHAREDLIB_BUILD NLCOMP SUBMIT)		
ERS_Ly3_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropQianRs.izumi_gnu.clm-cropMonthOutput	(SHAREDLIB_BUILD NLCOMP SUBMIT)		
ERS_Ly5_Mmpi-serial.1x1_smallvilleIA.I1850Clm50BgcCrop.izumi_gnu.clm-ciso_monthly	(SHAREDLIB_BUILD NLCOMP SUBMIT)		
ERS_Ly5_Mmpi-serial.1x1_smallvilleIA.I1850Clm50BgcCrop.izumi_gnu.clm-ciso_monthly--clm-matrixcnOn	(SHAREDLIB_BUILD NLCOMP SUBMIT)		
ERS_Ly6_Mmpi-serial.1x1_smallvilleIA.IHistClm50BgcCropQianRs.izumi_intel.clm-cropMonthOutput	(SHAREDLIB_BUILD NLCOMP SUBMI

Important details of your setup / configuration so we can reproduce the bug

The process I did was:

login to Izumi:
conda activate ctsm_pylib
cd $CTSMTESTDIR
./run_sys_tests -c ctsm5.3.015 -g ctsm5.3.016 -s aux_clm  --testid-base ctsm5316acl --baseline-root /fs/cgd/csm/ccsm_baselines  --verbose >& run_sys_tests.log & 

Important output or errors that show the problem

Seems to be a problem with using fromisoformat from datetime

 ---------------------------------------------------
2024-12-18 14:45:07: SHAREDLIB_BUILD FAILED for test 'ERP_D_Ld5_P48x1.f10_f10_mg37.I1850Clm60Bgc.izumi_nag.clm-ciso--clm-matrixcnOn'.
Command: ./case.build --sharedlib-only
Output: WARNING: Found difference in test BFBFLAG: case: True original value False
 Successfully created new case ERP_D_Ld5_P48x1.f10_f10_mg37.I1850Clm60Bgc.izumi_nag.clm-ciso--clm-matrixcnOn.GC.ctsm5316acl_nag from clone case ERP_D_Ld5_P48x1.f10_f10_mg37.I1850Clm60Bgc.izumi_nag.clm-ciso--clm-matrixcnOn.GC.ctsm5316acl_nag 
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Building test for ERP in directory /scratch/cluster/erik/tests_ctsm5316acl/ERP_D_Ld5_P48x1.f10_f10_mg37.I1850Clm60Bgc.izumi_nag.clm-ciso--clm-matrixcnOn.GC.ctsm5316acl_nag
WARNING: Test case setup failed. Case2 has been removed, but the main case may be in an inconsistent state. If you want to rerun this test, you should create a new test rather than trying to rerun this one.
Traceback (most recent call last):
  File "./case.build", line 267, in <module>
    _main_func(__doc__)
  File "./case.build", line 226, in _main_func
    test = find_system_test(testname, case)(case)
  File "/fs/cgd/data0/erik/ctsm_worktree/quickfix/cime/CIME/SystemTests/erp.py", line 29, in __init__
    **kwargs
  File "/fs/cgd/data0/erik/ctsm_worktree/quickfix/cime/CIME/SystemTests/restart_tests.py", line 30, in __init__
    **kwargs
  File "/fs/cgd/data0/erik/ctsm_worktree/quickfix/cime/CIME/SystemTests/system_tests_compare_two.py", line 146, in __init__
    self._setup_cases_if_not_yet_done()
  File "/fs/cgd/data0/erik/ctsm_worktree/quickfix/cime/CIME/SystemTests/system_tests_compare_two.py", line 450, in _setup_cases_if_not_yet_done
    self._setup_cases()
  File "/fs/cgd/data0/erik/ctsm_worktree/quickfix/cime/CIME/SystemTests/system_tests_compare_two.py", line 540, in _setup_cases
    self._case_one_setup()
  File "/fs/cgd/data0/erik/ctsm_worktree/quickfix/cime/CIME/SystemTests/restart_tests.py", line 36, in _case_one_setup
    self._set_restart_interval()
  File "/fs/cgd/data0/erik/ctsm_worktree/quickfix/cime/CIME/SystemTests/system_tests_common.py", line 198, in _set_restart_interval
    startdatetime = datetime.fromisoformat(startdate) + timedelta(
AttributeError: type object 'datetime.datetime' has no attribute 'fromisoformat'

[ekluzek]

from isoformat was added in python 3.7, and then added functionality in 3.11. So possibly this will work if we update ctsm_pylib to 3.11?