USRDAT resolution issues for both MCT and NUOPC drivers

[ekluzek]

The following test is failing for me in ctsm5.1.dev030

SMS_D_Lm1_Mmpi-serial.CLM_USRDAT.I1PtClm50SpRs.cheyenne_intel.clm-USUMB

with cime5.8.39

It fails to submit with the following error...

env_batch.xml appears to have changed, regenerating batch scripts
manual edits to these file will be lost!

Cannot download file since it lives outside of the input_data_root '/glade/p/cesm/cseg/inputdata'
Cannot download file since it lives outside of the input_data_root '/glade/p/cesm/cseg/inputdata'
Cannot download file since it lives outside of the input_data_root '/glade/p/cesm/cseg/inputdata'
Cannot download file since it lives outside of the input_data_root '/glade/p/cesm/cseg/inputdata'
Cannot download file since it lives outside of the input_data_root '/glade/p/cesm/cseg/inputdata'
Cannot download file since it lives outside of the input_data_root '/glade/p/cesm/cseg/inputdata'

ERROR: Could not find all inputdata on any server

check_input_data shows the following...

./check_input_data
Setting Environment OMP_STACKSIZE=1024M
Setting Environment TMPDIR=/glade/scratch/erik
Setting Environment MPI_TYPE_DEPTH=16
Setting Environment MPI_IB_CONGESTED=1
Setting resource.RLIMIT_STACK to -1 from (307200000, -1)
Loading input file list: 'Buildconf/ctsm.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/datm.input_data_list'
Model datm missing file domain1 = 'null/null'
Model datm missing file file1 = 'null/null'

DOMAIN files seem to be set correctly. But, the datm streams files are getting set like this in datm_in...

&shr_strdata_nml
datamode = "CLMNCEP"
domainfile = "/glade/p/cesm/cseg/inputdata/lnd/clm2/PTCLMmydatafiles.c171024/1x1pt_US-UMB/domain.lnd.1x1pt_US-UMB_navy.171024.nc"
dtlimit = 1.5, 1.5, 1.5
fillalgo = "nn", "nn", "nn"
fillmask = "nomask", "nomask", "nomask"
fillread = "NOT_SET", "NOT_SET", "NOT_SET"
fillwrite = "NOT_SET", "NOT_SET", "NOT_SET"
mapalgo = "nn", "nn", "nn"
mapmask = "nomask", "nomask", "nomask"
mapread = "NOT_SET", "NOT_SET", "NOT_SET"
mapwrite = "NOT_SET", "NOT_SET", "NOT_SET"
readmode = "single", "single", "single"
streams = "datm.streams.txt.CLM1PT.UNSET 1999 -999 -999",
"datm.streams.txt.presaero.clim_2000 1 2000 2000",
"datm.streams.txt.topo.observed 1 1 1"
taxmode = "extend", "extend", "extend"
tintalgo = "linear", "linear", "lower"
vectors = "null"
/

All of these values set to NOT_SET or UNSET are causing problems.

[ekluzek]

The NUOPC case also fails.

SMS_D_Lm1_Mmpi-serial_Vnuopc.CLM_USRDAT.I1PtClm50SpRs.cheyenne_intel.clm-USUMB

I think it's for similar reasons. But, it might also indicate issues in CDEPS or CMEPS.

[billsacks]

@ekluzek in my run of the test suite, I'm seeing different errors for the mct vs. nuopc tests: the mct test (SMS_D_Lm1_Mmpi-serial.CLM_USRDAT.I1PtClm50SpRs.cheyenne_intel.clm-USUMB) fails as above, in the SUBMIT stage, but the nuopc test (SMS_D_Lm1_Mmpi-serial_Vnuopc.CLM_USRDAT.I1PtClm50SpRs.cheyenne_intel.clm-USUMB) gets past submit and fails in the run with:

 ERROR: (dshr_mod:dshr_mesh_init) ERROR: model_meshfile UNSET does not exist
Image              PC                Routine            Line        Source
cesm.exe           00000000042495AA  Unknown               Unknown  Unknown
cesm.exe           0000000003912972  shr_abort_mod_mp_         114  shr_abort_mod.F90
cesm.exe           00000000039127D8  shr_abort_mod_mp_          61  shr_abort_mod.F90
cesm.exe           000000000380274F  dshr_mod_mp_dshr_         301  dshr_mod.F90
cesm.exe           000000000078F0F7  atm_comp_nuopc_mp         340  atm_comp_nuopc.F90
libesmf.so         00002B4FC342D9F7  _ZN5ESMCI6FTable1        2009  ESMCI_FTable.C
libesmf.so         00002B4FC34293AB  ESMCI_FTableCallE         747  ESMCI_FTable.C
libesmf.so         00002B4FC397B322  _ZN5ESMCI3VMK5ent        2041  ESMCI_VMKernel.C
libesmf.so         00002B4FC3998012  _ZN5ESMCI2VM5ente        1211  ESMCI_VM.C
libesmf.so         00002B4FC3429A94  c_esmc_ftablecall         899  ESMCI_FTable.C
libesmf.so         00002B4FC3FBFB6D  esmf_compmod_mp_e        1210  ESMF_Comp.F90
libesmf.so         00002B4FC4683015  esmf_gridcompmod_        1405  ESMF_GridComp.F90
libesmf.so         00002B4FC54644D3  nuopc_driver_mp_l        2460  NUOPC_Driver.F90
libesmf.so         00002B4FC544FD7E  nuopc_driver_mp_i        1743  NUOPC_Driver.F90
libesmf.so         00002B4FC342D9F7  _ZN5ESMCI6FTable1        2009  ESMCI_FTable.C
libesmf.so         00002B4FC34293AB  ESMCI_FTableCallE         747  ESMCI_FTable.C
libesmf.so         00002B4FC397B322  _ZN5ESMCI3VMK5ent        2041  ESMCI_VMKernel.C
libesmf.so         00002B4FC3998012  _ZN5ESMCI2VM5ente        1211  ESMCI_VM.C
libesmf.so         00002B4FC3429A94  c_esmc_ftablecall         899  ESMCI_FTable.C
libesmf.so         00002B4FC3FBFB6D  esmf_compmod_mp_e        1210  ESMF_Comp.F90
libesmf.so         00002B4FC4683015  esmf_gridcompmod_        1405  ESMF_GridComp.F90
libesmf.so         00002B4FC54644D3  nuopc_driver_mp_l        2460  NUOPC_Driver.F90
libesmf.so         00002B4FC544FFBB  nuopc_driver_mp_i        1747  NUOPC_Driver.F90
libesmf.so         00002B4FC54186FB  nuopc_driver_mp_i         439  NUOPC_Driver.F90
libesmf.so         00002B4FC342D9F7  _ZN5ESMCI6FTable1        2009  ESMCI_FTable.C
libesmf.so         00002B4FC34293AB  ESMCI_FTableCallE         747  ESMCI_FTable.C
libesmf.so         00002B4FC397B322  _ZN5ESMCI3VMK5ent        2041  ESMCI_VMKernel.C
libesmf.so         00002B4FC3998012  _ZN5ESMCI2VM5ente        1211  ESMCI_VM.C
libesmf.so         00002B4FC3429A94  c_esmc_ftablecall         899  ESMCI_FTable.C
libesmf.so         00002B4FC3FBFB6D  esmf_compmod_mp_e        1210  ESMF_Comp.F90
libesmf.so         00002B4FC4683015  esmf_gridcompmod_        1405  ESMF_GridComp.F90
cesm.exe           000000000042DA8D  MAIN__                    145  esmApp.F90
cesm.exe           000000000040F8E2  Unknown               Unknown  Unknown
libc.so.6          00002B4FC968B6E5  __libc_start_main     Unknown  Unknown
cesm.exe           000000000040F7E9  Unknown               Unknown  Unknown

It may be that the underlying issue is similar - or maybe not.

[ekluzek]

The problem seems to be that both ATM_GRID and LND_GRID are getting set to "UNSET". So there's something wrong in how that gets set.

@bishtgautam it looks like this problem applies to ELM for ELM_USRDAT as well.

[bishtgautam]

Thanks, @ekluzek. For ELM, CIME PR-3845 fixed the issue.

[jedwards4b]

For nuopc CLM_USRDAT requires that a lat and lon be provided before the case can be built.
For mct it was removed after discussion with you lead me to the impression that that was what you wanted. To restore the mct case add

    <domain name="CLM_USRDAT">                                                                                              
      <nx>1</nx> <ny>1</ny>                                                                                                 
      <file>$DIN_LOC_ROOT/share/domains/domain.clm/domain.lnd.${CLM_USRDAT_NAME}_navy.nc</file>                             
      <desc>user specified domain - only valid for DATM/CLM compset</desc>                                                  
    </domain>                                                                                                               
                                   

to the domains for driver="mct" around line 1876 of config_grids.xml

[jedwards4b]

@ekluzek Also this didn't break anything for elm and @bishtgautam since the mod was in cesm config section of cime.

[ekluzek]

@jedwards4b that's actually not the problem that I'm seeing, as what I'm seeing is that ATM_GRID and LND_GRID aren't getting set, and that's causing downstream problems. I am getting the domain files set, so the above won't help.

You are correct that I did ask you to make that change and there's an issue that corresponds with it as we don't use the above generic named domain files. So yes you did correctly make that change as I asked you to. So absolutely thank you for doing that! Please don't be hesitant to do something I ask -- in fear I'll change my mind later! :-) Although I honestly can't make that claim all of the time -- but at least I'm not in this case.

I actually don't think this problem with ATM_GRID and LND_GRID is due to the changes you made, I think it's something else. I think it just happened to come to cime at the same time as your changes. I'll find out for sure when I track it down (and I think I almost have it).

But, that's why I'm concerned about ELM is that I think the ATM_GRID and LND_GRID issue will apply to both cesm and e3sm.