Testing updates

[forsyth2]

Improvements to the testing process. Resolves #540. Resolves #530. Possibly #549.

[forsyth2]

Runs so far

Run 1

vars = "LAISHA,LAISUN"

tail -n 20 ts_land_monthly_1850-1851-0002.o474780
2024-02-21 21:04:46,214 [INFO]: handler.py(cmorize:247) >> lai: creating CMOR variable with CMOR axis objects.
2024-02-21 21:04:46,214_214:INFO:cmorize:lai: creating CMOR variable with CMOR axis objects.
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/e3sm_to_cmip/__main__.py", line 912, in _run_parallel
    out = res.result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
100%|██████████| 1/1 [00:01<00:00,  1.25s/it]
2024-02-21 21:04:46,699 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-02-21 21:04:46,699 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-02-21 21:04:46,699_699:INFO:_run_parallel:0 of 1 handlers complete
2024-02-21 21:04:46,699 [ERROR]: __main__.py(_run_parallel:934) >> lai failed to complete
2024-02-21 21:04:46,699 [ERROR]: __main__.py(_run_parallel:934) >> lai failed to complete
2024-02-21 21:04:46,699_699:ERROR:_run_parallel:lai failed to complete
2024-02-21 21:04:46,699 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-02-21 21:04:46,699 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-02-21 21:04:46,699_699:ERROR:_run_parallel:0 of 1 handlers complete
'LAISUN'
mv: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_test_debug_output/testing-updates/v2.LR.historical_0201/post/lnd/180x360_aave/cmip_ts/monthly/tmp_ts_land_monthly_1850-1851-0002/CMIP6/CMIP/*/*/*/*/*/*/*/*/*.nc': No such file or directory

Run 2

No changes, just re-running:

tail -n 20 ts_land_monthly_1850-1851-0002.o474784
2024-02-21 21:21:30,550 [INFO]: handler.py(cmorize:247) >> lai: creating CMOR variable with CMOR axis objects.
2024-02-21 21:21:30,550_550:INFO:cmorize:lai: creating CMOR variable with CMOR axis objects.
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/e3sm_to_cmip/__main__.py", line 912, in _run_parallel
    out = res.result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
100%|██████████| 1/1 [00:01<00:00,  1.07s/it]
2024-02-21 21:21:31,139 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-02-21 21:21:31,139 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-02-21 21:21:31,139_139:INFO:_run_parallel:0 of 1 handlers complete
2024-02-21 21:21:31,140 [ERROR]: __main__.py(_run_parallel:934) >> lai failed to complete
2024-02-21 21:21:31,140 [ERROR]: __main__.py(_run_parallel:934) >> lai failed to complete
2024-02-21 21:21:31,140_140:ERROR:_run_parallel:lai failed to complete
2024-02-21 21:21:31,140 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-02-21 21:21:31,140 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-02-21 21:21:31,140_140:ERROR:_run_parallel:0 of 1 handlers complete
'LAISUN'
mv: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_test_debug_output/testing-updates/v2.LR.historical_0201/post/lnd/180x360_aave/cmip_ts/monthly/tmp_ts_land_monthly_1850-1851-0002/CMIP6/CMIP/*/*/*/*/*/*/*/*/*.nc': No such file or directory

That didn't seem to work this time.

Run 3

Running with sbatch directly rather than using zppy.

tail -n 20 ts_land_monthly_1850-1851-0002.o474785 
2024-02-21 21:26:12,481 [INFO]: handler.py(cmorize:247) >> lai: creating CMOR variable with CMOR axis objects.
2024-02-21 21:26:12,481_481:INFO:cmorize:lai: creating CMOR variable with CMOR axis objects.
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/e3sm_to_cmip/__main__.py", line 912, in _run_parallel
    out = res.result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
100%|██████████| 1/1 [00:00<00:00,  1.53it/s]
2024-02-21 21:26:12,790 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-02-21 21:26:12,790 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-02-21 21:26:12,790_790:INFO:_run_parallel:0 of 1 handlers complete
2024-02-21 21:26:12,790 [ERROR]: __main__.py(_run_parallel:934) >> lai failed to complete
2024-02-21 21:26:12,790 [ERROR]: __main__.py(_run_parallel:934) >> lai failed to complete
2024-02-21 21:26:12,790_790:ERROR:_run_parallel:lai failed to complete
2024-02-21 21:26:12,790 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-02-21 21:26:12,790 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-02-21 21:26:12,790_790:ERROR:_run_parallel:0 of 1 handlers complete
'LAISHA'
mv: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_test_debug_output/testing-updates/v2.LR.historical_0201/post/lnd/180x360_aave/cmip_ts/monthly/tmp_ts_land_monthly_1850-1851-0002/CMIP6/CMIP/*/*/*/*/*/*/*/*/*.nc': No such file or directory

That also didn't seem to work this time.

Run 4

Changed vars = "LAISHA,LAISUN" to vars = "FSH,RH2M"

tail -n 20 ts_land_monthly_1850-1851-0002.o474788
2024-02-21 21:31:25,123 [INFO]: handler.py(cmorize:247) >> lai: creating CMOR variable with CMOR axis objects.
2024-02-21 21:31:25,123_123:INFO:cmorize:lai: creating CMOR variable with CMOR axis objects.
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/e3sm_to_cmip/__main__.py", line 912, in _run_parallel
    out = res.result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
100%|██████████| 1/1 [00:01<00:00,  1.14s/it]
2024-02-21 21:31:25,623 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-02-21 21:31:25,623 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-02-21 21:31:25,623_623:INFO:_run_parallel:0 of 1 handlers complete
2024-02-21 21:31:25,623 [ERROR]: __main__.py(_run_parallel:934) >> lai failed to complete
2024-02-21 21:31:25,623 [ERROR]: __main__.py(_run_parallel:934) >> lai failed to complete
2024-02-21 21:31:25,623_623:ERROR:_run_parallel:lai failed to complete
2024-02-21 21:31:25,623 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-02-21 21:31:25,623 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-02-21 21:31:25,623_623:ERROR:_run_parallel:0 of 1 handlers complete
'LAISUN'
mv: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_test_debug_output/testing-updates/v2.LR.historical_0201/post/lnd/180x360_aave/cmip_ts/monthly/tmp_ts_land_monthly_1850-1851-0002/CMIP6/CMIP/*/*/*/*/*/*/*/*/*.nc': No such file or directory

Why is is still reporting errors as if vars = "LAISHA,LAISUN"?

[forsyth2]

Run 5

Running vars = "FSH,RH2M" setting with sbatch

tail -n 20 ts_land_monthly_1850-1851-0002.o474802
2024-02-21 21:44:55,359 [INFO]: handler.py(cmorize:247) >> lai: creating CMOR variable with CMOR axis objects.
2024-02-21 21:44:55,359_359:INFO:cmorize:lai: creating CMOR variable with CMOR axis objects.
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/site-packages/e3sm_to_cmip/__main__.py", line 912, in _run_parallel
    out = res.result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/lcrc/soft/climate/e3sm-unified/base/envs/e3sm_unified_1.9.2_chrysalis/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
100%|██████████| 1/1 [00:00<00:00,  1.92it/s]
2024-02-21 21:44:55,563 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-02-21 21:44:55,563 [INFO]: __main__.py(_run_parallel:930) >> 0 of 1 handlers complete
2024-02-21 21:44:55,563_563:INFO:_run_parallel:0 of 1 handlers complete
2024-02-21 21:44:55,564 [ERROR]: __main__.py(_run_parallel:934) >> lai failed to complete
2024-02-21 21:44:55,564 [ERROR]: __main__.py(_run_parallel:934) >> lai failed to complete
2024-02-21 21:44:55,564_564:ERROR:_run_parallel:lai failed to complete
2024-02-21 21:44:55,564 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-02-21 21:44:55,564 [ERROR]: __main__.py(_run_parallel:935) >> 0 of 1 handlers complete
2024-02-21 21:44:55,564_564:ERROR:_run_parallel:0 of 1 handlers complete
'LAISUN'
mv: cannot stat '/lcrc/group/e3sm/ac.forsyth2/zppy_test_debug_output/testing-updates/v2.LR.historical_0201/post/lnd/180x360_aave/cmip_ts/monthly/tmp_ts_land_monthly_1850-1851-0002/CMIP6/CMIP/*/*/*/*/*/*/*/*/*.nc': No such file or directory

Same as previous.

Run 6

Changed unique_id to testing-updates2

tail -n 20 ts_land_monthly_1850-1851-0002.o474807 
2024-02-21 21:49:20,360_360:INFO:__init__:    * realm='lnd'
2024-02-21 21:49:20,360 [INFO]: __main__.py(__init__:151) >>     * Writing log output file to: logs/20240221_214920_355514
2024-02-21 21:49:20,360 [INFO]: __main__.py(__init__:151) >>     * Writing log output file to: logs/20240221_214920_355514
2024-02-21 21:49:20,360_360:INFO:__init__:    * Writing log output file to: logs/20240221_214920_355514
2024-02-21 21:49:32,277 [WARNING]: utils.py(derive_handlers:218) >> No handlers could be derived for the variables: ['mrsos', 'mrso', 'mrfso', 'mrros', 'mrro', 'prveg', 'evspsblveg', 'evspsblsoi', 'tran', 'tsl', 'lai', 'cLitter', 'cProduct', 'cSoilFast', 'cSoilMedium', 'cSoilSlow', 'fFire', 'fHarvest', 'cVeg', 'nbp', 'gpp', 'ra', 'rh']. Make sure the input E3SM datasets have the variables needed derivation.
2024-02-21 21:49:32,277 [WARNING]: utils.py(derive_handlers:218) >> No handlers could be derived for the variables: ['mrsos', 'mrso', 'mrfso', 'mrros', 'mrro', 'prveg', 'evspsblveg', 'evspsblsoi', 'tran', 'tsl', 'lai', 'cLitter', 'cProduct', 'cSoilFast', 'cSoilMedium', 'cSoilSlow', 'fFire', 'fHarvest', 'cVeg', 'nbp', 'gpp', 'ra', 'rh']. Make sure the input E3SM datasets have the variables needed derivation.
2024-02-21 21:49:32,277_277:WARNING:derive_handlers:No handlers could be derived for the variables: ['mrsos', 'mrso', 'mrfso', 'mrros', 'mrro', 'prveg', 'evspsblveg', 'evspsblsoi', 'tran', 'tsl', 'lai', 'cLitter', 'cProduct', 'cSoilFast', 'cSoilMedium', 'cSoilSlow', 'fFire', 'fHarvest', 'cVeg', 'nbp', 'gpp', 'ra', 'rh']. Make sure the input E3SM datasets have the variables needed derivation.
2024-02-21 21:49:32,278 [INFO]: __main__.py(_get_handlers:220) >> --------------------------------------
2024-02-21 21:49:32,278 [INFO]: __main__.py(_get_handlers:220) >> --------------------------------------
2024-02-21 21:49:32,278_278:INFO:_get_handlers:--------------------------------------
2024-02-21 21:49:32,278 [INFO]: __main__.py(_get_handlers:221) >> | Derived CMIP6 Variable Handlers
2024-02-21 21:49:32,278 [INFO]: __main__.py(_get_handlers:221) >> | Derived CMIP6 Variable Handlers
2024-02-21 21:49:32,278_278:INFO:_get_handlers:| Derived CMIP6 Variable Handlers
2024-02-21 21:49:32,278 [INFO]: __main__.py(_get_handlers:222) >> --------------------------------------
2024-02-21 21:49:32,278 [INFO]: __main__.py(_get_handlers:222) >> --------------------------------------
2024-02-21 21:49:32,278_278:INFO:_get_handlers:--------------------------------------
2024-02-21 21:49:32,278 [ERROR]: __main__.py(_get_handlers:230) >> No CMIP6 variable handlers were derived from the variables found in using the E3SM input datasets.
2024-02-21 21:49:32,278 [ERROR]: __main__.py(_get_handlers:230) >> No CMIP6 variable handlers were derived from the variables found in using the E3SM input datasets.
2024-02-21 21:49:32,278_278:ERROR:_get_handlers:No CMIP6 variable handlers were derived from the variables found in using the E3SM input datasets.
srun: error: chr-0510: task 0: Exited with exit code 1

This still causes an error, but not one about vars = "LAISHA,LAISUN", so there must be a state leak from having run vars = "LAISHA,LAISUN" before changing it to vars = "FSH,RH2M"

[forsyth2]

The PR as-is resolves #540 and #530, but not #549. I'm going to merge this so we can at least get theses fixes in