-
Notifications
You must be signed in to change notification settings - Fork 365
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update CIME to 5.3.0alpha34 #1788
Conversation
Allow --compiler and --mpilib in ctest scripts This PR allows ctest to use the new --compiler and --mpilib options in scripts_regression_tests.py by using env variables CIME_COMPILER and CIME_MPILIB Test suite: CIME_COMPILER=nag ./cime.ctest.hobart Test baseline: Test namelist changes: Test status: bit for bit Fixes #1770 User interface changes?: Update gh-pages html (Y/N)?: Code review: Bill Sacks
Inspired by ACME's gnu debug flags
Inspired by ACME's gnu debug flags Test suite: SMS_D.f09_g16.X.cheyenne_gnu Also ran all of CESM's gnu tests on an earlier version of this, which had some other changes folded in (which I haven't kept) Test baseline: N/A Test namelist changes: none Test status: bit for bit Fixes ESMCI/cime#1621 User interface changes?: none Update gh-pages html (Y/N)?: N Code review:
Allow running infnan unit tests even in DEBUG mode Up until now, we have been skipping the shr_infnan unit tests when building in DEBUG mode. Since we pretty much always build the unit tests in DEBUG mode, that means that we're pretty much never running the shr_infnan unit tests. In trying (and failing) to solve #1763, I came up with a way to get the shr_infnan unit tests to run and pass even when building in debug mode. That is implemented here. Test suite: ./scripts_regression_tests.py N_TestUnitTest on cheyenne Also, cime fortran unit tests on my mac (with gnu) Test baseline: N/A Test namelist changes: none Test status: bit for bit Fixes ESMCI/cime#1809 User interface changes?: no Update gh-pages html (Y/N)?: N Code review:jedwards
…nges Merge configuration changes from ACME to CIME This just copies the latest configuration files from ACME to CIME to fix Skybridge configuration changes and meet the new xmllint requirements Test suite: scripts_regression_tests Test baseline: Test namelist changes: Test status: BFB User interface changes?: N Update gh-pages html (Y/N)?: N Code review: @jedwards4b
Cray was not a valid edison compiler for ACME. Gnu should work for both ACME and CESM. Test machines must define SAVE_TIMING_DIR so that saving timings can be tested.
Get melvin scripts_regression_tests working again. Cray was not a valid edison compiler for ACME. Gnu should work for both ACME and CESM. Test machines must define SAVE_TIMING_DIR so that saving timings can be tested. Test suite: scripts_regression_tests Test baseline: Test namelist changes: Test status: bit for bit Fixes [CIME Github issue #] User interface changes?: None Update gh-pages html (Y/N)?: N Code review: None
The prior implementation was a bit too simple. Always giving the user 10 percent of nodes as spare nodes was way overkill for large jobs. The new implementation maxes out the number of spare nodes at 10. We also add a new variable to allow the user to pick the exact number of spare nodes that they want.
Without this change, skybridge was trying to run this on a single node: mpiexec --np 16 ppr=5 ... With the PPR lowered to 5 due to the ROOTPE changes, but the --np not adjusted, the node could not accomodate this request. After this change, it runs: mpiexec --np 9 ppr=5 ...
Fix SEQ test on skybridge. Without this change, skybridge was trying to run this on a single node: mpiexec --np 16 ppr=5 ... With the PPR lowered to 5 due to the ROOTPE changes, but the --np not adjusted, the node could not accomodate this request. After this change, it runs: mpiexec --np 9 ppr=5 ... Test suite: ./create_test SEQ_Ln9.f19_g16_rx1.A Test baseline: Test namelist changes: Test status: bit for bit Fixes [CIME Github issue #] User interface changes?: N Update gh-pages html (Y/N)?: Code review: @jedwards4b
Refactor how spare nodes get computed. The prior implementation was a bit too simple. Always giving the user 10 percent of nodes as spare nodes was way overkill for large jobs. The new implementation maxes out the number of spare nodes at 10. We also add a new variable to allow the user to pick the exact number of spare nodes that they want. Test suite: scripts_regression_tests T_TestRunRestart Test baseline: Test namelist changes: Test status: [bit for bit, roundoff, climate changing] Fixes [CIME Github issue #] User interface changes?: Yes, new case variables controlling spare nodes Update gh-pages html (Y/N)?: N Code review: @jedwards4b
ACME users were complaining of unhelpful error messages coming from xmlchange. This PR will make the handling of type_str consistent with the listofsettings block above. This should ensure a better error message.
seq_comm_mct code cleanup Refactored to get rid of a lot of repeated code. This will help with the multi_driver branch. Also remove some un-needed esmf code. Test suite: cesm2 prealpha tests on cheyenne Test baseline: cesm2_0_alpha07c Test namelist changes: Test status: bit for bit Fixes User interface changes?: Update gh-pages html (Y/N)?: Code review: RobJ, SteveG.
Improve error messages from xmlchange ACME users were complaining of unhelpful error messages coming from xmlchange. This PR will make the handling of type_str consistent with the listofsettings block above. This should ensure a better error message. Test suite: code_checker, by-hand Test baseline: Test namelist changes: Test status: bit for bit Fixes [CIME Github issue #] User interface changes?: Update gh-pages html (Y/N)?: Code review: jedwards
Test suite: scripts_regression_tests.py, PRE.f09_f09_ADESP_TEST.hobart_nag, PRE.f19_f19.ADESP_TEST.hobart_nag Test baseline: Test namelist changes: Test status: bit for bit Fixes #1801 User interface changes?: Update gh-pages html (Y/N)?: Code review:
Remove mask specification for test using DOCN%SOMAQP. The mg16 and mg17 grid has a hole over Greenland. This was causing Nans when running SOM aquaplanet at f09_f09_mg17. The fix is not to specify a mask when using DOCN%SOMAQP. Test suite: scripts_regression_tests.py, PRE.f09_f09_ADESP_TEST.hobart_nag, PRE.f19_f19.ADESP_TEST.hobart_nag Test baseline: Test namelist changes: Test status: bit for bit Fixes #1801 User interface changes?: Update gh-pages html (Y/N)?: Code review: jedwards
remove untested/unused scripts in Tools directory These scripts in the Tools directory appear to be unused and are untested. Test suite: scripts_regression_tests.py Test baseline: Test namelist changes: Test status: bit for bit Fixes [CIME Github issue #] User interface changes?: Update gh-pages html (Y/N)?: Code review:mvertens
New user interface features: option --skip-preview-namelist to case.submit to skip calling preview_namelist during case.run option -M MAIL_USER, --mail-user MAIL_USER to case.submit to set an email for batch system notification option -m {never,all,begin,end,fail}, --mail-type {never,all,begin,end,fail} to case.submit for when the batch system should send email. manage_case renamed to more accurate query_config and usage/arguments made easier and output more informative. Add --pesfile option to create_test New case.qstatus tool Allow user to adjust ulimits Add message at end of successful build Add new user-mods option to create_clone New ways to select queue in config_batch.xml scripts_regression_tests.py now has --compiler and --mpilibs option new case variables controlling spare nodes (for failing nodes workaround) User interface bug fixes: xmlquery options --file and --listall now work together. xmlquery options clarified and help message improved. Improve error message for passing incorrect grid alias to create_newcase. Improve error messages from xmlchange Removed user features: Remove long term archiving support. "case.lt_archive" no longer in caseroot. Other changes to case control system: Improve ERR test. New ERS2 test Add IRT test Implement NCR with SystemTestsCompareTwo Rewrite ERP with SystemTestsCompareTwo Convert the ERR and ERRI test to the compare2 framework Fix SEQ test on skybridge new case.test --reset option to manually reset a test to initial conditions. TOTAL_CORES was inaccurate so remove it. Only create rest directories if they are going to be populated get_timing.py should use largest ncpl value, not assume it's the atm value Improve comment for BASELINE status in TestStatus Better handling of fails in the build (don't print a python stacktrace) Add a new directory testmod that creates cpl history files on a daily basis and thereby permits answer changes to be compared more easily. Add ability for create_test to infer non-default machine from testname Change all python2 style strings to python3 style strings Improve handling of hist comparisons in TestStatus.log version 3.0 schema for entry_id files Downscale build parallelism on small machines (so we don't overload it) Bring bless_test_results behavior into an exact match with create_test -g lcdf4 flag does apply to pnetcdf. Replace cdf64 logical with PIO_NETCDF_FORMAT character variable Refactor how spare nodes get computed Preserve test order in test_scheduler Allow queue selection by node count instead of task count Improve error check for non-python sub processes Data model changes: Refactored data models Use %CPLHIST for DATM coupler hist forcing Add support for data model to read multiple time slices at once New aquaplanet capability: adds aquaplanet capability to DOCN where the input is from a file rather than analytic; adds new A compsets to the driver config_compsets.xml that have different DOCN functionality: ADSOM - DOCN SOM, ADSOMAQP - DOCN aquaplanet SOM, ADAQP3 - DOCN analytic aquaplanet (mode 3), ADAQFILE - DOCN aquaplanet from file DATM updates for clm Change SIAF compset name in dice to IAF to be consistent with other components Coupler/driver changes: Map SMB from lnd to glc using bilinear mapping with a conservation correction Remove more cesm refs in filenames. seq_comm_mct code cleanup Implementation of multiple couplers within multi-instance runs. Other: User's guide rst files now in cime/doc directory Get PFUnit working on melvin Homebrew generic mac port Fixes #1255 Fixes #1424 Fixes #1426 Fixes #1573 Fixes #1576 Fixes #1585 * jgfouca/update_to_cime_5.3.0.34: (932 commits) One more short term archiver fix Fix merge conflicts, should mostly work cetus xml fix Hack to get PET test to work on chama Fix HOMME Cmake pointing to wrong cime area Update run_acme version Fixes for AQUAP Removes long term archiving from run_acme Move xml compiler and mpilib tags from env and command elements This is required for the new xml schema requirements; compiler and mpilib tags should be in the modules and environment_variables elements rather than their members Cherry-pick fixes from older merge branch cime_developer now working on melvin new git pelayout doc Update ChangeLog Change atm_grid "48x96" to T31 to match the model_grid_alias that is actually being used. Minor fix Fix mistake Better encapsulation of common calls (create_test, wait_for_tests) Add a timeout option to wait_for_tests update to sphinx documentation Restore task_count ...
Merge #2 for this PR. * jgfouca/update_to_cime_5.3.0.34: short term archiver get_datenames call for last restarts fixed
Merge #3 for this PR. * jgfouca/update_to_cime_5.3.0.34: Fix jenkins script mistake made during conflict resolution.
Merge #4 for this PR. * jgfouca/update_to_cime_5.3.0.34: Fix jenkins script mistake made during conflict resolution.
Many namelist diffs on next. @jayeshkrishna what do you think of the PIO ones? cpl_modelio.nml' found extra variable: 'pio_netcdf_format' |
ERP test has these diffs in drv_in |
The dlimit in drof_in is to "Specifies delta time ratio limits placed on the time interpolation |
The PIO namelist changes (pio_netcdf_format) should be fine. A new feature was added in CIME to allow users to switch between CDF1 (classic), CDF2 (64 bit offset) and CDF5 (64 bit data) per component via env_run.xml . |
Reverted off of next by force pushing. We'll try again next week. |
State of next saved in branch /rljacob/nextcime530-34merge which has been pushed to github. |
…ime_5.3.0.34 Upstream merge for consistent conflict resolution. * origin/master: (32 commits) Rename TOOL_DIR to MPAS_TOOL_DIR Reload env for tests with multiple case-dirs Add option to use pre-built MPAS tools Update Theta modules Update Depends.intel file Add a default 64x1 PE layout Updates surface datasets Minor update to the flag for IMCLM tests add a flag for the reading of soil order so it only applies for ECA and RD mode Add a comment with an example of MV2_CPU_MAPPING Update Anvil threading environment - fix parameter dimension in PDynamicsMod.F90 Change BGC mode in RD compsts from CN to BGC Do not use CENTURY initial condition fille for RD and CN compsets when using nitrif_dentrif Reverts changes to CLMBuildNamelist.pm Adds a clm-rd test Fix for GSWP3 version in CIME Fix issues found during review. Add GSWP and coupler bypass compsets Update mpas-cice hash and supporting build scripts ...
Update CIME to 5.3.0alpha34 New user interface features: option --skip-preview-namelist to case.submit to skip calling preview_namelist during case.run option -M MAIL_USER, --mail-user MAIL_USER to case.submit to set an email for batch system notification option -m {never,all,begin,end,fail}, --mail-type {never,all,begin,end,fail} to case.submit for when the batch system should send email. manage_case renamed to more accurate query_config and usage/arguments made easier and output more informative. Add --pesfile option to create_test New case.qstatus tool Allow user to adjust ulimits Add message at end of successful build Add new user-mods option to create_clone New ways to select queue in config_batch.xml scripts_regression_tests.py now has --compiler and --mpilibs option new case variables controlling spare nodes (for failing nodes workaround) User interface bug fixes: xmlquery options --file and --listall now work together. xmlquery options clarified and help message improved. Improve error message for passing incorrect grid alias to create_newcase. Improve error messages from xmlchange Removed user features: Remove long term archiving support. "case.lt_archive" no longer in caseroot. Other changes to case control system: Improve ERR test. New ERS2 test Add IRT test Implement NCR with SystemTestsCompareTwo Rewrite ERP with SystemTestsCompareTwo Convert the ERR and ERRI test to the compare2 framework Fix SEQ test on skybridge new case.test --reset option to manually reset a test to initial conditions. TOTAL_CORES was inaccurate so remove it. Only create rest directories if they are going to be populated get_timing.py should use largest ncpl value, not assume it's the atm value Improve comment for BASELINE status in TestStatus Better handling of fails in the build (don't print a python stacktrace) Add a new directory testmod that creates cpl history files on a daily basis and thereby permits answer changes to be compared more easily. Add ability for create_test to infer non-default machine from testname Change all python2 style strings to python3 style strings Improve handling of hist comparisons in TestStatus.log version 3.0 schema for entry_id files Downscale build parallelism on small machines (so we don't overload it) Bring bless_test_results behavior into an exact match with create_test -g lcdf4 flag does apply to pnetcdf. Replace cdf64 logical with PIO_NETCDF_FORMAT character variable Refactor how spare nodes get computed Preserve test order in test_scheduler Allow queue selection by node count instead of task count Improve error check for non-python sub processes Data model changes: Refactored data models Use %CPLHIST for DATM coupler hist forcing Add support for data model to read multiple time slices at once New aquaplanet capability: adds aquaplanet capability to DOCN where the input is from a file rather than analytic; adds new A compsets to the driver config_compsets.xml that have different DOCN functionality: ADSOM - DOCN SOM, ADSOMAQP - DOCN aquaplanet SOM, ADAQP3 - DOCN analytic aquaplanet (mode 3), ADAQFILE - DOCN aquaplanet from file DATM updates for clm Change SIAF compset name in dice to IAF to be consistent with other components Coupler/driver changes: Map SMB from lnd to glc using bilinear mapping with a conservation correction Remove more cesm refs in filenames. seq_comm_mct code cleanup Implementation of multiple couplers within multi-instance runs. Other: User's guide rst files now in cime/doc directory Get PFUnit working on melvin Homebrew generic mac port Fixes #1255 Fixes #1424 Fixes #1426 Fixes #1573 Fixes #1576 Fixes #1585 * jgfouca/update_to_cime_5.3.0.34: (942 commits) Apply patch for cpl to avoid hangs Adjust ne4 PE layout on Anvil Reset configpes_all to what's on master Cherry-pick wait_for_tests fix from ESMCI Fix xmllint errors on Titan Convert attribute DEBUG to uppercase Update ne4-wcycl PEs on Anvil Fix the rest of the pylint errors for case_st_archive Fix jenkins script mistake made during conflict resolution. short term archiver get_datenames call for last restarts fixed One more short term archiver fix Fix merge conflicts, should mostly work cetus xml fix Hack to get PET test to work on chama Fix HOMME Cmake pointing to wrong cime area Update run_acme version Fixes for AQUAP Removes long term archiving from run_acme Move xml compiler and mpilib tags from env and command elements This is required for the new xml schema requirements; compiler and mpilib tags should be in the modules and environment_variables elements rather than their members Cherry-pick fixes from older merge branch ...
Update CIME to 5.3.0alpha34 New user interface features: option --skip-preview-namelist to case.submit to skip calling preview_namelist during case.run option -M MAIL_USER, --mail-user MAIL_USER to case.submit to set an email for batch system notification option -m {never,all,begin,end,fail}, --mail-type {never,all,begin,end,fail} to case.submit for when the batch system should send email. manage_case renamed to more accurate query_config and usage/arguments made easier and output more informative. Add --pesfile option to create_test New case.qstatus tool Allow user to adjust ulimits Add message at end of successful build Add new user-mods option to create_clone New ways to select queue in config_batch.xml scripts_regression_tests.py now has --compiler and --mpilibs option new case variables controlling spare nodes (for failing nodes workaround) User interface bug fixes: xmlquery options --file and --listall now work together. xmlquery options clarified and help message improved. Improve error message for passing incorrect grid alias to create_newcase. Improve error messages from xmlchange Removed user features: Remove long term archiving support. "case.lt_archive" no longer in caseroot. Other changes to case control system: Improve ERR test. New ERS2 test Add IRT test Implement NCR with SystemTestsCompareTwo Rewrite ERP with SystemTestsCompareTwo Convert the ERR and ERRI test to the compare2 framework Fix SEQ test on skybridge new case.test --reset option to manually reset a test to initial conditions. TOTAL_CORES was inaccurate so remove it. Only create rest directories if they are going to be populated get_timing.py should use largest ncpl value, not assume it's the atm value Improve comment for BASELINE status in TestStatus Better handling of fails in the build (don't print a python stacktrace) Add a new directory testmod that creates cpl history files on a daily basis and thereby permits answer changes to be compared more easily. Add ability for create_test to infer non-default machine from testname Change all python2 style strings to python3 style strings Improve handling of hist comparisons in TestStatus.log version 3.0 schema for entry_id files Downscale build parallelism on small machines (so we don't overload it) Bring bless_test_results behavior into an exact match with create_test -g lcdf4 flag does apply to pnetcdf. Replace cdf64 logical with PIO_NETCDF_FORMAT character variable Refactor how spare nodes get computed Preserve test order in test_scheduler Allow queue selection by node count instead of task count Improve error check for non-python sub processes Data model changes: Refactored data models Use %CPLHIST for DATM coupler hist forcing Add support for data model to read multiple time slices at once New aquaplanet capability: adds aquaplanet capability to DOCN where the input is from a file rather than analytic; adds new A compsets to the driver config_compsets.xml that have different DOCN functionality: ADSOM - DOCN SOM, ADSOMAQP - DOCN aquaplanet SOM, ADAQP3 - DOCN analytic aquaplanet (mode 3), ADAQFILE - DOCN aquaplanet from file DATM updates for clm Change SIAF compset name in dice to IAF to be consistent with other components Coupler/driver changes: Map SMB from lnd to glc using bilinear mapping with a conservation correction Remove more cesm refs in filenames. seq_comm_mct code cleanup Implementation of multiple couplers within multi-instance runs. Other: User's guide rst files now in cime/doc directory Get PFUnit working on melvin Homebrew generic mac port Fixes #1255 Fixes #1424 Fixes #1426 Fixes #1573 Fixes #1576 Fixes #1585 * jgfouca/update_to_cime_5.3.0.34: (942 commits) Apply patch for cpl to avoid hangs Adjust ne4 PE layout on Anvil Reset configpes_all to what's on master Cherry-pick wait_for_tests fix from ESMCI Fix xmllint errors on Titan Convert attribute DEBUG to uppercase Update ne4-wcycl PEs on Anvil Fix the rest of the pylint errors for case_st_archive Fix jenkins script mistake made during conflict resolution. short term archiver get_datenames call for last restarts fixed One more short term archiver fix Fix merge conflicts, should mostly work cetus xml fix Hack to get PET test to work on chama Fix HOMME Cmake pointing to wrong cime area Update run_acme version Fixes for AQUAP Removes long term archiving from run_acme Move xml compiler and mpilib tags from env and command elements This is required for the new xml schema requirements; compiler and mpilib tags should be in the modules and environment_variables elements rather than their members Cherry-pick fixes from older merge branch ...
Summary and discussion of this PR is at https://acme-climate.atlassian.net/wiki/spaces/SE/pages/160039354/CIME5.3+upgrade+alpha.10+to+alpha.34 |
New user interface features:
option
--skip-preview-namelist
to case.submit to skip calling preview_namelist during case.runoption
-M MAIL_USER, --mail-user MAIL_USER
to case.submit to set an email for batch system notificationoption
-m {never,all,begin,end,fail}, --mail-type {never,all,begin,end,fail}
to case.submit for when the batch system should send email.manage_case renamed to more accurate query_config and usage/arguments made easier and output more informative.
Add --pesfile option to create_test
New case.qstatus tool
Allow user to adjust ulimits
Add message at end of successful build
Add new user-mods option to create_clone
New ways to select queue in config_batch.xml
scripts_regression_tests.py now has --compiler and --mpilibs option
new case variables controlling spare nodes (for failing nodes workaround)
User interface bug fixes:
xmlquery options --file and --listall now work together.
xmlquery options clarified and help message improved.
Improve error message for passing incorrect grid alias to create_newcase.
Improve error messages from xmlchange
Removed user features:
Remove long term archiving support. "case.lt_archive" no longer in caseroot.
Other changes to case control system:
Improve ERR test.
New ERS2 test
Add IRT test
Implement NCR with SystemTestsCompareTwo
Rewrite ERP with SystemTestsCompareTwo
Convert the ERR and ERRI test to the compare2 framework
Fix SEQ test on skybridge
new case.test --reset option to manually reset a test to initial conditions.
TOTAL_CORES was inaccurate so remove it.
Only create rest directories if they are going to be populated
get_timing.py should use largest ncpl value, not assume it's the atm value
Improve comment for BASELINE status in TestStatus
Better handling of fails in the build (don't print a python stacktrace)
Add a new directory testmod that creates cpl history files on a daily basis and thereby permits answer changes to be compared more easily.
Add ability for create_test to infer non-default machine from testname
Change all python2 style strings to python3 style strings
Improve handling of hist comparisons in TestStatus.log
version 3.0 schema for entry_id files
Downscale build parallelism on small machines (so we don't overload it)
Bring bless_test_results behavior into an exact match with create_test -g
lcdf4 flag does apply to pnetcdf.
Replace cdf64 logical with PIO_NETCDF_FORMAT character variable
Refactor how spare nodes get computed
Preserve test order in test_scheduler
Allow queue selection by node count instead of task count
Improve error check for non-python sub processes
Data model changes:
Refactored data models
Use %CPLHIST for DATM coupler hist forcing
Add support for data model to read multiple time slices at once
New aquaplanet capability: adds aquaplanet capability to DOCN where the input is from a file rather than analytic; adds new A compsets to the driver config_compsets.xml that have different DOCN functionality: ADSOM - DOCN SOM, ADSOMAQP - DOCN aquaplanet SOM, ADAQP3 - DOCN analytic aquaplanet (mode 3), ADAQFILE - DOCN aquaplanet from file
DATM updates for clm
Change SIAF compset name in dice to IAF to be consistent with other components
Coupler/driver changes:
Map SMB from lnd to glc using bilinear mapping with a conservation correction
Remove more cesm refs in filenames.
seq_comm_mct code cleanup
Implementation of multiple couplers within multi-instance runs.
Other:
User's guide rst files now in cime/doc directory
Get PFUnit working on melvin
Homebrew generic mac port
Fixes #1255
Fixes #1424
Fixes #1426
Fixes #1573
Fixes #1576
Fixes #1585
[non-BFB] only for coupled ice sheet cases.