Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update MALI-Dev submodule #333

Merged
merged 4 commits into from
Apr 14, 2022

Conversation

trhille
Copy link
Collaborator

@trhille trhille commented Mar 24, 2022

Update the MALI-Dev submodule. This brings in the merge of MALI-Dev/E3SM#18. This also adds updates for COMPASS tests to work with the current version of Albany supported by COMPASS.

@trhille trhille requested a review from xylar March 24, 2022 01:13
@trhille trhille added land ice in progress This PR is not ready for review or merging labels Mar 24, 2022
@trhille
Copy link
Collaborator Author

trhille commented Mar 24, 2022

Test execution failed for three tests:

00:11 PASS landice_dome_2000m_sia_restart_test
00:06 PASS landice_dome_2000m_sia_decomposition_test
00:10 PASS landice_dome_variable_resolution_sia_restart_test
00:05 PASS landice_dome_variable_resolution_sia_decomposition_test
01:07 PASS landice_enthalpy_benchmark_A
00:31 PASS landice_eismint2_decomposition_test
00:28 PASS landice_eismint2_enthalpy_decomposition_test
00:30 PASS landice_eismint2_restart_test
00:26 PASS landice_eismint2_enthalpy_restart_test
00:20 FAIL landice_greenland_sia_restart_test
00:22 PASS landice_greenland_sia_decomposition_test
00:11 PASS landice_hydro_radial_restart_test
00:07 PASS landice_hydro_radial_decomposition_test
00:20 PASS landice_dome_2000m_fo_decomposition_test
00:17 PASS landice_dome_2000m_fo_restart_test
00:15 PASS landice_dome_variable_resolution_fo_decomposition_test
00:14 PASS landice_dome_variable_resolution_fo_restart_test
00:17 PASS landice_circular_shelf_decomposition_test
00:58 PASS landice_greenland_fo_decomposition_test
00:42 FAIL landice_greenland_fo_restart_test
00:38 PASS landice_thwaites_decomposition_test
00:43 FAIL landice_thwaites_restart_test

I don't have any ideas for why those failed, but I can try to look into it tomorrow.

@xylar
Copy link
Collaborator

xylar commented Mar 24, 2022

@trhille, are any of those failures new? Or did that happen also with master?

@xylar xylar mentioned this pull request Mar 24, 2022
85 tasks
Copy link
Collaborator

@xylar xylar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested the sia_integration suite on Anvil with intel and impi. It ran successfully, using the current submodule as a baseline.

I also looked at the commits in the new submodule compare with the old and they do, indeed, bring in the one PR we are expecting to add.

@trhille trhille changed the title Update MALI-Dev submodule for compass 1.0.0 release Update MALI-Dev submodule Mar 25, 2022
@xylar xylar added the python package DEPRECATED: PRs and Issues involving the python package (master branch) label Mar 28, 2022
@xylar xylar requested a review from matthewhoffman March 28, 2022 09:34
@xylar xylar removed the python package DEPRECATED: PRs and Issues involving the python package (master branch) label Apr 1, 2022
@matthewhoffman
Copy link
Member

I'm picking up this PR to re-establish our COMPASS MALI baselines before testing & merging a backlog of PRs we have for MALI. Our MALI COMPASS baselines have been sort of out of date because we had been using a very old version of Albany for testing. With Xylar's recent COMPASS merge #353 now allowing us to more easily keep Albany up to date, we can finally re-establish our baselines. The feature this PR is bringing in to MALI is not on in any test cases, so I'm using this PR as way to identify changes due to the Albany update.

I ran our integration suite with our old Badger Albany (/users/mhoffman/setup_badger_mods.2021-03-09.sh) as baseline and this PR with Albany from COMPASS as the test. I got these results:

Test Runtimes:
00:07 PASS landice_dome_2000m_sia_restart_test
00:05 PASS landice_dome_2000m_sia_decomposition_test
00:07 PASS landice_dome_variable_resolution_sia_restart_test
00:04 PASS landice_dome_variable_resolution_sia_decomposition_test
00:30 PASS landice_enthalpy_benchmark_A
00:13 PASS landice_eismint2_decomposition_test
00:13 PASS landice_eismint2_enthalpy_decomposition_test
00:12 PASS landice_eismint2_restart_test
00:12 PASS landice_eismint2_enthalpy_restart_test
00:16 PASS landice_greenland_sia_restart_test
00:11 PASS landice_greenland_sia_decomposition_test
00:08 PASS landice_hydro_radial_restart_test
00:06 PASS landice_hydro_radial_decomposition_test
00:15 FAIL landice_dome_2000m_fo_decomposition_test
00:16 FAIL landice_dome_2000m_fo_restart_test
00:10 FAIL landice_dome_variable_resolution_fo_decomposition_test
00:12 FAIL landice_dome_variable_resolution_fo_restart_test
00:17 FAIL landice_circular_shelf_decomposition_test
00:31 FAIL landice_greenland_fo_decomposition_test
00:33 FAIL landice_greenland_fo_restart_test
00:32 FAIL landice_thwaites_decomposition_test
00:43 FAIL landice_thwaites_restart_test
Total runtime 05:55
FAIL: 9 tests failed, see above.

All failures are baseline comparison failures, except one. I confirmed all the changes are small for most tests (e.g. order 1e-12) and could be expected from the changes in Albany since early 2021 (including the b.c. refactoring that Mauro implemented). Changes were not small for the greenland/fo_decomposition_test and greenland/fo_restart_test, which had huge changes:

thickness            Time index: 0, 1, 2, 3, 4, 5
0:  l1: 8.48173398570486e+06  l2: 1.36043042066009e+05  linf: 3.30191824586221e+03
1:  l1: 8.48053441800451e+06  l2: 1.36021986565058e+05  linf: 3.30091625666703e+03
2:  l1: 8.48053349287920e+06  l2: 1.36021943711097e+05  linf: 3.30091369138991e+03
3:  l1: 8.48053256775385e+06  l2: 1.36021900858400e+05  linf: 3.30091112624279e+03
4:  l1: 8.48053164262845e+06  l2: 1.36021858006964e+05  linf: 3.30090856122562e+03
5:  l1: 8.48053071750301e+06  l2: 1.36021815156791e+05  linf: 3.30090599633839e+03

This is the validation failure:

landice/circular_shelf/decomposition_test
  * step: setup_mesh
  * step: 1proc_run
  * step: 4proc_run
  test execution:      SUCCESS
  test validation:     FAIL
  baseline comparison: FAIL

These three tests require further investigation to resolve. They likely will require adjustments to Albany b.c. settings or initial conditions. I will investigate further and report solutions or issues in additional comments.

trhille and others added 2 commits April 14, 2022 11:00
Updated to: a9cc37502113d45b97457fdc2232b80ccedfcda1
The greenland albany_input.yaml file was out of date and resulting in
garbage results.  Albany had changed units on flowA.  But we also had
added a realistic temperature field to the input file, so I've removed
the Problem section entirely so we use temp-dependent flow rate
calculation.  I added betaSolve to the output for convenience.
@matthewhoffman matthewhoffman force-pushed the update_MALI_submodule branch from 93313b7 to 725cf82 Compare April 14, 2022 17:27
@matthewhoffman matthewhoffman removed the in progress This PR is not ready for review or merging label Apr 14, 2022
Albany changed units of flowA.  Also, Albany now supports using the
flowA value provided by MPAS.  So I have removed the flowA specification
from the albany_input.yaml file and updated the value in the namelist to
be consistent with what we had been using before.
Another adjustment required after updating Albany and after switching to
a real temperature field.
@matthewhoffman matthewhoffman force-pushed the update_MALI_submodule branch from 725cf82 to a9b965b Compare April 14, 2022 17:42
@matthewhoffman
Copy link
Member

I rebased this branch so it is on top of the recent Spack merge into master to allow easier testing. I also pushed fixes for the greenland and circular_shelf tests that were failing. Both had to do with changes in Albany and a need to update the Albany yaml file (details in the new commit messages). I reran the full_integration test suite and we still get BASELINE failures for these tests (as expected for the same reason as for the other FO tests), but the diffs have the expected magnitude based on the changes in Albany. So with these new changes, I bless all diffs and declare our baselines re-established! I will merge after the CI tests complete.

@matthewhoffman matthewhoffman merged commit 149694d into MPAS-Dev:master Apr 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants