Refactor land-ice mesh generation test cases #590

trhille · 2023-04-07T19:47:05Z

Move build_cell_width() to compass.landice.mesh and create new build_MALI_mesh() method to greatly reduce code redundancy between all mesh-generating test cases.

Also add make_region_masks() to compass.landice.mesh and add region masks for Greenland.

Checklist

User's Guide has been updated
Developer's Guide has been updated
API documentation in the Developer's Guide (api.rst) has any new or modified class, method and/or functions listed
Documentation has been built locally and changes look as expected
Document (in a comment titled Testing in this PR) any testing that was used to verify the changes

trhille · 2023-04-07T23:32:49Z

To do:

standardize mesh test-case and config section names
move _make_region_masks() to compass.landice.mesh
add region masks to greenland mesh case
compare meshes generated with this branch with previous versions
add new framework functions to Users Guide
add new framework functions to Developers Guide

trhille · 2023-04-11T15:35:12Z

Testing

I have visually confirmed in Paraview that cell locations are the same when created by this branch and on the current head of origin/main (a4720fb) for the following cases:

antarctica (an issue with islands not being culled as of c0333cf (14afb41 after force-push) is fixed by 5584ea8 (2141f32 after force-push))
greenland
humboldt
kangerlussuaq
koge_bugt_s
thwaites

Furthermore, all tests pass on this branch when using the default config files, while some tests fail on origin/main due to missing config options (which is my fault due to changes in #448 that were not propagated through all test cases).

trhille · 2023-04-11T17:26:20Z

Note: region masks only extend to the ice margin, which will be problematic in the future. This should eventually be dealt with, but is not within the scope of this PR:

Move build_cell_width() to compass.landice.mesh to greatly reduce code redundancy between all mesh-generating test cases.

Move most jigsaw and mpas-tools operations to compass.landice.mesh.build_MALI_mesh() to greatly reduce redundant code between mesh generation test cases.

Standardize mesh generation test case and config section names, using mesh_gen for the test case and 'mesh' for the config section. All .cfg files pertaining only to mesh generation have been moved into the mesh_gen modules.

Update landice api in developers guide to include renamed methods and classes, and new functions in landice.mesh.

Fix inconsistent usage of Humboldt.nc and Humboldt_1to10km.nc that caused an error at the end of the test case.

Add make_region_masks() function to landice.mesh, and add this capability for Greenland as well as Antarctica.

Fix netcdf format and engine definitions, which for some reason need to be called as mpas_tools.io.default_engine rather than using: from mpas_tools.io import default_engine. The latter approach returned a None type variable, which threw an error when passed to compute_mpas_region_masks.

Fix path to gridded dataset to use for building MALI mesh, in order to properly remove disconnected islands from the mesh.

Add compass.landice.mesh to framework documentation, and update some doc-strings.

Fix a few typos that only became apparent after building docs.

trhille · 2023-04-11T19:58:49Z

@hollyhan , @xylar , and @matthewhoffman, I often feel that requesting three reviews is overkill, but I think it is probably justified in this case, since it is a pretty major refactor. However, if any of you disagree, please let me know!

trhille · 2023-04-11T20:00:43Z

@hollyhan and @xylar, this should hopefully address the problems you flagged in #571. Let me know if there is anything else needed to close that.

trhille · 2023-04-11T20:02:31Z

And @xylar, I don't understand why the lint with pre-commit CI check is failing, so any guidance you can provide on fixing that would be much appreciated.

This should reportedly fix: ``` Error: There was an issue sorting changed files from Github. Exception: { "error": "500/TypeError", "from": "sortChangedFiles", "message": "There was an issue sorting files changed.", "payload": "{}" } ``` See: trilom/file-changes-action#104

xylar · 2023-04-11T20:18:43Z

And @xylar, I don't understand why the lint with pre-commit CI check is failing, so any guidance you can provide on fixing that would be much appreciated.

According to trilom/file-changes-action#104 (comment), the file-change-action needs to be update. This feels like deja vu but I couldn't figure out where I've done this elsewhere. Anyway, I'll try to make sure we update this action wherever we're using it.

trhille · 2023-04-11T20:20:24Z

According to trilom/file-changes-action#104 (comment), the file-change-action needs to be update. This feels like deja vu but I couldn't figure out where I've done this elsewhere. Anyway, I'll try to make sure we update this action wherever we're using it.

Thanks for taking care of that!

xylar

@trhille, this is amazing work! I really love to see redundancy reduced like this. It clearly show the power of compass over its predecessor and should make it a lot easier for future developers to make changes while also hopefully making things easier to maintain.

I do have some suggested changes, all in the realm of cosmetic improvements to the code, formatting changes to the docs, and clarification of the docs and docstrings. So nothing big just a lot of small things.

I didn't do any testing but am happy to if others don't have it covered.

compass/landice/mesh.py

docs/developers_guide/landice/api.rst

docs/developers_guide/landice/framework.rst

docs/users_guide/landice/framework.rst

Change function names to all lowercase in compass.landice.mesh.

Remove extraneous information from Users Guide. Add explanation of defining x/y bounds in example config file.

Clean up doc-strings and add a list of required config options for each function in compass/landice/mesh.py that requires section_name as a parameter. Refer to the Users and Developers guides for more information on the meaning of the config options.

Add a few more required config parameters to doc-strings.

Allow the user to define the bounds around a regional domain with any combination of floats and 'None'. For example, a config file that contained ``x_min = 'None'; x_max = 1.0e6; y_min = -1.0e6; y_max = 'None'`` would use you left and top boundaries of the gridded dataset, but also the user-defined values for the right and bottom boundaries.

Remove redundant use of ``flood_mask == 0`` Co-authored-by: Xylar Asay-Davis <xylarstorm@gmail.com>

Remove unnecessary calls to logger.info(), as check_call already takes care of writing the commands to the log file.

trhille · 2023-04-11T23:19:53Z

@xylar, thanks for the rapid turnaround with a very thorough review! I've addressed and responded to each of your requested changes. I've also re-run the mesh generation cases for Kangerlussaq and Koge_Bugt_S to ensure that I didn't break something in the process.

xylar

@trhille, thanks for making the changes I requested! I'm going to leave testing to you, @matthewhoffman and @hollyhan.

hollyhan · 2023-04-17T03:47:31Z

Hi @trhille,

I tested the koge_bugt_s test case on Chicoma with the dev_compass_1.2.0-alpha.5 version of the compass - it started all fine but ran into the following error soon in the step of culling to geojson file:

.
.
.

Running: MpasMaskCreator.x grid_preCull.nc mask.nc -f Koge_Bugt_S.geojson


************************************************************
MPAS_MASK_CREATOR:
  C++ version
  Creates a set of masks for a given MPAS mesh and a set of feature files. 


  Compiled on Mar 21 2023 at 22:57:31.
************************************************************



Using -180 to 180 degrees for longitude range.


Reading input grid.
Read dimensions:
    nCells = 17921
    nVertices = 36236
    nEdges = 54156
    maxEdges = 8
    vertexDegree = 3
    Spherical? = 0

 -- WARNING: This tool only works correctly with real latitude / longitude values for cell centers.
             If these are incorrect, you'll likely see odd behavior.
Building feature information.
Building 'all' feature groups.
Building polygon edge lines
Reading cell center locations
Built 17921 cells.
Marking cells based on region definitions
Marking points based on cell centers
 Building closest point location for line segments
 Reading cell graph
 Building cell transects
Deleting cell center information
Reading vertex locations
Built 36236 vertices.
Marking vertices based on region definitions
Marking points based on vertices
 Building closest vertex location for line segments
 Reading vertex graph
 Building vertex transects
Deleting vertex information
Reading edge locations
Built 54156 edges.
 Building closest edge location for line segments
 Reading edge graph
 Building edge transects
 Building edge transect path signs
Deleting edge information
Writing mask dimensions
Writing mask attributes
Writing mask fields



culling to geojson file


************************************************************
MPAS_CELL_CULLER:
  C++ version
  Remove cells/edges/vertices from a NetCDF MPAS Mesh file. 


  Compiled on Mar 21 2023 at 22:57:26.
************************************************************


Reading input grid.
Read dimensions:
    nCells = 17921
    nVertices = 36236
    nEdges = 54156
    vertexDegree = 3
    maxEdges = 8
    Spherical? = 0
    Periodic? = 0
Reading in mask information.
Marking cells for removal.
Removing 17921 cells.
Marking vertices for removal.
Removing 36236 vertices.
Marking edges for removal.
Removing 54156 edges.
Writing grid dimensions
NetCDF: NC_UNLIMITED size already in use
NetCDF: NC_UNLIMITED size already in use
NetCDF: NC_UNLIMITED size already in use
Writing grid attributes
Writing grid coordinates
NetCDF: Invalid dimension ID or name
NetCDF: Invalid dimension ID or name

Traceback (most recent call last):
  File "/usr/projects/climate/hollyhan/mambaforge/envs/dev_compass_1.2.0-alpha.5/bin/compass", line 33, in <module>
    sys.exit(load_entry_point('compass', 'console_scripts', 'compass')())
  File "/usr/projects/climate/hollyhan/compass_test_mesh_refactor/compass/__main__.py", line 63, in main
    commands[args.command]()
  File "/usr/projects/climate/hollyhan/compass_test_mesh_refactor/compass/run/serial.py", line 206, in main
    run_single_step(args.step_is_subprocess)
  File "/usr/projects/climate/hollyhan/compass_test_mesh_refactor/compass/run/serial.py", line 167, in run_single_step
    _run_test(test_case, available_resources)
  File "/usr/projects/climate/hollyhan/compass_test_mesh_refactor/compass/run/serial.py", line 417, in _run_test
    _run_step(test_case, step, test_case.new_step_log_file,
  File "/usr/projects/climate/hollyhan/compass_test_mesh_refactor/compass/run/serial.py", line 468, in _run_step
    step.run()
  File "/usr/projects/climate/hollyhan/compass_test_mesh_refactor/compass/landice/tests/koge_bugt_s/mesh.py", line 60, in run
    build_mali_mesh(
  File "/usr/projects/climate/hollyhan/compass_test_mesh_refactor/compass/landice/mesh.py", line 669, in build_mali_mesh
    dsMesh = cull(dsMesh, dsInverse=mask, logger=logger)
  File "/usr/projects/climate/hollyhan/mambaforge/envs/dev_compass_1.2.0-alpha.5/lib/python3.10/site-packages/mpas_tools/mesh/convers((d((d((

I'm happy to test it again, so let me know :)

trhille · 2023-04-17T14:11:47Z

@hollyhan , thanks for testing! I recognize this error, but in the past I've seen it when the geojson file is formatted incorrectly, which shouldn't be the case here. I'll look into it further.

trhille · 2023-04-17T14:26:12Z

@hollyhan, I'm unable to reproduce that error on Chicoma using dev_compass_1.2.0-alpha.5 @matthewhoffman, can you try running the koge_bugt_s/mesh_gen case and see if you run into the same issue that Holly did?

matthewhoffman · 2023-04-17T21:21:21Z

I was able to successfully run the koge_bugt_s/mesh_gen case on Chicoma using the branch as is, and creating a new compass env through the standard procedure.

trhille · 2023-04-17T21:32:35Z

@matthewhoffman, thanks! I also tried rebasing onto the current head of compass/main and recreating my compass env, and koge_bugt_s passed no problem.

matthewhoffman · 2023-04-17T21:40:34Z

compass/landice/mesh.py

+
+    Parameters
+    ----------
+    section_name : str


@trhille , I'm curious about the choice of having this as an argument. It seems in all the test cases using this function, they all use the section mesh. Can you imagine a use case where we wouldn't want to use that section name? If build_cell_width always used the mesh section, this argument could be removed. (And same for build_mali_mesh.)

Yeah, I thought about hard-coding it too. But I can imagine a case in which we want to produce multiple meshes by running a single test case, for example in a mesh convergence study, in which case we might have multiple sections called "mesh_1km", "mesh_2km", "mesh_4km" or similar. So, even though that case would require a bit of extra editing of that particular case's mesh.py, I think leaving it like this allows for more flexibility with minimal required changes in the future.

Wouldn't it be better, then, to not have the framework take config options but instead take parameters? That way, the parameters could be computed algorithmically based on a more limited number of config options. Having a bunch of kind of redundant config sections isn't really consistent with the philosophy of compass.

Wouldn't that just be trading redundancy in the config file for redundancy in the code?

No, you would have config options that are independent of resolution and a bit of code that can compute the associated resolution-dependent parameters. This is what we do in the cosine_bell test case, which lets a user supply a list of resolutions they would like to test:
https://github.com/MPAS-Dev/compass/blob/main/compass/ocean/tests/global_convergence/cosine_bell/cosine_bell.cfg#L26-L33
https://github.com/MPAS-Dev/compass/blob/main/compass/ocean/tests/global_convergence/cosine_bell/forward.py#L89-L112

The implementation is even a little cleaner in polaris.

Same kind of idea in polaris' version of the baroclinic channel:
https://github.com/E3SM-Project/polaris/blob/main/polaris/ocean/tests/baroclinic_channel/baroclinic_channel.cfg#L25-L39
https://github.com/E3SM-Project/polaris/blob/main/polaris/ocean/tests/baroclinic_channel/baroclinic_channel_test_case.py#L58-L64
https://github.com/E3SM-Project/polaris/blob/main/polaris/ocean/tests/baroclinic_channel/forward.py#L138-L154

But I don't think you need to change anything in this PR for now, it's just if you find you want to make meshes at a bunch of resolutions from one test case, we should talk about whether there are ways to not hard-code specific resolutions but rather to support any resolution in a reasonable range. We've found that saves quite a bit of time and adds a lot of flexibility if you decide you want a new resolution (or range of resolutions).

After talking with both of you, my preference would be to leave this as it is for now. The amount of work to modify it seems not worth the trouble for a slight preference. If/when we implement convergence tests we may want to make other adjustments, so it would be more efficient to wait to make any adjustments until then.

Yes, that was my suggestion as well.

matthewhoffman

@trhille , thanks for all the work on simplifying the mesh gen code. I recognize this was not a trivial amount of work, but I expect the reduction in complexity will pay off in the long run.

trhille changed the title ~~Move build_cell_width() to compass.landice.mesh~~ Move build_cell_width() to compass.landice.mesh Apr 7, 2023

trhille changed the title ~~Move build_cell_width() to compass.landice.mesh~~ Move build_cell_width() to compass.landice.mesh Apr 7, 2023

trhille added land ice clean-up in progress This PR is not ready for review or merging labels Apr 7, 2023

trhille changed the title ~~Move build_cell_width() to compass.landice.mesh~~ Refactor land-ice mesh generation test cases Apr 7, 2023

trhille added 10 commits April 11, 2023 11:34

Move build_cell_width() to compass.landice.mesh

2adc99b

Move build_cell_width() to compass.landice.mesh to greatly reduce code redundancy between all mesh-generating test cases.

Add parameters and returns to doc-string for build_cell_width()

83f27c9

Move most jigsaw and mpas-tools operations to compass.landice.mesh

73e37e7

Move most jigsaw and mpas-tools operations to compass.landice.mesh.build_MALI_mesh() to greatly reduce redundant code between mesh generation test cases.

Update landice api in developers guide

b69554d

Update landice api in developers guide to include renamed methods and classes, and new functions in landice.mesh.

Fix output file name in humboldt mesh_gen test case

4532644

Fix inconsistent usage of Humboldt.nc and Humboldt_1to10km.nc that caused an error at the end of the test case.

Add make_region_masks() function to landice.mesh.

dbf1b87

Add make_region_masks() function to landice.mesh, and add this capability for Greenland as well as Antarctica.

Fix culling of Antarctic islands

2141f32

Fix path to gridded dataset to use for building MALI mesh, in order to properly remove disconnected islands from the mesh.

Update documentation for compass.landice.mesh

1817dbb

Add compass.landice.mesh to framework documentation, and update some doc-strings.

trhille force-pushed the landice/mesh_test_cases_refactor branch from 5584ea8 to 1817dbb Compare April 11, 2023 18:45

Fix a few typos after building docs

4898a44

Fix a few typos that only became apparent after building docs.

trhille removed the in progress This PR is not ready for review or merging label Apr 11, 2023

trhille requested review from hollyhan, matthewhoffman and xylar April 11, 2023 19:56

xylar requested changes Apr 11, 2023

View reviewed changes

trhille and others added 7 commits April 11, 2023 15:39

Change function names to all lowercase

5543fa9

Change function names to all lowercase in compass.landice.mesh.

Update Users and Developers guides after code review

d2ba3d7

Remove extraneous information from Users Guide. Add explanation of defining x/y bounds in example config file.

Add a few more required config parameters to doc-strings.

95af2ff

Add a few more required config parameters to doc-strings.

Small clean-up from code review

8b327bd

Remove redundant use of ``flood_mask == 0`` Co-authored-by: Xylar Asay-Davis <xylarstorm@gmail.com>

Remove unnecessary calls to logger.info()

342856f

Remove unnecessary calls to logger.info(), as check_call already takes care of writing the commands to the log file.

trhille requested a review from xylar April 12, 2023 19:15

xylar approved these changes Apr 12, 2023

View reviewed changes

xylar assigned trhille Apr 13, 2023

trhille mentioned this pull request Apr 14, 2023

Add isunnguata sermia mesh generation test case #602

Merged

5 tasks

matthewhoffman reviewed Apr 17, 2023

View reviewed changes

matthewhoffman approved these changes Apr 18, 2023

View reviewed changes

matthewhoffman merged commit 51e65a5 into MPAS-Dev:main Apr 18, 2023

trhille mentioned this pull request Nov 13, 2023

Major bug in landice/antarctica/mesh_gen #729

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor land-ice mesh generation test cases #590

Refactor land-ice mesh generation test cases #590

trhille commented Apr 7, 2023 •

edited

Loading

trhille commented Apr 7, 2023 •

edited

Loading

trhille commented Apr 11, 2023 •

edited

Loading

trhille commented Apr 11, 2023

trhille commented Apr 11, 2023

trhille commented Apr 11, 2023

trhille commented Apr 11, 2023

xylar commented Apr 11, 2023

trhille commented Apr 11, 2023

xylar left a comment

trhille commented Apr 11, 2023

xylar left a comment

hollyhan commented Apr 17, 2023

trhille commented Apr 17, 2023 •

edited

Loading

trhille commented Apr 17, 2023

matthewhoffman commented Apr 17, 2023

trhille commented Apr 17, 2023

matthewhoffman Apr 17, 2023

trhille Apr 17, 2023 •

edited

Loading

xylar Apr 17, 2023

trhille Apr 17, 2023

xylar Apr 17, 2023

xylar Apr 17, 2023

xylar Apr 18, 2023

matthewhoffman Apr 18, 2023

xylar Apr 18, 2023

matthewhoffman left a comment

Refactor land-ice mesh generation test cases #590

Refactor land-ice mesh generation test cases #590

Conversation

trhille commented Apr 7, 2023 • edited Loading

trhille commented Apr 7, 2023 • edited Loading

trhille commented Apr 11, 2023 • edited Loading

Testing

trhille commented Apr 11, 2023

trhille commented Apr 11, 2023

trhille commented Apr 11, 2023

trhille commented Apr 11, 2023

xylar commented Apr 11, 2023

trhille commented Apr 11, 2023

xylar left a comment

Choose a reason for hiding this comment

trhille commented Apr 11, 2023

xylar left a comment

Choose a reason for hiding this comment

hollyhan commented Apr 17, 2023

trhille commented Apr 17, 2023 • edited Loading

trhille commented Apr 17, 2023

matthewhoffman commented Apr 17, 2023

trhille commented Apr 17, 2023

Choose a reason for hiding this comment

trhille Apr 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthewhoffman left a comment

Choose a reason for hiding this comment

trhille commented Apr 7, 2023 •

edited

Loading

trhille commented Apr 7, 2023 •

edited

Loading

trhille commented Apr 11, 2023 •

edited

Loading

trhille commented Apr 17, 2023 •

edited

Loading

trhille Apr 17, 2023 •

edited

Loading