New Mesh: WC12to60kmL60E3SMv2r01

[mark-petersen]

Design discussion Water Cycle mesh, 12km high resolution region.

[mark-petersen]

Instructions to create CUSP mesh on LANL IC. See general MPAS instructions for other machines and configurations.

1. Clone this branch

2. Create the test case:

source /usr/projects/climate/SHARED_CLIMATE/anaconda_envs/load_latest_compass.sh
cd testing_and_setup/compass
./list_testcases.py | grep CUSP
  65: -o ocean -c global_ocean -r CUSP12 -t init
  66: -o ocean -c global_ocean -r CUSP12 -t spin_up
./setup_testcase.py \
  --config_file general.config.ocean \
  --work_dir $WORKDIR \
  --model_runtime runtime_definitions/mpirun.xml \
  -o ocean -c global_ocean -r CUSP12 -t init

where $WORKDIR is a path, usually to your scratch space. For example,
--work_dir /lustre/scratch4/turquoise/mpeterse/runs/200429_test_new_branch
and general.config.ocean is a file that specifies directory and file paths. Here is an example for LANL IC: general.config.ocean

3. Create mesh

On LANL IC, you can do this interactively:

salloc -N 1 -t 4:0:0 --qos=interactive
source /usr/projects/climate/SHARED_CLIMATE/anaconda_envs/load_latest_compass.sh
cd $WORKDIR
cd ocean/global_ocean/CUSP12/init
./run.py

That will proceed through the base_mesh and culled_mesh steps. It creates these files:

base_mesh: base_mesh.nc cellWidthGlobal.png   mesh_construction.png
culled_mesh:  culled_mesh.nc

You should look at the png files, and can also explore the netcdf files.

4. Create the initial state

You need a compiled version of MPAS for this, as it uses the MPAS init mode. See machine specific instructions. Then:

salloc -N 1 -t 4:0:0 --qos=interactive
source /usr/projects/climate/SHARED_CLIMATE/anaconda_envs/load_latest_compass.sh
cd $WORKDIR
cd ocean/global_ocean/CUSP12/init/initial_state
./run.py

where there is a link to your compiled ocean_model there, and you can increase the core count in run.py if needed. You will then see:

initial_state:
initial_state.nc initial_state.png vertical_grid.png

and you should look at those files as well.

5. spin up

This is the same as step 2 above, but use the spin-up:

./setup_testcase.py \
  --config_file general.config.ocean \
  --work_dir $WORKDIR \
  --model_runtime runtime_definitions/mpirun.xml \
  -o ocean -c global_ocean -r CUSP12 -t spin_up

6. E3SM coupling files

source /usr/projects/climate/SHARED_CLIMATE/anaconda_envs/load_latest_compass.sh
cd $WORKDIR
cd ocean/global_ocean/CUSP12/init/e3sm_coupling/
./run.py

[mark-petersen]

Based on the conversations the last few months, I made a best guess at the CUSP mesh using the signed distance function (PR #523). I created the mesh today with this PR and compass 0.1.5. It produces this mesh:

and these auxiliary images:

mesh construction steps

vertical grid

initial state pdfs

[mark-petersen]

Here are a few questions for discussion for this mesh design:

• 1. What should the minimum resolution be? Currently 12km
  That is set here. I created these:
  12km min: 485k cells.
  14km min: 405k cells.
  For reference, EC60to30 has 265k cells.
  On IC, meshes are available at:

/lustre/scratch4/turquoise/mpeterse/runs/200511_cusp14_cusp_coastal_US_plus_mesh/ocean/global_ocean/CUSP14/init
/lustre/scratch4/turquoise/mpeterse/runs/200511_cusp12_cusp_coastal_US_plus_mesh/ocean/global_ocean/CUSP12/init

• 2. How many vertical levels? Currently 60 and 64 layers are generated in COMPASS.

• 3. Should the high-resolution region be revised? In particular,
  a. Coastal width is 600km, set here
  b. North Atlantic transition is set here with the file region_Gulf_Stream_extension.geojson, and can be easily changed.

• 4. Minor items
  a. Black and Capain seas are still there.
  b. At -180, high-res. region slightly degraded near Russia.

[xylar]

@mark-petersen, your instructions involve e3sm_coupling before spin_up but it needs to be the opposite. Though I wouldn't worry too much until after Thursday's meeting since we will likely need to do some reorganization anyway following that discussion.

[xylar]

4. Minor items
   Black and Caspian Seas are still there.

Are you sure this is still true once you make the mesh? There should be a critical land blockage to remove the Black Sea and the Caspian sea has no connection to the global ocean and should be culled by the normal flood-fill procedure. If you are going on the image above, it's just a cartopy map and has no relevance for the MPAS mesh.

At -180, high-res. region slightly degraded near Russia.

It looks like the region_Bering_Sea feature isn't periodic at +/-180:
https://github.com/mark-petersen/MPAS-Model/blob/ocean/cusp_coastal_US_plus_mesh/testing_and_setup/compass/ocean/global_ocean/CUSP12/init/region_Bering_Sea.geojson
For the boundary at +/-180 to be handled correctly, you need to have exactly the same points at exactly +180 and -180. It is not valid (at least for the purposes of the signed distance function) to have points above +180 or below -180. We should add checks and error messages to this effect.

[pwolfram]

Following on from our discussion at https://acme-climate.atlassian.net/wiki/spaces/ECG/pages/1410270496/05-14-2020+Special+Topics+MPAS+E3SM+initialization, would recommend capturing physical basis or reasoning (several bullets, 30s-1min view) of design of mesh.

[xylar]

@pwolfram, I think this is at least one point where the info you requested on why the resolution is distributed where it is could go here:
https://github.com/mark-petersen/MPAS-Model/blob/ocean/cusp_coastal_US_plus_mesh/testing_and_setup/compass/ocean/global_ocean/CUSP12/init/config_E3SM_coupling_files.ini#L14-L16

This comes from the discussion here #507

[xylar]

capturing physical basis or reasoning (several bullets, 30s-1min view) of design of mesh.

Would you do that here in the PR? In the metadata I pointed to above? Somewhere else?

[pwolfram]

@xylar, I'd do that in the PR description and copy/paste these top points into the field you point to above.

[proteanplanet]

@xylar and @mark-petersen. I've been running the ruler over this mesh over the past few days, and am close to sharing a host of plots and checks on the mesh. As I finish off my work, I want to supply you with a file of Arctic Shipping Assessment ship tracks through the Arctic that you may or may not wish to include in forced-open channels. With very minor changes, the 14km mesh would accommodate this. What is your preferred format for that - Would a KML file work, or a Geoshape, or netcdf trace, or GeoJSON?

[proteanplanet]

Of course, with my new-found knowledge after the tutorial, I could even venture to do this myself.... with your supervision.

[xylar]

@proteanplanet, it would be great if you want to add some new critical passages and critical blockages! Eventually, I will add the following to future COMPASS documentation.

Adding new transects (critical blockages or passages) can be done through geometric_features but currently not directly in COMPASS in MPAS-Model. Adding new features to geometric_features not quite trivial but it shouldn't be too hard.

• First, go to http://geojson.io and make your transects. It's fine to have multiple transects in the same file for now
• Add the required properties (name, component='ocean', object='transect', author= and tags), see Geometric Data for details
• Make sure the list of tags includes Critical_Passage or Critical_Land_Blockage. Other tags are up to you. Tags are separated by semicolons
• Copy the geojson data on the right and paste it into a file in the root of the branch of geometric_features for your modification.
• Make a copy of the example script for Adding New Features, change my_new_features.geojson to the name of your new geojson file, and run the script.
• Commit the changes to the repo and make a pull request to geometric_features, which I will review.

These changes will make their way into the next geometric_features release and the next compass conda environment. So they're not immediately available for testing. If you want me to set up a test compass environment, though, it's not too hard and we can talk about that. Eventually, that process will also be in the documentation, too...

[xylar]

@maltrud, @katsmith133 and possibly others had mentioned not being big fans of the name "CUSP" for this mesh. I think this PR is a reasonable place to discuss alternatives. @katsmith133, what was your suggestion again?

[katsmith133]

@xylar not something that should actually be considered. It was very much a joke, but I will think about real names for the mesh and post any that I think are good.

[philipwjones]

CONUS-Coast (like Kona coast and with our own coffee blend), crush (coastal refined US...), lots of other cr- words work. What about Crusty? Also, cousin, Custer, couscous, cuscus, cuss, cussed, custody...just a quick brain dump of c-words...can fill in an acronym for any of those and inverting word order opens up more...

[pwolfram]

How about CONA (Coastal Ocean North America)?

[vanroekel]

Now that we resolve a good chunk of the N. Atlantic and all the arctic, Coastal no longer seems like the most important item to include. Not sure what the best choice is though.

[maltrud]

North America Shelf Coastal and Arctic Refined
though we might have some copyright issues with this....

[philipwjones]

Excellent! But I only watch it for the crashes...

[proteanplanet]

I am close to publishing quality control on CUSP12 and CUSP14 for this PR. I see on Confluence that WC12 and WC14 may be the new names for these meshes per Luke. If so, I very much like this convention - shorter and focused on the simulation campaign at which they are targeted. The rest of the information is now in the metadata.

[pwolfram]

Sounds good-- happy with nice concise meta data description at https://github.com/mark-petersen/MPAS-Model/blob/ocean/cusp_coastal_US_plus_mesh/testing_and_setup/compass/ocean/global_ocean/CUSP12/init/config_E3SM_coupling_files.ini#L14-L16 and name is good too.

[proteanplanet]

Quality control of mesh for global and polar performance: The first stage is to share the mesh characteristics for 42 designated areas of oceanographic or coupled importance, including a comparison of the coastline with the E3SM-HR V1 mesh. These will be shared in two separate posts following this one for WC12 and WC14, respectively. Further quality control posts will follow these two with comments on parts of the mesh that need to be addressed.

[proteanplanet]

WC12 sector plots - Indicating zoomed-in regions for the next post

[proteanplanet]

WC12 zoomed plots - Numbers correspond to the figure in the previous post

[xylar]

Adding the "Don't merge" tag until changes to #584 get brought in and metadata gets updated.

[proteanplanet]

@pwolfram and @proteanplanet, if we want to maintain two meshes with a lot of redundancy in the long run, it would be good to reorganize things so they share the shapes used to construct them to avoid the current redundancy (in both shapes and commits).

Thanks @xylar. The critical deciding factor will likely be the be timing statistics for WC14 for InteRFACE. @pwolfram and I will discuss this in the coming week.

[vanroekel]

@xylar I've rebased and tested this. I think it is ready to merge, but want to check with you as you have a comment above about metadata. That PR is merged but want to make sure you are good with this going in.

[xylar]

@vanroekel, this looks ready to merge as far as I can tell.

[xylar]

@mark-petersen, is the timing right for @vanroekel to merge this or should we wait for an e3sm/develop merge before that?

[mark-petersen]

Should this still be merged in, or was it superseded by the 14km version in #628 or others?

[xylar]

@mark-petersen, my feeling is that all revisions need to be in the commit history for ocean/develop even if we don't end up using them. If we close the PR and delete the branch, all info on this mesh is lost. @vanroekel, what is your feeling?

[mark-petersen]

I agree. Let's merge in, so we have both 14 and 12km options.