WW3 identical mesh issue in coupled applications

[uturuncoglu]

Describe the bug
As a part of the work related with the UFS Coastal, we found possible bug in ESMF Mesh generation part of the WW3 preprocessing tool (specifically ww3_grid) and also in the coupling layer. We have a regression test under UFS Coastal to test SCHSIM+WW3 coupling and we were assuming both SCHSIM and WW3 models are using same (identical) mesh but it turns out that WW3 preprocessing tool is creating different mesh then SCHSIM even if we provide same mesh definition (via its ordinary test format) in both side. We have only triangular mesh elements in the input but output mesh from WW3 has higher order elements, extra elements and also node -> element center conversion (dual mesh). This leads to unmatched grids in WW3 and SCHSIM side. The following figure is showing part of the mesh used in the UFS Coastal RT (in this case red is SCHSIM mesh and white is WW3 mesh),

As it can be seen from the figure, WW3 mesh has higher order elements and also having dual mesh kind of feature.

In the current workflow, the SCRIP grid definition file is created via ww3_grid tool and we are using ESMF offline tool to create ESMF Mesh file. This is fine but WW3 SCRIP netcdf generation part was doing something special related with the mesh and it was ending up a different mesh (only for unstructured case) than what is defined in the text mesh input file. The result mesh is not completely different but it was switching nodes with elements (dual case) as well as some high order elements.

Following line of code basically shows that the dual mesh approach is hard coded (fixed to use DUALAPPROACH all the time) -

WW3/model/src/wmscrpmd.F90

Line 547 in abc77b9

SUBROUTINE GET_SCRIP_INFO_UNSTRUCTURED (ID_GRD, &

and as a result the SCRIP grid definition file is different from the input grid definition. The possible fix is to make this part of the code more flexible by adding an option to give users a flexibility between regular mesh or dual approach but for testing purpose, I did following,

• I set THEAPPROACH variable to 1 manually
• Then I fixed the first if statement by adding following code at the end (because it was missing and producing error),

      GRID_CORNERS=3

      ALLOCATE(GRID_DIMS(2), STAT=ISTAT)
      CHECK_ALLOC_STATUS ( ISTAT )
      GRID_DIMS(1)=MNE
      GRID_DIMS(2)=1
      GRID_SIZE=MNE

This basically allowed me to create exactly same mesh given in input file and I confirmed that the mesh is exactly same with the SCHSIM if I create the SCRIP grid definition with this fix (of course this SCRIP file converted to ESMF Mesh file as usual procedure).

But then I try this new mesh with the application (DATM+SCHSIM+WW3 test case) and it did not run correctly and NUOPC cap is tried to remove some part of my mesh (see following figure, the middle part of the mesh is removed by the NUOPC cap but it is fine if I use ESMF Mesh file to create same visualization).

So, it seems that it is normal since NUOPC cap assumes that mesh file has ghost elements (it is not true in our case) and removing those ghost elements to be compatible with the CMEPS mediator (we also found that CMEPS mediator and ESMF interpolation confuses when ghost elements are found in the mesh). So, probably we need some fix also in the cap around following regions,

WW3/model/src/wav_comp_nuopc.F90

Line 747 in c7004b6

#ifdef W3_PDLIB

and also maybe in the model itself

WW3/model/src/w3parall.F90

Line 1489 in abc77b9

#ifdef W3_PDLIB

but there could be also other places that assumes the given mesh has ghost elements. We also discussed the issue with @DeniseWorthen and @aliabdolali and we could not come up a solution since we don't know why the model itself and the cap is constructed with this way. It seems that preprocessing part of the code is written by @MathieuDutSik but not sure.

@DeniseWorthen also tried to use the fix that I did in the ww3_grid part and use the newly generated mesh with the global configuration in addition to the fix that removes the ghost points but it seems that the cap is not working correctly and producing wrong interpolation output. I am also putting her results as screen shots in here. So, the first plot is for original code that uses dual mesh way, and everything seems fine,

the second one is using original mesh (not the dual approach),

and it seems that we have issue around polar regions (tile 4 seems fine in this case and some regions in other tiles).

Anyway, I think there was no any intention in the UFS WM side to have identical mesh in ocean and wave side and maybe this part of code is not well tested and checked. But since we are trying to use identical meshes under UFS coastal regional applications, it is important to have this capability and fix the underlying issues.

To Reproduce
Just apply fix in ww3_grid tool and try to run the global s2s configuration with new WW3 mesh.

Expected behavior
I am expecting to have working WW3 configuration without using hardcoded dual mesh option.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

[uturuncoglu]

The only think that I need to test the new mesh without using PDLIB to see if that solved the issue in the cap and result interpolation. I'll update you soon about it.

[JessicaMeixner-NOAA]

@uturuncoglu thanks for creating this issue. One thing I noticed in your first image is that the white (ww3) looks to put points in the center of the red (schism) grid. I'm wondering if this is intentional? I need to go back and read, but I think knowing what those center points is used within the numerics of WW3 with the unstructured grid. I'll have to dig into the issue more to try to understand the issues, but it certainly looks like we have problems we need to solve. Thanks for pointing this out!!

[uturuncoglu]

@JessicaMeixner-NOAA Yes. You are right. This is because WW3 tool is creating dual mesh with other extra elements. I am not sure those are used internally by the WW3 but again the mesh that is created in here is different from the given as an input. Maybe it is good to use it internally but it is also exposed to NUOPC cap. I am not sure this is intentional or not but this is just a finding from UFS Coastal side. Maybe there is no way to fix it and we need to live with it but at least we need to know why this approach is used in WW3. Anyway, let me know if you want to discuss more. Thanks for your help.