Switch global ocean mesh culling and files for E3SM to use python masks

[xylar]

This merge switches the global_ocean test group to using a set of threaded python tools for creating masks. These same tools are being used in MPAS-Analyisis and have been shown to perform better on larger meshes than the C++ equivalents.

The python version of the transect mask creator also has the advantage that it follows the transect through polygons on the sphere rather than zig-zagging (which has been a problem with MpasMaskCreator.x).

The results with the new tools are not expected to be bit-for-bit as compared with the previous mask creator. So the test suites should pass but not bit-for-bit for global ocean test cases.

See MPAS-Dev/MPAS-Tools#399

[xylar]

@mark-petersen and @vanroekel, I know I've been bombarding you with comapss pull requests. There has been a backlog as #28 was being reviewed. This one is perhaps more important than #76 but also perhaps more involved to review. It would be important to take a look at the meshes that result from the new masks and make sure there are no unexpected surprises, particularly with the critical passages and land blockages.

I will post results from my testing, too, as soon as I have finished.

[xylar]

Womp-womp-womp... It turns out this branch won't work on Anvil until I get #64 taken care of. That's a big project, too, so I'm closing this for now.

[xylar]

Testing

I will test QU240 through the file_for_e3sm test case on Grizzly, Badger, Anvil and Chrysalis with intel. I will test EC30to60, QUwISC240, ECwISC30to60 and SOwISC12to60 on Chrysalis with intel only.

Tests (checkmark means completed successfully):

• Grizzly intel QU240
• Badger intel QU240
• Anvil intel QU240
• Chrysalis intel QU240
• Chrysalis intel EC30to60
• Chrysalis intel QUwISC240
• Chrysalis intel ECwISC30to60
• Chrysalis intel SOwISC12to60

Timing

Here is a note of the times for different test cases on Chrysalis with intel and impi, for comparison with #85 (comment):

QU240 70% of previous time:

00:32 PASS ocean_global_ocean_QU240_mesh
00:25 PASS ocean_global_ocean_QU240_PHC_init
00:43 PASS ocean_global_ocean_QU240_PHC_dynamic_adjustment
00:58 PASS ocean_global_ocean_QU240_PHC_files_for_e3sm
Total runtime 02:37

QUwISC240 69% of previous time:

00:38 PASS ocean_global_ocean_QUwISC240_mesh
01:19 PASS ocean_global_ocean_QUwISC240_PHC_init
00:44 PASS ocean_global_ocean_QUwISC240_PHC_dynamic_adjustment
01:27 PASS ocean_global_ocean_QUwISC240_PHC_files_for_e3sm
Total runtime 04:07

EC30to60 40% of previous time (5x speedup in mesh step, 1.5x in file_for_e3sm):

10:48 PASS ocean_global_ocean_EC30to60_mesh
00:38 PASS ocean_global_ocean_EC30to60_PHC_init
00:34 PASS ocean_global_ocean_EC30to60_PHC_performance_test
11:33 PASS ocean_global_ocean_EC30to60_PHC_dynamic_adjustment
08:54 PASS ocean_global_ocean_EC30to60_PHC_files_for_e3sm
Total runtime 32:26

ECwISC30to60 38% of the previous time (6x speedup in mesh and 2x in files_for_e3sm):

11:10 PASS ocean_global_ocean_ECwISC30to60_mesh
04:24 PASS ocean_global_ocean_ECwISC30to60_PHC_init
00:39 PASS ocean_global_ocean_ECwISC30to60_PHC_performance_test
11:10 PASS ocean_global_ocean_ECwISC30to60_PHC_dynamic_adjustment
15:58 PASS ocean_global_ocean_ECwISC30to60_PHC_files_for_e3sm
Total runtime 43:20

SOwISC12to60 <36% of the previous time (5x speedup in mesh and >2x in files_for_e3sm):

31:14 PASS ocean_global_ocean_SOwISC12to60_mesh
05:12 PASS ocean_global_ocean_SOwISC12to60_PHC_init
00:51 PASS ocean_global_ocean_SOwISC12to60_PHC_performance_test
14:29 PASS ocean_global_ocean_SOwISC12to60_PHC_dynamic_adjustment
35:31 PASS ocean_global_ocean_SOwISC12to60_PHC_files_for_e3sm
Total runtime 87:15

[xylar]

@mark-petersen and @vanroekel, I was going to have you both review this PR but you are both busy with other stuff this week.

I would invite you to browse the previous results here (from #85):
/home/ac.xylar/e3sm/ac.xylar/compass_1.0/chrysalis/test_20210429/sys_e3sm_intel
and the new results here:
/home/ac.xylar/e3sm/ac.xylar/compass_1.0/chrysalis/test_20210429/python_masks
if you have time and let me know if you notice any concerning changes in any of the 5 meshes.

If you don't have time or interest in looking at these, I think it is reasonable to revisit this as part of creating the next generation of meshes from compass whenever that may happen.

[vanroekel]

@xylar not sure if it is most appropriate to comment here, but just wanted to say that I'm happy to take a look at the meshes (may not happen until later next week though), but think it will also be good to also revisit this when new meshes come around. It would also be great to get Andrew R's scripts available as a part of the standard workflow analyzing these meshes.

[xylar]

@vanroekel, fine to comment here.

Andrew's scripts are in Matlab as I understand it. The port would be a nontrivial amount of work but I agree with you.