MITgcm/N-BLING with Compressed Staggered Grids #640

mgharamti · 2024-02-20T16:57:29Z

Description:

The MITgcm code now supports compressed grids, especially suited for areas like the Red Sea where land occupies more than 90% of the domain. Thanks to Ed Liu (SIParCS intern, 2022) and Helen Kershaw for developing this approach.

Fixes issue

The updated code allows writing the bgc fields into MITgcm's pickup files. It also allows for different compression of the regular and staggered grids. These changes have been tested at KAUST for the Arabian Gulf in cycling DA runs.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Documentation changes needed?

My change requires a change to the documentation.
- I have updated the documentation accordingly.

Tests

Please describe any tests you ran to verify your changes.

Checklist for merging

Updated changelog entry
Documentation updated
Update conf.py

Checklist for release

Merge into main
Create release from the main branch with appropriate tag
Delete feature-branch

Testing Datasets

Dataset needed for testing available upon request
Dataset download instructions included
No dataset needed

todo: use dart modules rather than standalone, netcdf, kinds, etc

mit_to_dart, dart_to_mit seems like place to put the offline squishing

FVAL a parameter

3d

used when uncompressing

integers for coord index

the caching was fixed in NCAR#368

CHL is not updated, but may be helpful when testing code. For example, perturbing uncompressed state to get an ensemble of mit .data files to test bitwise between compressed and non-compressed.

Changes made to trans_mit_dart: - trans_mit_dart now uses separate compression dimensions for U and V - T, SAL and other tracers still use original compression - Eta's compression is done through the T (or SAL) surface layer to avoid any zero pressure (height) issues - The tracers after DART are written into MITgcm's pickup files Changes to model_mod: - Getting compressed state index now directly uses comp_inds from the restart files

Suggested changes to the compressed code after testing for the Arabian Gulf. The code has been reported to work fine for several DA cycles. A small bug was fixed when parsing the vertical layers for the pickup files.

hkershaw-brown · 2024-03-01T21:40:06Z

Hi Moha, great work on this and thanks to the KAUST folks for testing it thoroughly. It is a really good improvement on the current code.
One question for you:
There is a program expand_netcdf which was used to uncompress the netcdf file into a non-compressed netcdf file - this was for MITgcm novices (like me) taking a look at the data with ncview while writing the compression code. It does not uncompress for the different grids, and so its not complete at the movement. Are you ok removing this (assuming it is not what regular DA user would need?)

I do want check on the netcdf_utilies_mod 64 bit offset change to see if we need to be careful/paranoid about breaking netcdf writes: reviewers see notes https://github.com/NCAR/DART/pull/455/files#r1101999108, #287

Then a couple of notes for whoever does the release: check with Ed how he wants his name, and here is his AMS presentation and siparcs page to add to the release notes.
From work by Jiachen Liu which he presented at AMS 2023
The work was done as part of SIParCS 2022 under the Red Sea Initiative project

hkershaw-brown

Setting this as 'request changes' purely for the netcdf file_creation 64_bit_offset change. Need to check (& prevent) unintended consequences for this.

hkershaw-brown · 2024-03-01T21:49:28Z

assimilation_code/programs/model_mod_check/model_mod_check.f90

   ! CLM has (potentially many) columns and needs i7 ish precision
-   write(string1,'(i11,1x,''i,j,k'',3(1x,i7),'' domain '',i2)') &
+!    write(string1,'(i11,1x,''i,j,k'',3(1x,i7),'' domain '',i2)') &
+!                   iloc, ix, iy, iz, dom_id
+   ! EL: integer to short for the new I/O method
+   ! Change to long int to avoid problems


we can tidy this comment i21 for long ints.

mgharamti · 2024-03-01T21:55:26Z

Helen, thanks for taking time to review this PR.
To your first question, I have no problem removing the expand_netcdf program. I understand the reason behind it yet I didn't run it myself. Siva, used it a couple of times before extending the compression capability but as you noted it's not needed for operating filter.
Good observation regarding the 64 bit netcdf thing, I am not sure if this causes any issues with other nc files. I guess I'm not experienced enough on this front to give a useful suggestion.
I fully support acknowledging Ed in the best possible way for this release.

models/MITgcm_ocean/expand_netcdf.f90

see pull NCAR#640 for discussion * does not have the u, v compression

hkershaw-brown

Approved!

@hkershaw-brown add Ed to release notes, edit:
the smaller pull requests are getting bundled with the wrf-tutorial fixes #636 #651 #637

…t of this pull request see NCAR#640 (comment) for discussion

Ed Liu and others added 30 commits June 6, 2022 15:33

Initial commit of the adaptive close_state_caching

d67b475

Comment out the close_state_cach = .false. in get_close_state_cached

874cf22

Add do_output for the print statements

0428283

New I/O scheme with offline processing of large input state

774b9cc

Merge branch 'main' into netcdf-reduce

5a8b812

moved dart_nc_reduce one directory up to compile

7d5867f

grid size is harded coded - change to small case

d376f80

todo: use dart modules rather than standalone, netcdf, kinds, etc

using dart netcdf utilites and types modules

869d89d

mirror of dart_nc_reduce. untested.

fbbb201

mit_to_dart, dart_to_mit seems like place to put the offline squishing

ZC is double - check dart_nc_reduce

b14007d

zc double dart_nc_expand

d9e2979

netcdf and model_mod_check for comparision with main

d197e63

Merge branch 'main' into netcdf-reduce

e9273c8

function for defining variables mit2dart

a409ff4

FVAL a parameter

function for dart to mit. untested.

3d0c053

bug-fix: returns need to be inside if statement

e007252

bitwise mit_to_dart non-compressed with main, log and nolog

fba76b4

bitwise with main dart_to_mit, no compression

951235c

size of compressed variables

60c742c

partway through compressed write, I think separate writes for 2d vs

8b556ba

3d

write compressed. untested. missing coord

b239419

note on delX,Y - does delX,Y vary?

1a7bf39

record indices for X,Y,Z

d4c3905

used when uncompressing

compressing out vals=0.0

39568db

integers for coord index

replace hardcoded 0.0_r8 with binary_fill variable

329cd42

somthing funky with ETA

6fbbdf0

bug-fix: 2D ETA variable is th k=1 slice

a7d60db

revert assim_tools_mod to main

0c2c781

the caching was fixed in NCAR#368

move initializing to fill outside read_compressed

80e22dc

Merge branch 'main' into netcdf-reduce

0a30429

hkershaw-brown and others added 14 commits October 11, 2022 09:47

Option to output CHL for dart_to_mit

cd107d1

CHL is not updated, but may be helpful when testing code. For example, perturbing uncompressed state to get an ensemble of mit .data files to test bitwise between compressed and non-compressed.

Merge branch 'main' into netcdf-reduce

2c0b275

Merge branch 'main' into netcdf-reduce

5fe6781

Merge branch 'main' into netcdf-reduce

630b5a1

one place to set recl3d recl2d

d048a05

Merge branch 'main' into netcdf-reduce

134570b

Merge branch 'main' into netcdf-reduce

f0da199

Merge branch 'main' into netcdf-reduce

de015c6

Merge branch 'main' into netcdf-reduce

a2c0d26

Merge branch 'main' into netcdf-reduce

41b35e3

Merge branch 'main' into netcdf-reduce

92142d8

Merge branch 'main' into netcdf-reduce

e0dd62b

Changes after cycling DA testing

e9001e2

Suggested changes to the compressed code after testing for the Arabian Gulf. The code has been reported to work fine for several DA cycles. A small bug was fixed when parsing the vertical layers for the pickup files.

mgharamti requested a review from hkershaw-brown February 20, 2024 16:57

Merge branch 'main' into MITgcm_comp

0febdfe

hkershaw-brown requested changes Mar 1, 2024

View reviewed changes

hkershaw-brown reviewed Mar 1, 2024

View reviewed changes

hkershaw-brown reviewed Mar 5, 2024

View reviewed changes

models/MITgcm_ocean/expand_netcdf.f90 Outdated Show resolved Hide resolved

hkershaw-brown and others added 2 commits March 11, 2024 09:00

remove expand netcdf program

ecfdb88

see pull NCAR#640 for discussion * does not have the u, v compression

Merge branch 'main' into MITgcm_comp

ce1fcb6

hkershaw-brown added the release! bundle with next release label Mar 11, 2024

hkershaw-brown approved these changes Mar 11, 2024

View reviewed changes

hkershaw-brown and others added 3 commits March 12, 2024 15:40

Merge branch 'main' into MITgcm_comp

058cdac

revert netcdf utilties to main, since expand_netcdf.f90 no longer par…

1fe14ea

…t of this pull request see NCAR#640 (comment) for discussion

bump conf.py and CHANGELOG for release

1044da3

hkershaw-brown merged commit cd7189e into NCAR:main Mar 12, 2024
4 checks passed

hkershaw-brown mentioned this pull request Mar 12, 2024

Feature request: mitgcm_ocean compressed netcdf files for dart #391

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MITgcm/N-BLING with Compressed Staggered Grids #640

MITgcm/N-BLING with Compressed Staggered Grids #640

mgharamti commented Feb 20, 2024 •

edited by hkershaw-brown

Loading

hkershaw-brown commented Mar 1, 2024

hkershaw-brown left a comment

hkershaw-brown Mar 1, 2024

mgharamti commented Mar 1, 2024

hkershaw-brown left a comment •

edited

Loading

MITgcm/N-BLING with Compressed Staggered Grids #640

MITgcm/N-BLING with Compressed Staggered Grids #640

Conversation

mgharamti commented Feb 20, 2024 • edited by hkershaw-brown Loading

Description:

Fixes issue

Types of changes

Documentation changes needed?

Tests

Checklist for merging

Checklist for release

Testing Datasets

hkershaw-brown commented Mar 1, 2024

hkershaw-brown left a comment

Choose a reason for hiding this comment

hkershaw-brown Mar 1, 2024

Choose a reason for hiding this comment

mgharamti commented Mar 1, 2024

hkershaw-brown left a comment • edited Loading

Choose a reason for hiding this comment

mgharamti commented Feb 20, 2024 •

edited by hkershaw-brown

Loading

hkershaw-brown left a comment •

edited

Loading