Support for writing Self Attraction and Loading (SAL) files in NetCDF for the ADCIRC model #231

krober10nd · 2021-06-19T17:17:39Z

Addresses Create SAL input files (fort.24) in NetCDF #229 @zcobell
Currently SAL forcing files for ADCIRC are written in ASCII text which can become cumbersome.
This PR writes these forcing files (fort.24) in a compressed NetCDF format by default and deprecates the ASCII files.
Note this requires upstream development in the ADCIRC source code that reflects these changes netCDF based SAL files (fort.24) adcirc/adcirc#271 (comment)
For a small mesh with 191,600 nodes the reduction from the previous ASCII format is ~2x (e.g., 55 mb to 22 mb).
MWE

clearvars; close all; clc;

m = msh('dummy.14');

TPXO9 = '/Volumes/KeithJaredR/MESHING_DATASETS/TIDAL_CONDITIONS/h_tpxo9.v1.nc';

TS = '2012 Aug 01 00:00';

TE = '2012 Nov 01 00:00';

SALDATA =  'FES2004';

DT = 1.0;

m = Make_f15(m, TS, TE, DT, 'tidal database', TPXO9, 'const',{'major8'}) ;

m = Make_f24(m, SALDATA, 1, 1);

write(m,'testing','f24');

The following codes creates a fort.24 file called testing.24.nc with the following header containing 8 major tidal constituents. Note the constituents are named with a unique alphanumeric identifier corresponding to FES2004. Any thoughts on a better way to do this are suggested. Note the *_omega values are scalars.

(base) keiths-MacBook-Pro:private Keith$ ncdump -h testing.24.nc 
netcdf testing.24 {
dimensions:
	valspernode = 3 ;
	nnodes = 191600 ;
variables:
	double Q1_omega ;
	double Q1_vals(nnodes, valspernode) ;
	double O1_omega ;
	double O1_vals(nnodes, valspernode) ;
	double P1_omega ;
	double P1_vals(nnodes, valspernode) ;
	double K1_omega ;
	double K1_vals(nnodes, valspernode) ;
	double N2_omega ;
	double N2_vals(nnodes, valspernode) ;
	double M2_omega ;
	double M2_vals(nnodes, valspernode) ;
	double S2_omega ;
	double S2_vals(nnodes, valspernode) ;
	double K2_omega ;
	double K2_vals(nnodes, valspernode) ;
}

…and 0/360 format forced for fes2014, oceanmesh mesh can be in either format and will work

…ent for incorrect database entry

…e() function. adding error catch for wrong format input

…rt.24

WPringle · 2021-06-21T16:59:28Z

I think fort.24 netcdf format could be improved by adding number of frequency dimensions and metadata information. I can work on this a bit later today.

…descriptions.

WPringle · 2021-06-22T04:11:12Z

@zcobell @krober10nd see what you think of the format..

Example:

WPringle

made adjustments. back to KJR for re-review.

zcobell · 2021-06-22T04:24:10Z

Are there CF conventions for tide parameters? The current adcirc format is loosely CF-ish in naming and variable fields and attributes

krober10nd · 2021-06-22T13:07:39Z

These appear to be the CF conventions https://www.bodc.ac.uk/resources/delivery_formats/cfnetcdf_format/

Also Zach brought this up offline but the __FillValue should be 0.0 so that NetCDF (at least the compressed versions) won't write that data to the file and save a lot of memory potentially for meshes with many dry nodes.

So in each variable probably should have

units 
standard_name
long_name 
_FillValue

Probably should also have this at the end:

title — a succinct description of the data set
insititute — the organisation where the data were produced
source — how the data were produced, e.g. model type, run number and circumstances
history — an audit trail of data set processing
references — a list of references that describe the data or the methodology used
comment — other useful information not covered elsewhere that adds value
author — the person(s) who generated the data

zcobell · 2021-06-22T15:52:38Z

There is a standard_name table worth consulting too

https://cfconventions.org/standard-names.html

krober10nd · 2021-06-25T17:26:25Z

A couple suggestions:

I wouldn't called it salnodes it should be consistent with whatever is used in the other output files for the nodes.
I wouldn't abbreviate the fields (e.g., numconst -> number_of_constants and I would separate them with an underscore (snake case).
sal is an acronym so it should be capitalized SAL and then it's long name should be self-attraction and loading
salamp -> SAL_amplitude
salphs -> SAL_phase
_FillValue should be set to 0 for all the fields/variables.

WPringle · 2021-06-25T18:55:16Z

The salnodes is different to other output files for the nodes, because it is (potentially) a subset of nodes where SAL is defined on.

The fill value of zero is redundant because actually SAL is defined non-zero everywhere (even overland, same as the potential).

krober10nd · 2021-06-25T20:11:33Z

Oh okay, that part wasn't clear. Wouldn't this then be different then from how all the other output files are defined?

Is it really more efficient to store indices of where the values are defined vs. use a _FillValue?

WPringle · 2021-06-25T20:20:07Z

Oh okay, that part wasn't clear. Wouldn't this then be different then from how all the other output files are defined?

Is it really more efficient to store indices of where the values are defined vs. use a _FillValue?

It's just that ADCIRC requires knowing the indices that each SAL point is defined on. Although as a standalone file its worth knowing the geographic locations this is for an input file for ADCIRC.

krober10nd · 2021-06-26T19:21:30Z

Okay this is good to go by me. I added some global attributes to the created file. I also added a warning telling users about the potential to use the NetCDF format.

>> ncdisp testing.24.nc
Source:
           /Volumes/KeithJaredR/testing.24.nc
Format:
           netcdf4_classic
Global Attributes:
           title         = 'The self-attraction and loading terms for an ADCIRC simulation'
           creation_date = '26-Jun-2021 16:17:49'
           source        = 'Made by OceanMesh2D writefort24'
           references    = 'https://github.com/CHLNDDEV/OceanMesh2D/'
Dimensions:
           sal_nodes = 191600
           num_const = 8
           char_len  = 2
Variables:
    mesh_indices
           Size:       191600x1
           Dimensions: sal_nodes
           Datatype:   int32
           Attributes:
                       long_name = 'index of SAL values into mesh coordinates'
    const       
           Size:       8x2
           Dimensions: num_const,char_len
           Datatype:   char
           Attributes:
                       long_name = 'names of the tidal harmonic constituents'
    frequency   
           Size:       8x1
           Dimensions: num_const
           Datatype:   double
           Attributes:
                       long_name = 'frequency of harmonic constituents'
                       units     = 'rad/s'
    salamp      
           Size:       8x191600
           Dimensions: num_const,sal_nodes
           Datatype:   single
           Attributes:
                       long_name = 'amplitude of self-attraction and loading tide elevation'
                       units     = 'm'
    salphs      
           Size:       8x191600
           Dimensions: num_const,sal_nodes
           Datatype:   single
           Attributes:
                       long_name = 'phase-lag of self-attraction and loading tide elevation'
                       units     = 'degrees with respect to GMT/UTC'

WPringle · 2021-06-27T22:47:28Z

Cool, you didn't want to change to SAL_amplitude and SAL_phase etc? I think that is probably a good idea.

zcobell · 2021-06-28T14:43:55Z

I think this format should probably mirror the fort.63 format as closely as we can for consistency across ADCIRC files and without the lookup table. I also would not abbreviate things so they aren't clear. I'd recommend something like this:

dimensions:
        node = 1578383 ;
        num_constituents = 8 ;
        char_len  = 2 ;
variables:
        double x(node) ;
                x:long_name = "longitude" ;
                x:standard_name = "longitude" ;
                x:units = "degrees_east" ;
                x:positive = "east" ;
        double y(node) ;
                y:long_name = "latitude" ;
                y:standard_name = "latitude" ;
                y:units = "degrees_north" ;
                y:positive = "north" ;
         constituents(num_constituents,char_len);
                constituents:long_name = 'names of the tidal harmonic constituents'
                constituents:standard_name = 'name_of_tidal_harmonic_constituents' // is there a standard name? if not we should follow the CF standard name construction format
         frequency(num_constituents);
                frequency:long_name = 'frequency of tidal harmonic constieuents' ;
                frequency:standard_name = 'frequency_of_tidal_harmonic_constituents' ;
                frequency:units = 'radians/second';
         sal_amplitude(num_constituents,node);
                sal_amplitude:long_name = 'amplitude of self-attraction and loading tide elevation' ;
                sal_amplitude:standard_name = 'amplitude_of_self_attraction_and_loading_tide_elevation' ;
                sal_amplitude:units = 'm' ;
                sal_amplitude:_FillValue = 0.0 ;
         sal_phase(num_constituents,node);
                sal_phase:long_name = 'phase-lag of self-attraction and loading tide elevation' ;
                sal_phase:standard_name = 'phase_lag_of_self_attraction_and_loading_tide_elevation' ;
                sal_phase:units = 'degrees with respect to GMT/UTC' ;
                sal_phase:_FillValue = 0.0 ;

Pass vertices to writefort24.

krober10nd · 2021-06-28T15:38:46Z

>> ncdisp testing.24.nc
Source:
           /Users/Keith/junk/OceanMesh2D/testing.24.nc
Format:
           netcdf4_classic
Global Attributes:
           title         = 'The self-attraction and loading terms for an ADCIRC simulation'
           creation_date = '28-Jun-2021 12:35:29'
           source        = 'Made by OceanMesh2D writefort24'
           references    = 'https://github.com/CHLNDDEV/OceanMesh2D/'
Dimensions:
           node             = 191600
           num_constituents = 8
           char_len         = 2
Variables:
    x            
           Size:       191600x1
           Dimensions: node
           Datatype:   int32
           Attributes:
                       standard_name = 'longitude'
                       units         = 'degrees_east'
                       positive      = 'east'
    y            
           Size:       191600x1
           Dimensions: node
           Datatype:   int32
           Attributes:
                       standard_name = 'latitude'
                       units         = 'degrees_north'
                       positive      = 'north'
    constituents 
           Size:       8x2
           Dimensions: num_constituents,char_len
           Datatype:   char
           Attributes:
                       standard_name = 'name_of_tidal_harmonic_constituents'
                       long_name     = 'name of tidal harmonic constituents'
    frequency    
           Size:       8x1
           Dimensions: num_constituents
           Datatype:   double
           Attributes:
                       standard_name = 'frequency_of_tidal_harmonic_constituents'
                       long_name     = 'frequency of tidal harmonic constituents'
                       units         = 'radians/second'
    sal_amplitude
           Size:       8x191600
           Dimensions: num_constituents,node
           Datatype:   single
           Attributes:
                       _FillValue    = 0
                       standard_name = 'amplitude_of_self_attraction_and_loading_tide_elevation'
                       long_name     = 'amplitude of self attraction and loading tide elevation'
                       units         = 'm'
    sal_phase    
           Size:       8x191600
           Dimensions: num_constituents,node
           Datatype:   single
           Attributes:
                       _FillValue    = 0
                       long_name     = 'phase-lag of self-attraction and loading tide elevation'
                       standard_name = 'phase_lag_of_self_attraction_and_loading_tide_elevation'
                       units         = 'degrees with respect to GMT/UTC'

WPringle · 2021-06-28T16:23:41Z

Cool this is fine by me, since it is more standalone.

@zcobell so, is your proposal to do the mapping of the nodes inside adcirc? Or, since practically speaking we generally always apply sal nodes everywhere you could enforce that number of nodes between this file and the fort.14 are the same?

zcobell · 2021-06-28T16:25:51Z

Right, I think this file becomes tied to the mesh numbering similar to the fort.13 and apply the values inside of adcirc as appropriate. Having the lookup table also complicates the decomposed fort.24

WPringle · 2021-06-28T16:28:02Z

I'm fine to merge it if we are good btw.

krober10nd · 2021-06-28T16:50:32Z

In regard to the CF standard names for name_of_tidal_harmonic_constituents this appears fine and in agreement with the guidelines (no standard name already exists upon searching their database).

http://cfconventions.org/Data/cf-standard-names/docs/guidelines.html

zcobell · 2021-06-28T16:52:14Z

For merging, I'd say let's throw the format example up on the adcirc issue for feedback and probably no use merging this here until ADCIRC/ADCPREP/PADCIRC actually support it too.

krober10nd · 2021-06-28T16:53:34Z

Yep, agreed it has to wait for changes in ADCIRC anyway.

krober10nd · 2022-02-25T10:07:31Z

Let's update this with the current branch and merge it in. Having SAL files in a simple NetCDF format may be useful for other models besides ADCIRC at some point too.

Keith Roberts added 2 commits June 19, 2021 14:07

support for writing fort24 in netcdf

d93e32d

using compression when creating netcdf files

6019056

krober10nd requested a review from WPringle June 19, 2021 17:17

WPringle added 4 commits June 21, 2021 10:59

removing format option and making sure -180/180 enforced for fes2004 …

b818007

…and 0/360 format forced for fes2014, oceanmesh mesh can be in either format and will work

bugfix for if statements without end in Make_f24. adding error statem…

9956841

…ent for incorrect database entry

making netcdf option for fort.24 callable through the normal msh.writ…

b78e16d

…e() function. adding error catch for wrong format input

moving mesh indices into its own variable as an integer for netcdf fo…

6a178a9

…rt.24

making SAL format more like fort.53.nc style with long_name and unit …

9a867bc

…descriptions.

WPringle approved these changes Jun 22, 2021

View reviewed changes

adding some global attributes to the file and a warning

2a7c86b

updating README

0d8484f

krober10nd linked an issue Jun 26, 2021 that may be closed by this pull request

Create SAL input files (fort.24) in NetCDF #229

Closed

Merge remote-tracking branch 'origin/Projection' into sal_netcdf

73b7272

variable name modification

558504a

Keith Roberts and others added 2 commits June 28, 2021 12:36

updating format

8378a39

Update msh.m

5d609b1

Pass vertices to writefort24.

krober10nd added the enhancement New feature or request label Jul 21, 2021

WPringle added 2 commits February 25, 2022 09:27

updating changelog for this PR

cf1a84f

Merge branch 'Projection' into sal_netcdf

ac4c6a2

WPringle merged commit 75e2b00 into Projection Feb 25, 2022

WPringle deleted the sal_netcdf branch February 25, 2022 15:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for writing Self Attraction and Loading (SAL) files in NetCDF for the ADCIRC model #231

Support for writing Self Attraction and Loading (SAL) files in NetCDF for the ADCIRC model #231

krober10nd commented Jun 19, 2021 •

edited

Loading

WPringle commented Jun 21, 2021 •

edited

Loading

WPringle commented Jun 22, 2021

WPringle left a comment

zcobell commented Jun 22, 2021

krober10nd commented Jun 22, 2021 •

edited

Loading

zcobell commented Jun 22, 2021 •

edited

Loading

krober10nd commented Jun 25, 2021

WPringle commented Jun 25, 2021

krober10nd commented Jun 25, 2021

WPringle commented Jun 25, 2021

krober10nd commented Jun 26, 2021

WPringle commented Jun 27, 2021

zcobell commented Jun 28, 2021

krober10nd commented Jun 28, 2021

WPringle commented Jun 28, 2021 •

edited

Loading

zcobell commented Jun 28, 2021

WPringle commented Jun 28, 2021

krober10nd commented Jun 28, 2021

zcobell commented Jun 28, 2021

krober10nd commented Jun 28, 2021

krober10nd commented Feb 25, 2022

Support for writing Self Attraction and Loading (SAL) files in NetCDF for the ADCIRC model #231

Support for writing Self Attraction and Loading (SAL) files in NetCDF for the ADCIRC model #231

Conversation

krober10nd commented Jun 19, 2021 • edited Loading

WPringle commented Jun 21, 2021 • edited Loading

WPringle commented Jun 22, 2021

WPringle left a comment

Choose a reason for hiding this comment

zcobell commented Jun 22, 2021

krober10nd commented Jun 22, 2021 • edited Loading

zcobell commented Jun 22, 2021 • edited Loading

krober10nd commented Jun 25, 2021

WPringle commented Jun 25, 2021

krober10nd commented Jun 25, 2021

WPringle commented Jun 25, 2021

krober10nd commented Jun 26, 2021

WPringle commented Jun 27, 2021

zcobell commented Jun 28, 2021

krober10nd commented Jun 28, 2021

WPringle commented Jun 28, 2021 • edited Loading

zcobell commented Jun 28, 2021

WPringle commented Jun 28, 2021

krober10nd commented Jun 28, 2021

zcobell commented Jun 28, 2021

krober10nd commented Jun 28, 2021

krober10nd commented Feb 25, 2022

krober10nd commented Jun 19, 2021 •

edited

Loading

WPringle commented Jun 21, 2021 •

edited

Loading

krober10nd commented Jun 22, 2021 •

edited

Loading

zcobell commented Jun 22, 2021 •

edited

Loading

WPringle commented Jun 28, 2021 •

edited

Loading