OG1.0 checker

[vturpin]

There will be a need for OG1.0 checker that will also check the cf compliance.

OG1.0 is cf.1.10 compliante, but cf.1.10 do not offer any complient checker yet. However, cf1.8 can be checked.
The change from cf.1.8 to cf1.10 does not affect OG1.0

It is being discussused here that the OG1.0 checker will, amongst other verification, check compliance with cf1.8 using what exist and assuming that OG1.0 is compliant with cf.1.10 is compliant with cf1.8.

@castelao is working on this particular topic.

[jcermauwedu]

Has this progressed to any degree? Is this to be a separate tool or is it suitable to be included an existing compliance checker? I also raised this an an issue on the compliance-checker repo to see if there are any opinions there. I cross linked this discussion in that issue. @castelao

[castelao]

@jcermauwedu , yes, I have a running proof of concept and will look for beta testers soon. It will be an independent library but there is no reason why it couldn't be used by that tool.

[jcermauwedu]

It would be great to have a gold standard example netCDF files to test the checker against much like NCEI has for their templates.

There are two initial examples that can be found in the manual for reference:

• sp041_20191205T1757-instrument-scalar_v2
• minimal

It would be best to have several examples to exercise as much of the template as possible.

[castelao]

@jcermauwedu , great point! This was discussed early this year in an OG meeting and it was agreed to work on that. The plan is exactly on having examples of the minimal setup for different platforms as well as expanded examples with more sensors and special cases. I believe @callumrollo already started to work on that, but I don't recall which platform(s).

@jcermauwedu , could you be interested in volunteer to provide one example yourself? Maybe start with a minimal case, i.e. only the 'mandatory' required attributes. I expect that to be a challenging task, thus your feedback on what is not clear in the manual, or even conflicting definitions, would be helpful and much appreciated by the OG committee.

[jcermauwedu]

@castelao Sounds like a good challenge. We are attempting to construct a suitable structure for an active acoustic (echogram) dataset and a companion acoustic metrics dataset. Here is our challenge: We do not have a single point at depth for each time coordinate, we have a vertical scan (assuming the acoustic sensor is pointed straight up) of 60 meters. We are painting a swath of data along a profile. That presents a difficulty in the general handling of glider data.

Sv(t, [d0, d1, d2, d3, d4,...])

We could complicate this by also integrating the pitch and yaw from the glider in more complicated example where we have a scan that will not only vary by depth, but in the full 3D spatial frame for each scan.

Sv(t, [(d0, lat0, lon0), (d1, lat1, lon1)]

The acoustic metrics are more straight forward. These have a typical time coordinate with depth (at the moment).

I'll start looking at the OG-1.0 manual and take shot at construction.

Where or who can we send the constructed netCDF files to see if they can be considered "good" examples?

At the moment, I am kicking the tires on the IOOS compliance checker and plugins by working through the existing python tests. The existing two plugins might serves as a good template to begin rolling a OG-1.0 plugin to the checker through use of the future library.

[castelao]

@jcermauwedu , if I got it right, you would need an extra dimension. I believe that the current OG structure is based on N_MEASUREMENTS, so your data could be something like (N_MEASUREMENTS, N_BINS). Each N_MEASUREMENTS can be mapped to a time and depth. With the information of the position of the platform, each bin could be projected in a 3D space, but for the purpose of storing the data, 2D arrays would be enough. Extending a little further, this would require some metadata on the alignment of the bins so one can estimate those angles. I don't know which precision you would need.

Thinking together on these special cases helps to identify flaws in the standard. We already fixed many, but sure there are still a few. My suggestion is to share early so more people can give ideas. This is a fantastic team, supportive and knowledgeable, so don't be shy.

@emmerbodc and @glidermann are the chairs, and are probably the best point of contact. My suggestion is to open a PR with your example so everyone can see and comment directly on it, but I'll leave to the chairs to decide the best way to move forward.

[emmerbodc]

Yes please open a PR @jcermauwedu . Would this sensor data come back off the glider in real time? I only ask as the format has been built with real time data in mind and not for a delayed dataset which we will need to consider next.

Thanks,
Emma

[jcermauwedu]

@emmerbodc I opened an issue, not a PR. (#176). There is a near real time feed. In fact, there is a glider with an EK80 in the water now. The sensor is currently off as it is a battery hog. It will be on again in the near future. Once I find a structure that also seems to work with ERDDAP, I will submit a sample as a PR. The structure that is semi-operational is flattened. The link is in the issue. Thank you all for responding.

[jcermauwedu]

A web checker is now available here for starting to curate datasets: http://conventions.castelao.net/OG/1.0/upload

[jcermauwedu]

There is community support to also keep the companion python IOOS Compliance Checker plugin started for OG-1.0. The repository is https://github.com/uw-farlab/cc-plugin-og. When it is further along, it can be proposed to be brought under the general IOOS repository. The base CF checking needs to be improved to match the level supported by OG-1.0, but the basic rules of OG-1.0 can be checked at least. It is great to see the final v1.0 published. I am fine if this plugin is eventually deprecated in favor of the Rust version in the future.