UGRID with TDS and edal-java/ncwms

[lesserwhirls]

Get a UGRID example from @rsignell-usgs and see how it works with TDS 5 and the new edal-java/ncwms plugin.

[lesserwhirls]

Ok, so there is a bit of a hangup here, in the the new version godiva3 is causing an error with the TDS, in that it is trying to call:

http://<server>/thredds/getconfig

which currently returns a 404. Note that wms seems to be working just fine - it's a godiva3 thing that we have not implemented in the TDS. I need to look into what exactly godiva3 is looking for here and make sure we return that. @guygriffiths - any pointers to documentation as to what exactly godiva3 wants from the getconfig request? Thanks in advance!

[guygriffiths]

That shouldn't be a problem. It exists so that Godiva3 can get settings about the map size, data proxy, and the location of the documentation from a server. It was written for a specific project - we don't actually use it in the default ncWMS2 implementation, but saw no reason to remove it. Best thing is to ignore it.

[rsignell-usgs]

So we figured we would help out here and get TDS 5.0 installed locally where we have a lot of UGRID data (and the provider is very keen to get WMS services going).

We installed TDS 5.0 using the docker container from https://hub.docker.com/r/axiom/docker-thredds/ at http://www.smast.umassd.edu/thredds/catalog/catalog.html and although OPeNDAP service and HTTPS services are working okay, the WMS service is not working.

For example, on this UGRID dataset:
http://www.smast.umassd.edu/thredds/catalog/forecasts.html?dataset=gom3_nocache

If we click the WMS link (http://www.smast.umassd.edu/thredds/wms/FVCOM/NECOFS/Forecasts/NECOFS_GOM3_FORECAST.nc?service=WMS&version=1.3.0&request=GetCapabilities) we get:

<ServiceExceptionReport xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.3.0" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wms/1.3.0/exceptions_1_3_0.xsd">
<ServiceException>
The location FVCOM/NECOFS/Forecasts/NECOFS_GOM3_FORECAST.nc doesn't refer to any existing files.
</ServiceException>
</ServiceExceptionReport>

So it would seem like there is some sort of path issue.

We can see that WMS getCapabilities is working on Jumbo, so we are wondering what we could be doing wrong. Ideas?

[rsignell-usgs]

BTW, if you are wondering which version of TDS 5.0 we used, we pulled the latest artifact from your nexus repo.

[rsignell-usgs]

@lesserwhirls ping!

[lesserwhirls]

Hi @rsignell-usgs - the snapshot you are using is newer than the one on thredds-test.unidata.ucar.edu. I'm building the latest locally and going to test the wms on my machine, and if everything looks good, I'll publish a new set of 5.0 snapshots. Godiva3 still won't function, but we are digging into that (not sure why the WMS isn't working in the snapshot you are using though).

[rsignell-usgs]

@lesserwhirls , awesome. While Godiva3 would be nice, we really care most about the WMS and ISO services. If those are working we will be able to harvest ISO metadata from UGRID complaint data on TDS5.0 servers and access the ncWMS2 WMS services in our portals like
http://goo.gl/BG4LN8.
And that will be most cool.

[lesserwhirls]

Ok, I'm currently publishing a new snapshot which should work.

I will be out all next week, but I can try to get threddsIso integrated into 5.0 during the evenings. I have a version that builds with the new 5.0 stuff, but it's not yet tied into the TDS.

[lesserwhirls]

@rsignell-usgs http://artifacts.unidata.ucar.edu/content/repositories/unidata-snapshots/edu/ucar/tds/5.0.0-SNAPSHOT/tds-5.0.0-20160708.184540-12.war

[rsignell-usgs]

@kwilcox, can you rebuild the Docker container for 5.0 with this artifact?

[rsignell-usgs]

Moved this artifact discussion to #495 (comment)

[lesserwhirls]

@rsignell-usgs - have you tried the wms service using that snapshot for a regular gridded dataset? I think I need to tweak a few things on the TDS side to recognize the possibility for ugrid to work with wms before it will actually pass the dataset to the wms server (this hit me about half-way through Kansas on my way back from Indiana...something about the plains I suppose).

[rsignell-usgs]

Yes, when I hit the WMS Capabilities on this regular structured-grid FMRC endpoint:
http://gamone.whoi.edu/thredds/catalog/coawst_4/use/fmrc/catalog.html?dataset=coawst_4/use/fmrc/coawst_4_use_best.ncd

I get back:

<ServiceExceptionReport xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.3.0" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wms/1.3.0/exceptions_1_3_0.xsd">
<ServiceException>
The location fmrc:coawst_4_use doesn't refer to any existing files.
</ServiceException>
</ServiceExceptionReport>

[rsignell-usgs]

Is there something different required in the catalog for FMRC in 5.0?
Is the catalog for thredds-jumbo (http://thredds-jumbo.unidata.ucar.edu/thredds/catalog/catalog.html) on github somewhere?

[lesserwhirls]

The 5.0 catalogs used on thredds-jumbo (which is actually thredds-dev) can be found here:

https://github.com/Unidata/TdsConfig/tree/master/threddsDev

Not sure why wms isn't working with your gridded dataset...only thing I can think of is something in FMRC, but I'm not positive.

[lesserwhirls]

Ah, looks like FMRC stuff did change - see:

https://www.unidata.ucar.edu/software/thredds/v5.0/tds/reference/collections/FmrcCollection.html
Specifically, the <fmrcDataset> element in the server configuration catalogs is no longer used - it's all under <featureCollection ...> now.

[rsignell-usgs]

@lesserwhirls, that didn't change in 5.0 though -- it changed in 4.2. We have been using <featureCollection ...> for our FMRC for years now. So it must be something else...

I just checked and none of our WMS endpoints are working -- even regular bathymetry grids like
http://gamone.whoi.edu/thredds/catalog/global_bathy.html?dataset=bathy/etopo2_v2c.nc

when I click on the WMS getcaps endpoint are reporting back:

<ServiceExceptionReport xmlns="http://www.opengis.net/ogc" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.3.0" xsi:schemaLocation="http://www.opengis.net/ogc http://schemas.opengis.net/wms/1.3.0/exceptions_1_3_0.xsd">
<ServiceException>
The location bathy/etopo2_v2c.nc doesn't refer to any existing files.
</ServiceException>
</ServiceExceptionReport>

Does that give any clues?

[lesserwhirls]

This is so odd. I cannot even grab the bathymetry grid for testing locally using the HTTPServer (I get a 404). Would you mind if I took a look at your threddsConfig.xml, the catalog used to serve up the bathymetry grid, as well as your top level catalog.xml file?

[rsignell-usgs]

Okay, here are the files. https://drive.google.com/open?id=0BzAHlPEEP_ujM3p2MEl5TmUwajg

[lesserwhirls]

I've moved the issue of simple grids not being accessible to the new ncWMS to #618. We'll keep this one open for UGRID specific discussions once we get the simple grids to work.

[drf5n]

Unidata is packaging TDS 5.0 for Docker: https://github.com/Unidata/thredds-docker
Currently this might be 2016-12-13
Axiom's is https://github.com/axiom-data-science/docker-thredds/tree/5.0 looks like 2016-09-21

Is this working?

[kwilcox]

Please use the docker container from the Unidata org! Everything from the axiom container was merged into it. I really need to take down the Axiom one.

[drf5n]

Does FMRC stuff work on UGRID datasets? I saw old discussions around http://www.unidata.ucar.edu/mailing_lists/archives/thredds/2013/msg00503.html but I can't seem to make it work.

[rsignell-usgs]

@drf5n, AFAIK, FMRC currently requires featureType=grid, so no, it doesn't work.

This is especially frustrating because the logic in constructing the FMRC doesn't have anything to do with the horizontal grid system.

So if NetCDF-Java recognined featureType=ugrid, I'm pretty sure the FMRC could just accept that type also and it would just work.

Is that right @lesserwhirls ?

[lesserwhirls]

That's a really good question @rsignell-usgs! I think netCDF-Java still needs to be able to recognize the data as "gridded data", which means that it understands the horizontal coordinate system (even though FMRC might not to anything with that info).

[drf5n]

@rsignell-usgs, based on your old email, I seem to be able to make my ugrid data regognizable as dataType='grid' with a call to NCO's ncrcat to add a degenerate record dimension.

    ...
float depth(record, node) ;
	depth:long_name = "Bathymetry" ;
	depth:units = "meters" ;
	depth:positive = "down" ;
	depth:mesh = "mesh" ;
	depth:location = "node" ;
float elev(record, time, node) ;
	elev:long_name = "elev.61" ;
	elev:mesh = "mesh" ;
	elev:location = "node" ;
int crs_a(record) ;
	crs_a:long_name = "WGS84 coordinate reference system and transformation" ;
	crs_a:_CoordinateAxes = "lat lon zpos time" ;
	crs_a:_CoordinateTransforms = "zpos" ;
	crs_a:_CoordinateSystemFor = "elev depth" ;
float lat(record, node) ;
	lat:units = "degrees_north" ;
	lat:standard_name = "latitude" ;
	lat:mesh = "mesh" ;
	lat:location = "node" ;
float lon(record, node) ;
	lon:units = "degrees_east" ;
	lon:standard_name = "longitude" ;
	lon:mesh = "mesh" ;
	lon:location = "node" ;
    ...

[drf5n]

@rsignell-usgs -- See #778 (comment) for hints at an FMRC-ish workaround with NCML's <logicalSection section="(0:24,:,:,:)"> Still, real FMRC on UGRID through recognizing UGRID as a dataType=grid seems clearer.

I wonder what in NetCDF-Java recognizes UGRID files with the added degenerate dimension as dataType=grid? If the coordinate's rank is >1 it passes?

[rsignell-usgs]

Thanks @drf5n!

@lesserwhirls, to me this indicates how easy it should be to allow a UGRID to be an FMRC! 😸

[lesserwhirls]

I agree, these results look very promising! What I'd like to do is hammer out featureType=grid and include ugrid in that. The new edal-java library is capable to handling the ugrid spec, and I wonder what it take to use that to make a CoordSysBuilder for ugrid. We need to get v5.0 out first, but this is high up on the list.

[rsignell-usgs]

I can verify that WMS is working with UGRID in TDS v5.0.0 beta2, so good to close!