Geothermal prospector support in NGDS CKAN v2

[smrgeoinfo]

see #404, #459
Function works in NGDS CKAN v1. the Button should show up with the WMS distribution links for search results that have WMS distributions
@ccaudill
here's some notes from Dan Getman e-mail:
From: Getman, Dan Dan.Getman@nrel.gov
Sent: Tuesday, August 19, 2014 1:52 PM
To: Adrian Sonnenschein;
...

1. Just a point of clarification on parameters. The parameters that you should be using, and are for the most part, have not changed and include:

wmsHost: a string (URL escaped) containing the endpoint URL for a WMS; this is the part of the URL that precedes the WMS parameters, basically a WMS request with no parameters part (leave out the question mark). REQUIRED

wmsLayerName: the name of a layer in the WMS to display. ONLY one layer name allowed. REQUIRED

wfsHost: a string (URL escaped) containing the endpoint URL for a WFS that provides feature access to the data displayed by the WMS. The string argument is the part of the URL that precedes the WFS parameters, basically a WFS request with no parameters part (leave out the question mark). OPTIONAL

wfsFeatureTypeName: the name of the wfs feature type exactly as it needs to be specified in the TypeName WFS parameter; the namespace qualifier might or might not be required, I think that might depend on the service implementation. OPTIONAL

You can also use:
zoomlevel: an integer to set the desired zoom level
mapcenter: a string (URL escaped) containing the coordinates for the center of the map "lon,lat"
overlaylayer: Setting this to 0 will turn all layers off by default.
Setting this to anything else may be unpredictable.

It is important to note that the layer ids for baselayer parameter have indeed changed and will likely change in the future as well. This parameter has been inadvertently included in some of our example URLs (cutting and pasting error in emailing unfortunately) but was not intended to be used to call GTP from NGDS. This parameter, and many others that you may see loaded into the URL by GTP, are for our use in allowing users to send other users URLs.

2. The URL that you sent has a couple of other issues.

First, setting the baselayer to a now invalid ID is causing no base layer to be loaded. This should explain the gray base map that you observed.

Second, the layer that you are referencing only shows up at zoom level 7 and you have zoom level 1 specified. The result is that the layer actually loads, but is not visible.

You can specify a zoom level of 7 and it will work:
https://maps-stage.nrel.gov/geothermal-prospector/#/?zoomlevel=7&wmsHost=ht
tp://www.geocommunicator.gov/arcgis/services/PLSS/MapServer/WMSServer&wmsLa
yerName=11

Third, the map service that you sent is reporting a non-working legend URL. I'm not sure what the issue is with this, but it will prevent the layer from displaying a legend.

Next Steps:

On my end, we will make sure that the baselayer parameter is handled such that any layer id that is not valid will result in a default base layer being loaded, which should prevent the gray base map issue that you observed.

Also, to address your main concern, I believe it unlikely that the URL parameters which are specified here and in our original emails providing the descriptions will change. If that does occur, we will notify you beforehand to make sure that it will not create a problem on your end.

[jessicagood]

Testing http://uat-ngds.reisys.com/dataset/indiana-borehole-temperatures-wms-wfs-2 (as specified in https://github.com/ngds/ckanext-metadata/issues/14) and it looked like it worked as should. WMS was drawn and features could be queried. Though testing a few other services may have crashed the server hosting the Geothermal Prospector - or it was coincidental that the Geothermal Prospector stopped working at the same time.

[JihadMotii-REISys]

I've done a changing, here https://github.com/ngds/ckanext-metadata/issues/14#issuecomment-68073532. it could be a fix for this (if it wasn't from Geothermal Prospector server)

[ccaudill]

@JihadMotii-REISys I don't see Geothermal Prospector button in new build

[JihadMotii-REISys]

is the new build UAT ? otherwise, can you send me the link to the new build ?
Thank you @ccaudill

[ccaudill]

Sorry, it's our internal servers, updated to the latest rpm.
159.87.39.4
159.87.39.5

[JihadMotii-REISys]

I created a dataset in UAT that is up to date with all new code/ configuration, this is the link:
http://uat-ngds.reisys.com/dataset/test-dataset

In order to make geothermal button working, your internal servers has to have access to external and also geoserver needs to be set up. (if you're behind reverse proxy) ...

[ccaudill]

I don't understand why this opens in GP with the data that it does, and not well log data that is specified. Perhaps that doesn't matter for this test. Also, the button just vanished recently, so there shouldn't be any work with getting it configured again. You say that external servers need to be accessed and GeoServer needs to be set up, but that's already done in any installation, so I'm confused about why you mention that.

The button is there, I'm just not completely sure that it's working properly.

[ykhadilkar]

Hey @ccaudill

Just want to make sure you followed below steps to set up geoserver after initial installation

1. After all the steps please update two config files. Update "ckan.hostname" variable with correct URL(without http://) in file /etc/ckan/production.ini
2. update proxyBaseUrl in file /var/lib/tomcat6/webapps/geoserver/data/global.xml, replacing 127.0.0.1 with correct URL. Restart server after update.

Also make sure internal servers can access https://maps-stage.nrel.gov/ url. And as far as I understand Geothermal Prospector application at https://maps-stage.nrel.gov/ contacts back to CKAN node to get the data to be displayed on map. And that is why we suspect Geothermal Prospector is not able to display data from internal server.

Also I'm little confused about button visibility ... Are you saying button now shows up but when clicked shows up wrong map?

[ccaudill]

You're saying that after running the update rpm at https://github.com/ngds/install-and-run#updating-ngds there's other things that need to be done?

[ccaudill]

The button isn't there at all. Should I be looking in another place?

[ykhadilkar]

No ... not after update only after initial installation. If you did after initial installation then don't worry about it.

[ykhadilkar]

I suspect its because server not accessible from outside. Like I said above..
"as far as I understand Geothermal Prospector application at https://maps-stage.nrel.gov/ contacts back to CKAN node to get the data to be displayed on map. And that is why we suspect Geothermal Prospector is not able to display data from internal server."
Is it possible to open this server to internet just make sure that is the problem?

[ccaudill]

We followed the instructions here to install: https://github.com/ngds/install-and-run#installation
If more steps are required than are listed here, please edit that readme to include them.

[ccaudill]

The server is open to the internet. You should be able to hit it.

[ykhadilkar]

@ccaudill Can you please email be below 2 files or give me SSH access to the server?

• /etc/ckan/production.ini
• /var/lib/tomcat6/webapps/geoserver/data/global.xml

[ykhadilkar]

@ccaudill in production.ini ckan.hostname was incorrect (It was default 127.0.0.1) So changed it to correct IP address. But there is something wrong with datapusher / datastore. I am not able to push the CSV data to datastore at all. @FuhuXia any idea?
Also in your dataset http://159.87.39.5/dataset/vermont-thermal-conductivity99cc3 I don't see any CSV file at all ... My understanding was that the thing works only with CSV files? @JihadMotii-REISys any comments?

[ccaudill]

@ykhadilkar-rei You can only publish services through this CKAN UI (GeoServer) using a CSV. However, harvested tier 3 data and services will still be perfectly valid, though published through a different process (usually with ArcMap) which does not require a CSV file.

[ccaudill]

Can we get an update on what happened to the Geothermal Prospector button?? This is CRITICAL and should take precedence over other things.

[dano-reisys]

@ccaudill We are looking into it, you should hear from us soon.

[dano-reisys]

@ccaudill @smrazgs I was able to add the well log csv file to the datastore:
http://159.87.39.5/dataset/geo-pros-datapusher1
by changing the value of ckan.site_url to 127.0.0.1

Seems like there is some kind of NAT problem going on between the local ip address and public ip address

@159.87.39.5
wget http://159.87.39.5 yields a "Connection time out"
wget 127.0.0.1 works fine

Was this instance set up in Amazon?

[ccaudill]

That was a bad example ... that link did not work. But this one does, no button:
http://final.geothermaldata.org/dataset/alaska-thermal-springs9f940

[dano-reisys]

same problem i think... but the link is valid, i'll have to take a look at it.

[ccaudill]

The service works fine. http://services.azgs.az.gov/ArcGIS/services/aasggeothermal/AKThermalSprings1_8/MapServer/WMSServer?request=GetCapabilities&service=WMS

[dano-reisys]

Should be fixed in instance: http://uat-ngds.reisys.com/dataset/test-wms/resource/5fc4c88d-4a12-4407-bbdf-0c3a029caa45

it will require a code push

[ccaudill]

What fix are we looking at there @dan-olaru-reisys ?

[dano-reisys]

@ccaudill The WMS preview should work now, looking into why the geo-prospector button isn't showing up atm

[dano-reisys]

@ccaudill I was able to get the geo-prospector button to show up for the other datasets, but i am getting a WFS error on maps-stage.nrel.gov (looks like the WFS has no wfsFeatureTypeName... or am I missing something?)

here is the link to the dataset on uat-ngds: http://uat-ngds.reisys.com/dataset/alaska-thermal-springs2

the equivalent of:
http://final.geothermaldata.org/dataset/alaska-thermal-springs9f940

On the other hand, the tire 3 dataset does provide wfsFeatureTypeName for WFS)

http://uat-ngds.reisys.com/dataset/colon-tag-test

[ccaudill]

Perhaps that layer for DEWellLogs you created does open in GeoProsp without error, but you still can't access the WFS features "Data for this layer is not available for download." and the WMS isn't showing on the map in GeosProsp either.

I think programmatically, you will have to add the wfsFeatureTypeName to the URL that NGDS CKAN is providing to NREL. It will not come like that in the harvested data for any resource. As indicated in the the first comment of this issue, the URL must be constructed to include "wfsFeatureTypeName: the name of the wfs feature type exactly as it needs to be specified in the TypeName WFS parameter".

[smrgeoinfo]

The WMS is working as it should; Since there's no parameters for the WFS in the prospector URL, it gives that alert message about can't query the data.

See list of parameters for prospector URL call, https://github.com/ngds/ckanext-ngds/issues/404#issue-29764334. need to include wfsHost and wfsFeatureTypeName. Need to see if there is a WFS and WMS distribution

-<gmd:MD_DigitalTransferOptions id="">
   <gmd:onLine>
      <gmd:CI_OnlineResource>   
          <!-- elements left out in this example code -->
            <gmd:description>
            <gco:CharacterString> parameters:{typeName:"ThermalSpring"}                            
             </gco:CharacterString>
           </gmd:description>

wfsHost: a string (URL escaped) containing the endpoint URL for a WFS that provides feature access to the data displayed by the WMS. The string argument is the part of the URL that precedes the WFS parameters, basically a WFS request with no parameters part (leave out the question mark). OPTIONAL

wfsFeatureTypeName: the name of the wfs feature type exactly as it needs to be specified in the TypeName WFS parameter; the namespace qualifier might or might not be required, I think that might depend on the service implementation. OPTIONAL

see bug fix at
ngds/ckanext-ngds@f269001

[smrgeoinfo]

NOTE ngds/metadata-issues#47. Our metadata isn't following the profile correctly (the issues never end), so to detect if there is a WMS or WFS distribution, probably have to look for 'service=WMS' or 'service=WFS' in the gmd:linkage/gmd:URL:

<gmd:onLine>
   <gmd:CI_OnlineResource>
      <gmd:linkage>
         <gmd:URL>
http://services.azgs.az.gov/ArcGIS/services/aasggeothermal/AKThermalSprings1_8/MapServer/WMSServer?request=GetCapabilities&service=WMS
         </gmd:URL>

[dano-reisys]

@smrazgs
So do we have two issues here?

1. http://uat-ngds.reisys.com/dataset/colon-tag-test

Geothermal prospector button url:
https://maps-stage.nrel.gov/geothermal-prospector/#/?baselayer=6&zoomlevel=3&wmsHost=http://uat-ngds.reisys.com/geoserver-srv/DEWellLog/wms?SERVICE=WMS&&wmsLayerName=WellLog&wfsHost=http://uat-ngds.reisys.com/geoserver-srv/DEWellLog/wfs&wfsFeatureTypeName=DEWellLog:WellLog

@ccaudill wrote:

Perhaps that layer for DEWellLogs you created does open in GeoProsp without error, but you still can't > access the WFS features "Data for this layer is not available for download." and the WMS isn't showing > on the map in GeosProsp either.

*2.

• http://final.geothermaldata.org/dataset/alaska-thermal-springs9f940
• http://uat-ngds.reisys.com/dataset/alaska-thermal-springs2 on our uat server

@smrazgs wrote:

NOTE ngds/metadata-issues#47. Our metadata isn't following the profile correctly (the issues never end), so to detect if there is a WMS or WFS distribution, probably have to look for 'service=WMS' or 'service=WFS' in the gmd:linkage/gmd:URL:

Are you saying that this dataset should not have a geothermal prospector button?
Or perhaps we need to fix package_to_iso.xml so that the iso metadata is generated correctly (with a valid geothermal prospector button?)

[ccaudill]

Neither of those things. You just need to look here in the metadata for the wfsTypeName:
-<gmd:MD_DigitalTransferOptions id="">
gmd:onLine
gmd:CI_OnlineResource

gmd:description
gco:CharacterString parameters:{typeName:"ThermalSpring"}

And make that wfsTypeName be added programmatically to the URL that CKAN NGDS provides to GeoProsp.

[dano-reisys]

@ccaudill what about issue "1." from above?
I talked to @JihadMotii-REISys and it seems like this is the first time he has seen this issue.
Did this work before? Seems like the geothermal prospector link is valid and the preview for WFS is fine on uat-ngds.

any ideas?

[dano-reisys]

@ccaudill @smrazgs

this appears to be the valid geothermal prospector link for the dataset at this url: http://uat-ngds.reisys.com/dataset/alaska-thermal-springs2

Geothermalprospector link:
https://maps-stage.nrel.gov/geothermal-prospector/#/?baselayer=6&zoomlevel=3&wmsHost=http://services.azgs.az.gov/arcgis/services/aasggeothermal/AKThermalSprings1_8/MapServer/WmsServer?&wmsLayerName=ThermalSpring&wfsHost=http://services.azgs.az.gov/arcgis/services/aasggeothermal/AKThermalSprings1_8/MapServer/WmsServer&wfsFeatureTypeName=ThermalSpring

the wfs issue is still there...

ThermalSpring Data for this layer is not available for download.

[ccaudill]

From that link, the AK WMS is showing up exactly as it should in GeosProsp.

[smrgeoinfo]

prospector link seems to be working perfectly at this point. The data are actually for Arkansas, not Delaware. The Metadata for this record is correct, it includes the correct gmd:protocol values in the metadata. I don't know for sure that prospector is supposed to enable download of data from a WFS...

the metadata for the Alaska service
http://uat-ngds.reisys.com/dataset/alaska-thermal-springs2
is incorrect (missing the gmd:protocol parameter), so I'm wondering why the prospector button comes up? see comment above (starts "NOTE ngds/metadata-issues#47.") for workaround for metadata problem.

[ccaudill]

FYI -
No GeoProsp button in new install. Today this server was wiped and reinstalled.
http://test.geothermaldata.org/dataset/vermont-thermal-conductivity

[dano-reisys]

That should be the normal behavior. The fix is in the code, which is not in the v307 rpm

[smrgeoinfo]

can we get it in the rpm soon?

[dano-reisys]

@smrazgs We'll are planning on building a new rpm on 03/31 @FuhuXia

[dano-reisys]

@smrazgs @ccaudill
Looks like the reason why the wfs part of the geothermal prospector link is not there is because harvested datasets like: http://test.geothermaldata.org/dataset/vermont-thermal-conductivity lack the parent_source filed in the wms resource. I am trying to modify the logic, so that it doesn't use the parent_source for such datasets.

questions:

• can there be more than one wms resource per dataset?
• can there be more than one wfs per dataset?

From looking at the code it seems like there only one wms per dataset, but there might be multiple wfs resources because its trying to loop over a list of wfs resources

[smrgeoinfo]

@dan-olaru-reisys -- looks like the problem is that action.py geothermal_prospector_url is not finding the WFS connection information in the CKAN package resource. Lets try and focus the issues on specific problems, and close them when we resolve the problem. I'm closing this thread and starting a new more specific thread. If there are other unresolved issues in this thread, please start a new, specific thread.