Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WMS Sync Bot #1031

Merged
merged 1 commit into from
Feb 1, 2021
Merged

WMS Sync Bot #1031

merged 1 commit into from
Feb 1, 2021

Conversation

github-actions[bot]
Copy link
Contributor

Update url and available_projections of WMS sources using scripts/sync_wms.py. Sources are tested with scripts/check.py before PR is created. Delete .github/workflows/sync_wms.yml to stop.

@rbuffat
Copy link
Collaborator

rbuffat commented Jan 24, 2021

@grischard @don-vip I suspect that the bot should not have updated "sources/europe/lv/lv_orthophoto_cycle6_2016-2018.geojson", I will look into this. The last time I checked this wms server does not follow the axis ordering of wms 1.3.0 for epsg:4326. I will look into this.

@rbuffat
Copy link
Collaborator

rbuffat commented Jan 24, 2021

I found the issue, there should be a return following this block:

# Only sources are updated where the new query returns the same image
if not image_hash == new_image_hash:
error_original_img_messages = "\n\t".join(original_img_messages)
error_new_img_messages = "\n\t".join(new_img_messages)
logging.info(
f"Image hash not the same for: {filename}: '{image_hash}' vs '{new_image_hash}' \n\t{error_original_img_messages} \n\t{error_new_img_messages}"
)

This issue is that some WMS server do not support the differnt axis ordering dependent on the CRS of WMS 1.3.0. Unfortunately, there are both variants present (server where 1.3.0 returns the correct extent and not 1.1.1 and vice versa). The good thing is that based on my experience I expect only a small amount of servers that are affected by this.

The intended behavior was to trust that in the existing configuration the server works correctly for EPGS:4326 and only to upgrade if the upgraded version returns the same image.

Currently, I think it would be better to determine the correct working WMS version based on the source. E.g. we can compare the EPSG:3857 against the EPSG:4326 image. Additionally, the EPSG:4326 image can be compared to a WMS 1.3.0/EPSG:4326 image with reversed axis ordering as well as a WMS<1.3.0/EPSG:4326 image. I think this should be enough to determine the best WMS version, but I need probably some days until I have time to implement this.

@don-vip
Copy link
Collaborator

don-vip commented Jan 24, 2021

The intended behavior was to trust

That's the problem with dealing with worldwide servers, you cannot trust all of them :)

@rbuffat
Copy link
Collaborator

rbuffat commented Jan 30, 2021

"Trust" is probably the wrong word. The script should not break something that is not already broken and in the best case repair if something is broken.

Actually, there is nothing wrong with WMS 1.3.0 for "sources/europe/lv/lv_orthophoto_cycle6_2016-2018.geojson", but there is a bug in iD: openstreetmap/iD#8322

As the script by default tests with EPSG:4326 and does nothing when the test image returns an empty image (which is very likely the case when a WMS server does not support the axis ordering of WMS 1.3.0), it seems as there were no sources updated. However, there seem to be sources that were added that have troubles with EPSG:4326. #1034 logs some information about these sources:

The following sources have different image hashes for EPSG:3857 and EPSG:4326 images. I need to go over them manually to identify if there is actually something wrong or the differences are just from reprojection / other quirks

WARNING:root:sources/europe/cz/CzechCUZKKM.geojson: (other) ImageHash for EPSG:3857 and EPSG:4326 not similiar: 1008000700070701 - 10080403060e0000: 10
WARNING:root:sources/europe/no/MiljdirektoratetProtectedAreasoverlay.geojson: (other) ImageHash for EPSG:3857 and EPSG:4326 not similiar: ffffffffffffff7f - a2ff7fffffffff80: 14
WARNING:root:sources/europe/at/TirisContourlines.geojson: (elevation) ImageHash for EPSG:3857 and EPSG:4326 not similiar: c066ffbed8e0c4c0 - c060cf1f7f7366ce: 22
WARNING:root:sources/europe/de/Maps4BW.geojson: (map) ImageHash for EPSG:3857 and EPSG:4326 not similiar: 030307a2852fa791 - 0f070280850737f1: 13
WARNING:root:sources/europe/no/kartverket-dom-skygge.geojson: (elevation) ImageHash for EPSG:3857 and EPSG:4326 not similiar: f0b19d1d0d0c1d7f - e0b0a51d8d021979: 12
WARNING:root:sources/europe/no/kartverket-dtm-skygge.geojson: (elevation) ImageHash for EPSG:3857 and EPSG:4326 not similiar: b0b1111c1f0d1d6f - e0f0203c1f073969: 14

Furthermore, some photo sources report an empty image, which is also strange and could indicate a server that does not support axis order, but again I need to look into this:

WARNING:root:sources/europe/pl/GliwiceOrthophotomap2020(aerialimage).geojson has category photo but image hash is 0000000000000000
WARNING:root:sources/europe/pl/GliwiceOrthophotomap2018(aerialimage).geojson has category historicphoto but image hash is 0000000000000000
WARNING:root:sources/south-america/br/Rio_2015.geojson has category photo but image hash is 0000000000000000
WARNING:root:sources/europe/ch/Stadt_Zurich_Orthophoto2013_wms.geojson has category photo but image hash is 0000000000000000
WARNING:root:sources/europe/pl/GliwiceOrthophotomap2017(aerialimage).geojson has category historicphoto but image hash is 0000000000000000
WARNING:root:sources/europe/pl/GliwiceOrthophotomap2013(aerialimage).geojson has category historicphoto but image hash is 0000000000000000
WARNING:root:sources/world/eox.at-2018-cloudless.geojson has category photo but image hash is 0000000000000000

@github-actions github-actions bot force-pushed the create-pull-request/patch branch from 073717b to 4bbb975 Compare January 31, 2021 00:53
@rbuffat
Copy link
Collaborator

rbuffat commented Jan 31, 2021

WARNING:root:sources/europe/cz/CzechCUZKKM.geojson: (other) ImageHash for EPSG:3857 and EPSG:4326 not similiar: 1008000700070701 - 10080403060e0000: 10
WARNING:root:sources/europe/no/MiljdirektoratetProtectedAreasoverlay.geojson: (other) ImageHash for EPSG:3857 and EPSG:4326 not similiar: ffffffffffffff7f - a2ff7fffffffff80: 14
WARNING:root:sources/europe/at/TirisContourlines.geojson: (elevation) ImageHash for EPSG:3857 and EPSG:4326 not similiar: c066ffbed8e0c4c0 - c060cf1f7f7366ce: 22
WARNING:root:sources/europe/de/Maps4BW.geojson: (map) ImageHash for EPSG:3857 and EPSG:4326 not similiar: 030307a2852fa791 - 0f070280850737f1: 13
WARNING:root:sources/europe/no/kartverket-dom-skygge.geojson: (elevation) ImageHash for EPSG:3857 and EPSG:4326 not similiar: f0b19d1d0d0c1d7f - e0b0a51d8d021979: 12
WARNING:root:sources/europe/no/kartverket-dtm-skygge.geojson: (elevation) ImageHash for EPSG:3857 and EPSG:4326 not similiar: b0b1111c1f0d1d6f - e0f0203c1f073969: 14

They all seem to be ok. It looks as the maps are differently rendered depending on the projections and the DEM seem to have different shading.

WARNING:root:sources/europe/cz/CzechCUZKKM.geojson: (other) ImageHash for EPSG:3857 and EPSG:4326 not similiar: 1008000700070701 - 10080403060e0000: 10:
        Image URL: https://wms.cuzk.cz/wms.asp?LAYERS=parcelni_cisla_i,obrazy_parcel_i,RST_KMD_I,hranice_parcel_i,DEF_BUDOVY,RST_KN_I,dalsi_p_mapy_i,prehledka_kat_prac,prehledka_kat_uz,prehledka_kraju-linie&STYLES=&FORMAT=image/png&TRANSPARENT=TRUE&CRS=EPSG:3857&WIDTH=256&HEIGHT=256&BBOX=1729311.3279238273,6413372.421239428,1731757.3128289531,6415818.406144551&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap
        ImageHash: 1008000700070701
        Image URL: https://wms.cuzk.cz/wms.asp?LAYERS=parcelni_cisla_i,obrazy_parcel_i,RST_KMD_I,hranice_parcel_i,DEF_BUDOVY,RST_KN_I,dalsi_p_mapy_i,prehledka_kat_prac,prehledka_kat_uz,prehledka_kraju-linie&STYLES=&FORMAT=image/png&TRANSPARENT=TRUE&CRS=EPSG:4326&WIDTH=256&HEIGHT=256&BBOX=49.809631563563094,15.53466796875,49.823809085132496,15.556640625&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap
        ImageHash: 10080403060e0000
WARNING:root:sources/europe/no/MiljdirektoratetProtectedAreasoverlay.geojson: (other) ImageHash for EPSG:3857 and EPSG:4326 not similiar: ffffffffffffff7f - a2ff7fffffffff80: 14:
        Image URL: https://kart.miljodirektoratet.no/arcgis/services/vern/MapServer/WmsServer?LAYERS=naturvern_klasser_omrade&STYLES=&FORMAT=image/png&TRANSPARENT=TRUE&CRS=EPSG:3857&WIDTH=256&HEIGHT=256&BBOX=2233184.2183797094,14301673.740269622,2235630.203284835,14304119.725174747&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap
        ImageHash: ffffffffffffff7f
        Image URL: https://kart.miljodirektoratet.no/arcgis/services/vern/MapServer/WmsServer?LAYERS=naturvern_klasser_omrade&STYLES=&FORMAT=image/png&TRANSPARENT=TRUE&CRS=EPSG:4326&WIDTH=256&HEIGHT=256&BBOX=77.87419904054089,20.06103515625,77.87881372624744,20.0830078125&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap
        ImageHash: a2ff7fffffffff80
WARNING:root:sources/europe/at/TirisContourlines.geojson: (elevation) ImageHash for EPSG:3857 and EPSG:4326 not similiar: c066ffbed8e0c4c0 - c060cf1f7f7366ce: 22:
        Image URL: https://gis.tirol.gv.at/arcgis/services/Service_Public/terrain/MapServer/WmsServer?LAYERS=Hoehenschichtlinien_20m&STYLES=&FORMAT=image/png&TRANSPARENT=TRUE&CRS=EPSG:3857&WIDTH=256&HEIGHT=256&BBOX=1241337.3393512624,5982879.077937315,1242560.3318038252,5984102.070389879&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap
        ImageHash: c066ffbed8e0c4c0
        Image URL: https://gis.tirol.gv.at/arcgis/services/Service_Public/terrain/MapServer/WmsServer?LAYERS=Hoehenschichtlinien_20m&STYLES=&FORMAT=image/png&TRANSPARENT=TRUE&CRS=EPSG:4326&WIDTH=256&HEIGHT=256&BBOX=47.24940695788844,11.151123046875,47.25686404408872,11.162109375&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap
        ImageHash: c060cf1f7f7366ce
WARNING:root:sources/europe/no/kartverket-dom-skygge.geojson: (elevation) ImageHash for EPSG:3857 and EPSG:4326 not similiar: f0b19d1d0d0c1d7f - e0b0a51d8d021979: 12:
        Image URL: https://wms.geonorge.no/skwms1/wms.hoyde-dom_somlos_skyggerelieff?LAYERS=las_dom_skyggerelieff_somlos&STYLES=&FORMAT=image/jpeg&CRS=EPSG:3857&WIDTH=256&HEIGHT=256&BBOX=1286588.0600960867,9483083.477172108,1289034.0450012123,9485529.462077234&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap
        ImageHash: f0b19d1d0d0c1d7f
        Image URL: https://wms.geonorge.no/skwms1/wms.hoyde-dom_somlos_skyggerelieff?LAYERS=las_dom_skyggerelieff_somlos&STYLES=&FORMAT=image/jpeg&CRS=EPSG:4326&WIDTH=256&HEIGHT=256&BBOX=64.52009732015117,11.5576171875,64.52954819942194,11.57958984375&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap
        ImageHash: e0b0a51d8d021979
WARNING:root:sources/europe/no/kartverket-dtm-skygge.geojson: (elevation) ImageHash for EPSG:3857 and EPSG:4326 not similiar: b0b1111c1f0d1d6f - e0f0203c1f073969: 14:
        Image URL: https://wms.geonorge.no/skwms1/wms.hoyde-dtm_somlos_skyggerelieff?LAYERS=las_dtm_skyggerelieff_somlos&STYLES=&FORMAT=image/jpeg&CRS=EPSG:3857&WIDTH=256&HEIGHT=256&BBOX=1286588.0600960867,9483083.477172108,1289034.0450012123,9485529.462077234&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap
        ImageHash: b0b1111c1f0d1d6f
        Image URL: https://wms.geonorge.no/skwms1/wms.hoyde-dtm_somlos_skyggerelieff?LAYERS=las_dtm_skyggerelieff_somlos&STYLES=&FORMAT=image/jpeg&CRS=EPSG:4326&WIDTH=256&HEIGHT=256&BBOX=64.52009732015117,11.5576171875,64.52954819942194,11.57958984375&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap
        ImageHash: e0f0203c1f073969
WARNING:root:sources/europe/de/Maps4BW.geojson: (map) ImageHash for EPSG:3857 and EPSG:4326 not similiar: 030307a2852fa791 - 0f070280850737f1: 13:
        Image URL: https://owsproxy.lgl-bw.de/owsproxy/ows/WMS_Maps4BW?LAYERS=webatlasde&STYLES=&FORMAT=image/png&TRANSPARENT=TRUE&CRS=EPSG:3857&WIDTH=256&HEIGHT=256&BBOX=1007745.7809117635,6215247.643924252,1010191.7658168894,6217693.628829373&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap
        Could not download image in try 0: Server disconnected
        ImageHash: 030307a2852fa791
        Image URL: https://owsproxy.lgl-bw.de/owsproxy/ows/WMS_Maps4BW?LAYERS=webatlasde&STYLES=&FORMAT=image/png&TRANSPARENT=TRUE&CRS=EPSG:4326&WIDTH=256&HEIGHT=256&BBOX=48.64742780553353,9.052734375,48.66194284607007,9.07470703125&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap
        Could not download image in try 0: Server disconnected
        ImageHash: 0f070280850737f1

@github-actions github-actions bot force-pushed the create-pull-request/patch branch from 4bbb975 to cbc224d Compare February 1, 2021 01:14
@rbuffat
Copy link
Collaborator

rbuffat commented Feb 1, 2021

All has category photo but image hash is 0000000000000000 sources except sources/europe/ch/Stadt_Zurich_Orthophoto2013_wms.geojson seem to be resolved with #1038

The WMS server of sources/europe/ch/Stadt_Zurich_Orthophoto2013_wms.geojson does not follow the axis ordering of WMS 1.3.0, but unfortunately it seems to be a buggy QGIS server where WMS 1.1.1 seems to be broken: qgis/QGIS#41051

WMS 1.3.0 query with correct axis ordering:
https://www.ogd.stadt-zuerich.ch/wms/geoportal/Orthofoto_2013_Stadt_Zuerich___Fruehling?FORMAT=image/jpeg&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap&LAYERS=OP_2013_STZH.tif&STYLES=&CRS=EPSG:4326&WIDTH=256&HEIGHT=256&BBOX=47.36859434521338,8.525390625,47.3834738721015,8.54736328125

WMS 1.3.0 query with x,y axis ordering:
https://www.ogd.stadt-zuerich.ch/wms/geoportal/Orthofoto_2013_Stadt_Zuerich___Fruehling?FORMAT=image/jpeg&VERSION=1.3.0&SERVICE=WMS&REQUEST=GetMap&LAYERS=OP_2013_STZH.tif&STYLES=&CRS=EPSG:4326&WIDTH=256&HEIGHT=256&BBOX=8.525390625,47.36859434521338,8.54736328125,47.3834738721015

WMS 1.1.1 query with x,y axis ordering:
https://www.ogd.stadt-zuerich.ch/wms/geoportal/Orthofoto_2013_Stadt_Zuerich___Fruehling?FORMAT=image/jpeg&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetMap&LAYERS=OP_2013_STZH.tif&STYLES=&SRS=EPSG:4326&WIDTH=256&HEIGHT=256&BBOX=8.525390625,47.36859434521338,8.54736328125,47.3834738721015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants