update Stamen map tiles

[FuhuXia]

Occasionlly seeing map tile warning messages.

These basemap tiles will no longer be available as of October 31, 2023. 
The site administrator can upgrade to the new map at stamen.com/faq.

Sketch

By October 31, 2023, Find alternatives, or Stamen's migration guide.

[rshewitt]

alternative vector tile providers. most of these are free up to a tile limit. i don't suppose we know how many tiles the widget uses at maximum? if it's low enough we could potentially use one of those alternatives. apparently if we use < 200k tiles per month it's free for non-commercial use ( which we are not?... ) we can use the new service.

[rshewitt]

context on stamen and stadia parternship

[FuhuXia]

Communicated with Stadia Support. It is very nice of them that they privided a catalog.data.gov's monthly usage based on historical data, 400k credits. So it sits right in the free tier 200k and starter plan ($25) 1,000k credits. Looks like a starter plan is what we need.

We can go ahead and set up and free tier account, monitor the usage over the next a few weeks, by the time we reaching the 200k limit, we will have a good idea about the plan choice.

[rshewitt]

pmtiles cloud-optimized single-file archive for vector/raster tiles. there's a section for serverless implementation.

[rshewitt]

mbtile download

[rshewitt]

osm download

[FuhuXia]

Here is the facts I collected so far:

• Everytime a new user hits page /dataset, it loads the zoom level 0 top tile /0/0/0.png. That is one map tile credit consumed. Our catalog has 20k hits by 5k unique users, so roughly 5k credits daily.
• Historical data shows we will need 400k credits / month, per Stadia Support.
• Any user's map search activity will drastically increase tiles usage. A single user with very light activity consumed 75k credits during the test.

Here is what I suggest:

• Go live with the test account. We can calculate the real usage.
• @rshewitt is trying self tile serving research with pmtiles.
• We can take map search function down for a while if things did not worked out.

NEW 💡
With the assumption that vast majority of tile request is /0/0/0.png, we could make it a static image hosted on our own server, customize ckanext-spatial/leaflet to only send other tile request to the tile server. This will keep our tile usage well under free plan quota.
[UPDATE] Stadia does not allow any use of tile caching or downloading. https://stadiamaps.com/terms-of-service. So this is a bad 💡.

[rshewitt]

here's the 4 options we have to move forward with creating our own tiling server

• replace the current mapping library in ckan spatial ( or use an appropriate leaflet plugin ) with something which can ingest vector tiles
• or if we want to continue using leaflet in ckan spatial as-is then we need to create pre-rendered stylized raster images to deliver
• or get our hands on a planet file of stylized images ( a basemap )
• render images on-the-fly from a vector tile using something like mapnik

[rshewitt]

pausing on my work in favor of purchasing/using a license from a tile provider for the time being. if costs are high we'll revisit deploying our own tiling service.

[nickumia]

This is a very strange issue... I don't know if I'll be adding anything useful to this ticket, but it seems relevant to post these links:

• Search of stamen in ckanext-spatial
• stamen seems to be the default if nothing else is selected... but I can't find the actual default in a ckan.ini or a code file anywhere...
• There is configuration in ckanext-spatial to change the url that points to tiles.
• And they do reference a list of providers (however thin).
• There's also this file.. which has relevant information and generates into this more user-friendly page.
• List of tile servers? (not sure paid vs. free)
• More lists of servers? (not sure paid vs. free)

With all of this said... I still don't know what the real problem is (other than the server not supporting free hosting of picture data?) It feel like there's more options than listed in the ticket though, hence my attempt 🤷‍♀️

[FuhuXia]

@nickumia FYI, this is the upstream ticket for this same issue.

[FuhuXia]

Summarizing slack discussion, we have the following options.

1. Continue using Stamen/Stadia map tile server.
   Pros: Nice looking map; 18 zoom levels; Good support.
   Cons: Paid service. $25 / mo is cheap but paper work is a hassle; Risk of Unexpected bill spike; Caching prohibited.
2. Use basemap.nationalmap.gov map tile server.
   Pros: Free; .gov
   Cons: 8 zoom levels (city level world-wide), 16 zoom levels in the U.S.
3. Use openstreetmap.org map tile server.
   Pros: No restriction for creative and unexpected uses; 19 zoom levels.
   Cons: Need to cache the tiles to avoid heavy usage.

[FuhuXia]

After discussion, the team decided to go with option 2 using basemap.nationalmap.gov in favor of its .gov nature. We have also implemented option 3 and deployed to development space to prove it is a viable solution.

Option 2:

ckanext.spatial.common_map.custom_url = https://basemap.nationalmap.gov/arcgis/rest/services/USGSTopo/MapServer/tile/{z}/{y}/{x}

Option 3:

ckanext.spatial.common_map.custom_url =  '/maptiles/{z}/{x}/{y}.png';

# add this block to the nginx-common.conf
location /maptiles {
    rewrite ^/maptiles/(.*)/(.*)/(.*).png$ /$1/$2/$3.png break;
    proxy_redirect off;
    proxy_pass https://tile.openstreetmap.org/;
}
 

[FuhuXia]

We are live with option 3.

[FuhuXia]

CloudFront stats show the map tile caching is working as expected, caching averagely 99.4% map tile requests, sending less than 100 requests to original openstreetmap server in the past 7 days.

object	Requests	Hits	Hits %
/maptiles/3/1/2.png	85,246	84,234	98.81%
/maptiles/3/2/2.png	83,330	82,779	99.34%
/maptiles/3/2/3.png	82,019	81,811	99.75%
/maptiles/3/1/3.png	81,702	81,451	99.69%