-
-
Notifications
You must be signed in to change notification settings - Fork 201
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Caching #96
Comments
It's been a point of discussions many times, though we have not decided how we want to proceed. Our current suggestion is to use a CDN or caching layer in front of tegola. This keeps tegola focused on slicing tiles from data providers. We have played with the idea of supporting a local disk cache using the the z/x/y.pbf naming convention as well as "seeding tools" (#64) which would precompute all the tiles with the same convention. The tiles could then be persisted to somewhere like S3. Maybe our discussion will turn into a fun project for you to give Go a try. What would be your desired caching solution? |
Local disk cache using the the z/x/y.pbf schema would be perfect (at least for my project). we have some huge datasets on PostGis. tegola has to do the heavy processing and produce the tiles (pbf files) each user request. The beauty is that tegola takes care of the simplification and does an awesome job with that. Much better that a simple ST_simplify on postgis, for instance. |
@andreschardong got it. What are your thoughts on clearing out the cache? We go back and forth on this one. A couple options we have considered:
Also if you're interested in taking a stab at the cache functionality I'm happy to bounce around ideas on the design. |
@ARolek, regarding the options: Some functionalities that could be added (based on geoserver):
|
I think caching is essential when rendering from an updating data like OSM. There are two strategies in production: pre-rendering the entire planet into the cache, or pre-rendering some parts into cache and rendering the rest on-demand. When pre-rendering everything, you need to be able to seed the cache and then as osm2pgsql/etc reports tile expiry, re-render them. If pre-rendering some and live-rendering others, the standard way is what mod_tile does with the idea of a "dirty" tile, one which is not flushed from cache, but kept around and not re-rendered until re-requested. It's worth looking at the logic for Mapzen's tilezen, mod_tile/renderd or tirex/renderd, and Kartotherian. t-rex might also be worth a look, but I haven't investigated what they're doing yet. |
Is there any documentation on how to setup some caching with Tegola and Nginx for instance? |
@bagage are you using nginx Content Caching? An alternative suggestion would be to use a CDN in front of tegola to handle the caching. I'm not quite sure what you mean by:
Are you trying to find the tiles that intersect with a lat / lng bounding box? |
@ARolek yes I'm using Content Caching. It does the job well, except when I wish to force refreshing a given zone.
Yes, I'd like to get the list of
But most of these are not existing/empty files (because nothing is located at this position), and sending too much PURGE requests is not working well. |
@bagage I'm trying to think what's the best way to pull this off. I think you're going to need to do the following steps:
You can see the math for converting This is an interesting request around caching. I'm going to think on it a bit to see if we should support "area purging". I can see the application, basically purge the cache when a subset of data has been modified. I don't know if exposing this via HTTP is a good idea but maybe as a sub command that could be issued, i.e. |
@ARolek Thanks for the pointers. That's actually how I implemented it, but it results in way too much PURGE requests (around 28K requests for zoom 18 for a "screen size bounding box"). Maybe my code is wrong though, will recheck it. Not sure what the ideal solution could be here, exposing a purge area via HTTP could be useful imo (even if in my case I will have access to |
@bagage yeah something has to be off with your script. What did you write it in? I can try to take a quick look if you want to post it up on a gist. |
@bagage great solution! When I get working on caching support I will design for bounding box & zoom area purging too. Thank you for the feedback! |
…strategy. the cacher interface needs to be thought about a bit more. #96
…ntation. extra error checking and durability for the file cache. #96
A file cache has been implemented in the release candidate for v0.4.0. The fist implementation is a file cache but we're positioned to support other caching backends as necessary. Seeding and purging are coming up next. I'm going to address purging in another issue so we can close this one. |
Is caching pbf on our plans? I'm wondering if I could help. Never worked with Go before, however could try.
The text was updated successfully, but these errors were encountered: