Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update mosaic tile generation notebook #12

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

KlaasH
Copy link

@KlaasH KlaasH commented Jul 16, 2024

Proposed Changes:

Applies some changes to mosaic.ipynb for adjustments I made in the course of generating static tiles for WikiWatershed/model-my-watershed#3635.
Most notably, the last section now

  • takes a list of zoom levels and loops over them
  • uses asyncio.Semaphore, via a decorator, to limit how many tile requests will run simultaneously.

I also copied a few variables into the final cell (i.e. COLORMAP and tile_url_template) so it can be run independently of the earlier cells.

Related Issue(s):

Notes:

  • Zoomlevels 0-3 can't be generated with the Lambda tiler because individual tiles take longer than 15 minutes. For those, I used a modified version of the notebook code pointed at a local titiler-mosaicjson server (and limited to concurrency 1. see my comment on #9 for notes on running titiler-mosaicjson locally).
  • The tile downloading cell mostly works if you pull it into a script to run it from the command line, but the
    results = await asyncio.get_event_loop().create_task(download_tiles())
    
    line needs to be changed to
    result = asyncio.run(download_tiles())
    
  • I'm not sure this is the strategy to pursue long-term. Given that the low-zoomlevel tiles can't be generated with Lambda, it's never going to be possible to wrap this up into a simple script that can just be run against the existing tiler endpoint. And if it's going to require local processing and more-extensive supervision anyway, it might make sense to just switch to a "download the files and make a tile pyramid with GDAL tools" strategy. But that's a question for issue AWS 2-6: Evaluate Adding Hybrid Tiling Endpoint #10.

PR Checklist:

  • I have added my changes to the CHANGELOG or a CHANGELOG entry is not required.

Applies some changes to mosaic.ipynb for adjustments I made in the course of
generating static tiles for WikiWatershed/model-my-watershed#3635.
The last section now
- takes a list of zoom levels and loops over them
- uses asyncio.Semaphore, via a decorator, to limit how many tile requests will
  run simultaneously.

I also copied a few variables into the final cell (i.e. COLORMAP and tile_url_template)
so it can be run independently of the of the earlier cells.
@KlaasH
Copy link
Author

KlaasH commented Oct 30, 2024

I added the download_tiles.py script I had made back when I was generating tiles into the scripts/ directory on this branch. It's the same logic as in the notebook, but having it as a script rather than a notebook is kind of nice for something that runs a long time and that you'll potentially want to run several iterations of.

As another note related to tile generation, I described running TiTiler-Mosaicjson locally on issue #9, but the instructions were a bit muddled. Here's a simpler version (without the bits about the unsuccessful attempt to use a local DynamoDB):

Clone and install:

git clone https://github.com/element84/titiler-mosaicjson.git
cd titiler-mosaicjson
python3 -m venv .venv
source .venv/bin/activate
pip install uvicorn
pip install -e src/titiler/core -e src/titiler/extensions -e src/titiler/mosaic -e src/titiler/application

Start the local server, with a very long timeout:

MOSAIC_TILE_TIMEOUT=5000 uvicorn --port 8444 titiler.application.main:app --reload

You should now have a server running at http://127.0.0.1:8444

Note that the local server, with its default file backend, is not very good at concurrency. It seems to get sad and just hang if you ask too much of it. So one tile at a time is probably the right number. Which means it's better to use the deployed endpoint for any tiles (z4 or higher, I think. or maybe z3 is OK too) that can be generated without exceeding the Lambda timeout.

Adds a script that reproduces the section of the notebook where it downloads
all the tiles, slightly modified to run on the command line.
@KlaasH KlaasH force-pushed the kjh/update-mosaic-notebook branch from 3875120 to e8d8a0a Compare October 30, 2024 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants