rio-tiler 2.0

[vincentsarago]

rio-tiler Version 2 🎉

closes #152 #144 #139 #124 #105 #150 #145

[vincentsarago]

This will be a major release with a lot of breaking changes. I'll write up an issue explaining how we will release (alpha, beta ... ) and how to switch to this new version.

[kylebarron]

Is there anything else you need help with on 2.0?

I see 2.0 brings support for partial reads, so I'm thinking of trying to adapt rio-tiler-mosaic/cogeo-mosaic-tiler to work with rio-tiler 2.0 and see how it goes and if I can improve performance.

[vincentsarago]

thanks @kylebarron , really appreciated.

I don't think it will change much to rio-tiler-mosaic because it takes .tile function as input. but in cogeo-mosaic-tiler we will need to change from rio_tiler.main import tile as cogeoTiler by from rio_tiler.io.cogeo import tile as cogeoTiler`.

to support part in rio-tiler-mosaic we could just add a mosaic_part function that takes rio-tiler.reader.part like function as input (there is only one right now but we might add another one to support expression`.

to support change in cogeo-mosaic-tiler or cogeo-tiler I'll also need to update https://github.com/remotePixel/cogeo-layer

[vincentsarago]

following @kylebarron comment. I guess we should also make this optional, and only use transform_bounds if bounds_crs != dst_crs ?

[kylebarron]

Actually, to be clearer, I found that when bounds_crs == dst_crs, it only took ~1ms, it's only when bounds_crs != dst_crs that it took 85ms.

In current _tile_read, transform_bounds is called twice, where the second is reprojecting to mercator coordinates, and you can see it's only the second that's slow

[vincentsarago]

Excellent, so we can keep it as it is then.

Can I ask how are you profiling the function?

[kylebarron]

I'm using AWS X-Ray, and using their Python SDK to define manual timing blocks.

I'm timing parts of the underlying libraries by shadowing their function definitions, i.e.

# local import
from cogeo_mosaic_tiler.temp import (
    fetch_and_find_assets
)

and have a file temp.py locally that's essentially the same as the original file, but with

xray_recorder.begin_subsegment('name')
...
xray_recorder.end_subsegment()

interspersed throughout. E.g.

def _aws_get_data(key, bucket, client: boto3_session.client = None) -> BinaryIO:
    if not client:
        xray_recorder.begin_subsegment('make session')
        session = boto3_session()
        xray_recorder.end_subsegment()

        xray_recorder.begin_subsegment('make client')
        client = session.client("s3")
        xray_recorder.end_subsegment()

    xray_recorder.begin_subsegment('client.get_object')
    response = client.get_object(Bucket=bucket, Key=key)
    xray_recorder.end_subsegment()
    return response["Body"].read()

Then make a new lambda package, deploy, pan the web map a little, and go into the AWS console to see the trace output.

I have a couple screenshots in issues here and here (the latter didn't profile within mosaic_tiler)

[kylebarron]

Cool, I'll see how it goes.

As an aside, for a newcomer I've found it a little confusing keeping track of what each repository does. Each project handles a subtly distinct need, but they're so intertwined that it's hard to keep them straight.

rio-tiler
rio-cogeo
rio-tiler-mosaic
cogeo-mosaic
cogeo-mosaic-tiler

I don't really have a suggestion for how things could be clearer, but just something to keep in mind.

[vincentsarago]

thanks @kylebarron

there are a lot of repo (split in mainly 2 orgs) which could hardly be simplified (maybe we should move everything to cogeotiff ?)

I'll think about this any maybe create an awesome-cog repo/doc ;-)

name	org	description
rio-tiler	cogeotiff	Core module to read data from a COG
rio-cogeo	cogeotiff	CLI + python module to easily create COG
rio-tiler-mosaic	cogeotiff	rio-tiler plugin to handler multiple overlapping dataset (was made as a separate module to keep rio-tiler simple)
rio-tiler-mvt	cogeotiff	rio-tiler plugin to create vector tiles from a numpy array
rio-tiler-crs	cogeotiff	rio-tiler plugin to create tile in other CRS
---	---	---
cogeo-mosaic	devseed	CLI+python module to create MosaicJSON (not linked directly to rio-tiler nor rio-cogeo)
cogeo-mosaic-tiler	devseed	AWS stack - Dynamic tiler for mosaics.
cogeo-tiler	devseed	AWS stack - Dynamic tiler for simple COG.
titiler	devseed	FastAPI stack for dynamic tiling
---	---	---
rio-viz	devseed	visualize COG in browser
---	---	---
cogeo-watchbot	devseed	AWS stack - Create COG + MosaicJSON at scale
cogeo-watchbot-light	devseed	AWS stack - Create COG at scale
cogeo-watchbot-fargate	devseed	AWS stack - Create COG at scale using ECS fargate

[kylebarron]

maybe create an awesome-cog

I think that would be awesome and really helpful, just to have a description of all the repositories in one place.

If you're going to make an awesome-cog, I'd also add rio-glui, and there are probably more Mapbox COG repositories

[kylebarron]

I think you should take out deprecations before the final 2.0 release

[vincentsarago]

yes that's the plan. If we do a Beta release, all the depreciation warning will go