Create mosaic from S3 requester-pays tiffs

[kylebarron]

It appears that currently the only way to create a mosaic is with http links. Would it be possible to add functionality to create a mosaic through s3:// URIs? For requester pays buckets like NAIP/CBERS, I don't know if it's possible to use an http url to fetch the tiffs.

I did attempt

echo s3://naip-visualization/wa/2015/100cm/rgb/45122/m_4512229_ne_10_1_20150728.tif | cogeo-mosaic create - -o test_naip.json

but that threw an exception ValueError: max() arg is an empty sequence.

[vincentsarago]

👋 @kylebarron
if the bucket in in requester-pays you'll need to set AWS_REQUEST_PAYER="requester" in your environement variable.

Note: the mosaic won't work in the regular cogeo-mosaic-tiler stac because we don't support requester-pays bucket

[kylebarron]

Ah, that did work once I set AWS_REQUEST_PAYER="requester"! Thanks!

And yeah, I'm assuming I'll have to fork/edit some part of cogeo-mosaic-tiler for it to work with requester-pays. Regardless, great work on these projects!

[vincentsarago]

@kylebarron 😄

so to support Requester-pays bucket you'll need to add it in https://github.com/developmentseed/cogeo-mosaic-tiler/blob/master/serverless.yml#L44

[kylebarron]

Is there a way to print rasterio errors to stdout? Setting the AWS_REQUEST_PAYER env variable worked on my local machine, but wasn't on an AWS EC2 instance. Doing

export AWS_REQUEST_PAYER="requester"
echo s3://naip-visualization/wa/2015/100cm/rgb/45122/m_4512229_ne_10_1_20150728.tif | cogeo-mosaic create - -o test_naip.json

gave the following opaque error that I've found results from no downloads succeeding

Traceback (most recent call last):
  File "/home/ubuntu/local/miniconda3/bin/cogeo-mosaic", line 8, in <module>
    sys.exit(cogeo_cli())
  File "/home/ubuntu/local/miniconda3/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/ubuntu/local/miniconda3/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/ubuntu/local/miniconda3/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ubuntu/local/miniconda3/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ubuntu/local/miniconda3/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/ubuntu/local/miniconda3/lib/python3.7/site-packages/cogeo_mosaic/scripts/cli.py", line 81, in create
    quiet=quiet,
  File "/home/ubuntu/local/miniconda3/lib/python3.7/site-packages/cogeo_mosaic/utils.py", line 248, in create_mosaic
    minzoom = max(minzoom)
ValueError: max() arg is an empty sequence

After a bit more research, I discovered that rio info was itself failing, and after setting

export CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

(from cogeotiff/rio-tiler#19, rasterio/rasterio#942), cogeo-mosaic create now works. But because rasterio.open inside cogeo-mosaic was failing silently, there was no good way to know that was the issue. It might be helpful if those errors could be propagated to stdout.

[vincentsarago]

oh thanks @kylebarron,
Sadly the rasterio calls are made in multithreads, I'm trying to log the error here https://github.com/developmentseed/cogeo-mosaic/blob/master/cogeo_mosaic/utils.py#L65 but I'm not fully understand why we don't see it.

I'm sorry you had to spent some time debuging this

[kylebarron]

I think it's an issue with logging. I haven't used the module too much, but this says that if you use logger.info, you must change the level and add a handler. So it seems that just setting logger.setLevel(logging.INFO) isn't enough, and that's why we don't see the error messages.

I haven't researched how to add a handler, but changing

cogeo-mosaic/cogeo_mosaic/utils.py

Line 65 in f8bc8e6

logger.info(str(err))

to

            logger.warning(str(err))

prints Access Denied when trying to access a requester pays S3 bucket without setting AWS_REQUEST_PAYER:

> echo s3://naip-visualization/wa/2015/100cm/rgb/45122/m_4512229_ne_10_1_20150728.tif | cogeo-mosaic create - -o test_naip.json

Get files footprint
  [####################################]  100%
Access Denied
Traceback (most recent call last):
  File "/Users/kyle/local/anaconda3/bin/cogeo-mosaic", line 11, in <module>
    load_entry_point('cogeo-mosaic', 'console_scripts', 'cogeo-mosaic')()
  File "/Users/kyle/local/anaconda3/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/Users/kyle/local/anaconda3/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/Users/kyle/local/anaconda3/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/kyle/local/anaconda3/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/kyle/local/anaconda3/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/Users/kyle/github/mapping/naip-tiled/cogeo-mosaic/cogeo_mosaic/scripts/cli.py", line 81, in create
    quiet=quiet,
  File "/Users/kyle/github/mapping/naip-tiled/cogeo-mosaic/cogeo_mosaic/utils.py", line 248, in create_mosaic
    minzoom = max(minzoom)
ValueError: max() arg is an empty sequence