Does rioxarray use internal GDAL config parameters?

[Greveley]

Hi,

I have a lot of data stored on s3 and while processing it using dask, xarray and rioxarray I was noticing that it was having some trouble reading the the data from S3. It looked like the connection was dropping intermittently between S3 and my instance that I was working on.

My question is related to whether rioxarray uses internal GDAL config parameters that can be set within the script to be used by rioxarray e.g. GDAL_HTTP_MAX_RETRY (from the below link):

https://gdal.org/user/configoptions.html#config-GDAL_HTTP_MAX_RETRY

I'm just curious as to whether rioxarray is able to use some of the environment variables internal within GDAL to retry on http failures such as the one I think I'm seeing.

Thanks for all the help.

[snowman2]

At it's core, rioxarray is powered by GDAL through the rasterio python bindings. So, the config should work.

[artttt]

Yes..... But
If you are useing a dask cluster you need to make sure that the gdal environment is set up on the dask workers.

[Greveley]

Any idea how to do that? It would be very useful to know as I'm fairly frequently using a LocalCluster to do the processing

[artttt]

It's more of an issue with a cluster across many machines. A local cluster should pick up environment variables no problem.

If you want to double check what the workers see as an environment you can use client.run to call a function that returns the current environment ie use os.environ in the function

[cboettig]

I am also struggling with how to pass gdal configuration to rioxarray. It's not clear that it is picking up any environmental variables from the global environment (e.g. set with os.environ). The rasterio package documents two mechanisms to set environmental variables for GDAL, neither one seems to directly set the global environment either https://rasterio.readthedocs.io/en/stable/topics/configuration.html. I have tried these with rioxarray but without any luck.

[artttt]

When faced with this in the past ive found a gdal config option that i can be definitively sure is being used. When reading data if used CPL_VSIL_CURL_ALLOWED_EXTENSIONS and set it such that the file im trying to read will NOT be read, ie if i get an error I know that however i've set the gdal config is working.

rioxarray doesn't seem to make setting a gdal config easy to achieve. The only way i've found is to make sure each worker has the gdal config you want set in the environmental variables.

os.environ should work in the simplest case when not using dask. However when using a dask cluster the environment variables probably wont magically appear on the workers. distributed.diagnostics.plugin.Environ seems it be a useful tool for this, but there other ways that may be better.

Also see this related discussion.
#630