VSI plugins

[tbonfort]

This PR allows to dynamically register a VSI handler at runtime by passing a select number of callback functions. Our principle use-case is to be able to bypass the vsicurl handler when accessing remote datasets, to provide observability, custom error handling, retrying, logging, fine-grained concurrency control, fine-grained caching, and better stability under heavily multi-threaded loads.

[rcoup]

@tbonfort possible to share some of that work so we can improve what is in GDAL core?

[tbonfort]

@rcoup I'm not able to share more code, but there really is nothing special to it. I should have explained a bit more that the use case is to call back into scripted language (python, go in our case), so you can easily add your own logging semantics, error handling, authentication, etc... i.e. nothing really applicable to the core gdal implementation.

Two take-backs that could be useful for vsicurl et. al. when dealing with COGs:

• use a two-tiered block cache: one for the first block (supposing the blocksize is big enough to contain the full COG headers), and one for the remaining blocks. This can help lru evictions to not affect the header blocks, and the two caches can be sized based on expected usage.
• implement a global lock on a block when downloading it, i.e. if a block is not present in the block-cache, ensure that there's a single GET to fetch it instead of having each concurrent thread do the same GET multiple times

[tbonfort]

cc @sgillies , could this spark some interest for rasterio ?

[sgillies]

@tbonfort I'm interested!

[rouault]

If anyone has concerns on the interface speak up. I'll merge this soon otherwise

[tbonfort]

I'm wondering if it would make sense to add a way for the callbacks to explicitely return an error condition, without relying on it to correctly set errno and/or call CPLError.

Something in the lines of:

typedef struct {
  int code;
  char *message
} CallbackError;

typedef size_t (*VSIFilesystemPluginReadCallback) ( void *pFile,
               void *pBuffer, size_t nSize, size_t nCount , error *CallbackError);

[rouault]

What is the advantage of this compared to the callback implementation calling VSIError / CPLError ?

[tbonfort]

I was thinking it could be easier for the e.g. python code to set an error message, but I'm fine if that is handled by the C callback function itself.

[rouault]

I see that the behavior of joining consecutive ranges can be disabled with a config option in the vsicurl implementation, but fail to see when that could be desirable.

Yes that's what I mentionned. In some rare circumstances it might help not to merge consecutive ranges, but this is not a good default

Maybe this part could be moved up a bit higher in the vsi call tree?

There's not so many places where that could be done, except in the VSIFReadMultiRangeL() function itself, that will be the one generally called by drivers. The VSICurlHandle::ReadMultiRange() implementation probably saves a buffer allocation/copy compared to your generic implementation though.

[tbonfort]

Lets move the discussion about multi range requests to #1295

[rouault]

@tbonfort Are you happy with the PR as it ?

[tbonfort]

@rouault ready for merge, can I let you squash this through the github interface to avoid re-running the CI?

[rouault]

Done

[mmomtchev]

@rouault, @tbonfort, is there any documentation for this or an example plugin available somewhere?

[tbonfort]

@mmomtchev there are no public implementations that I know of, and the only documentation available is the doxygen one. There is however a similar implementation that does not use this plugin but replicates its intent at https://github.com/airbusgeo/godal/blob/ee62c71eebf81d25ef07983ada2e4ba6ea7baaf7/godal.cpp#L1102 and is used to hand off vsi read-only access to a go function.

[sgillies]

@mmomtchev @tbonfort there's a recent example in rasterio/rasterio#2141.