Open on-disk kerchunk references as a virtual dataset

[TomNicholas]

It might be useful to be able to open an existing kerchunk json/parquet file as a virtual dataset, e.g to make changes to it before writing it back out.

This is essentially the kerchunk version of suggestion (2) here #63 (comment).

This should be really easy to implement: We already have a function for doing it (dataset_from_kerchunk_refs), we just have to teach open_virtual_dataset that existing kerchunk json/parquet files are also valid filetypes to pass in.

[jsignell]

I can take a crack at this one.

[TomNicholas]

You might have to fight @norlandrhagen haha

[jsignell]

Oh! I can back off :) I do think there is an argument to be made for having different methods for open_virtual_dataset and open_as_virtual_dataset or something depending on whether the input is kerchunk-style refs vs actual data.

[norlandrhagen]

Sorry @jsignell! I should have mentioned it in this issue :) If I open a PR, would you mind taking a look at it?

[TomNicholas]

I do think there is an argument to be made for having different methods for open_virtual_dataset and open_as_virtual_dataset or something depending on whether the input is kerchunk-style refs vs actual data.

Yeah this is an interesting question. The same thing will arise for Zarr stores too: should there be a different function to open zarr arrays backed by chunk manifests vs zarr arrays backed by actual bytes on-disk in the store? I think in that context it would be confusing to have two functions, especially as "mixed" zarr stores are possible (and useful).

[jsignell]

should there be a different function to open zarr arrays backed by chunk manifests vs zarr arrays backed by actual bytes on-disk in the store? I think in that context it would be confusing to have two functions, especially as "mixed" zarr stores are possible (and useful).

I would say it makes sense to my brain to have separate functions for "just reading" vs "doing work" so that I can form an expectation about how long something will take to run. But I would expect the open_as_virtual_dataset to accept either a legacy file or a kerchunk reference file or a zarr or a ref zarr.