Is there a way to load only apartial portion of a given nwb file from cloud storage?

[GMCadode]

For my lab I'm hoping for a cloud storage solution that allows the user to query only certain parts of a large NWB file (say, the spike times only, and not the raw voltage traces). Does anybody know if there's already an established way to load only parts of a file from cloud storage? My current setup requires downloading an entire large file from cloud storage, which often takes a long time.. Is this sort of functionality already an established practice in managing online repositories of data?

[CodyCBakerPhD]

Hi there!

This is indeed very possible with NWB files

See our streaming tutorial using fsspec on an s3_url here: https://pynwb.readthedocs.io/en/stable/tutorials/advanced_io/streaming.html#streaming-method-1-fsspec

Though fsspec in principle ought to allow you to use any other type of cloud storage, not just S3 (which we choose since that is how the DANDI archive stores NWB files)

fsspec is also especially good with caching, which saves those 'small parts' of the 'big remote file' file to your local system so that you don't need to re-download them every time you want to use them again (read more about all the various options here: https://filesystem-spec.readthedocs.io/en/latest/features.html#caching-files-locally)

We plan to add some convenience functions in the future to make this sort of thing a bit easier to specify and learn (together with instructions)

Let me know if you run into any difficulties or have any questions about how to harness the power of remote data streaming

[GMCadode]

Awesome, thanks for that, and for the prompt reply!

[bjhardcastle]

Is streaming supported for zarr NWB files?

I've tried modifying the method in the tutorial, replacing h5py.File() with zarr.open(), and while the zarr file can be opened and read fine (from s3), the resulting storage type FSStore isn't supported by hdmf-zarr.

I thought faster streaming from the cloud would be one of the primary benefits of using zarr for NWB files - am I doing something wrong?

[oruebel]

I've created the following issue for this hdmf-dev/hdmf-zarr#134 FSStore is indeed currently not listed. That being said, I think as a work-around for now, you may be able to simply monkey-patch it by setting:

hdmf_zarr.backends.SUPPORTED_ZARR_STORES = (DirectoryStore, TempStore, NestedDirectoryStore, FSStore)

before opening the file. For read-only support, I believe all that is really needed is to make sure it is listed as an allowed store.

[bjhardcastle]

Fantastic - thank you for the quick reply and for working on official support for FSStore!

If anyone else is trying this:

• I wasn't able to get it to work via monkey-patching
  • there's a closure over SUPPORTED_ZARR_STORES at import in ZarrIO's docval decorator, which seems to be used to validate any input args
• modifying SUPPORTED_ZARR_STORES in hdmf_zarr/backend.py also doesn't work
  • throughout ZarrIO there is usage of os.path that assumes a local path was provided
• I was eventually able to load a zarr nwb directly from an S3 URI with hdmf_zarr.NWBZarrIO by messing with some of the os.path usage in ZarrIO but I don't fully understand the repercussions of what I changed
  • for the small zarr file I tried (from the tutorial here: https://hdmf-zarr.readthedocs.io/en/latest/tutorials/plot_nwb_zarrio.html#sphx-glr-tutorials-plot-nwb-zarrio-py), reading the file with NWBZarrIO still took much longer than with zarr.open() (over 60 seconds, compared to ~4 sec)

So, probably best to wait until a proper implementation is released!

[oruebel]

Thanks for the additional notes. I'll take a look. The "repercussions" regarding path are most likely mainly related to writing and reading of links and references. I think getting read to work should be pretty quick, which is I think the main need right now.