external file for image series

[Saksham20]

In addition to a bunch of raw and processed data, I'm trying to create an image series in the nwb file to store a video.
This video is stored remotely and has a web url.

Is it an acceptable practice to store data behind a web url as an external file arg thus reducing the nwb file size substantially?

[bendichter]

@satra and @yarikoptic, this is a current challenge for us. HDF5 does not currently support popular video codecs, and even if you use gzip the file size is still huge in comparison to the original video files. NWB can get around this by allowing you to store the video externally and put a relative path to that video in the ImageSeries object of an NWB file. However, it does not currently appear to be possible to do this with DANDI. When you dandi organize, only NWB files are considered and rearranged. Do we want to support external video files in DANDI? Is it currently possible? If so, what would be the best practice way to do this?

As @Saksham20 has noted, one potential solution is to host the video online, and have the ImageSeries contain the video's URL, however this might not be an ideal solution because that video could change or be removed from that URL without any changes to the dandiset.

[satra]

we need to improve dandi organize to allow for other things anyway (a dataset image, bids files, potentially zarr indices, etc.,.).

one could go bids-style and say there are corresponding associated files to a given nwb file. the keys for your specific issue with video files are going to be how to address local referencing on a file system and Web-based referencing. something needs to be able to update the pointers appropriately and also ensure things are valid.

let's perhaps create a design doc for this problem. do you want to start one in the CLI repo?