Support modifications of a read file in an external overlay

[rly]

In neurophysiology, metadata is often not static. Sometimes, the experiment description, related publications, or data annotations need to be changed after the file is written. Currently, users can open the file in HDMF in append mode and add containers. Users can also open the file in read mode, make a modification to any part of a container in-memory, and export the modified in-memory container to a new file on disk. Simple changes to metadata cannot be made in append mode and require rewriting the file, which can be expensive (e.g., a 10 GB file is written but one attribute needs to be changed.) Metadata changes can be done in h5py but is hacky.

Data archives would also like to be able to support versioning in a lean way, where making small metadata changes would not require maintaining a complete copy of a file with each change.

The HDF5 group is planning to add support for similar changes / versioning but it will likely be a while before this feature is widely supported.

The NWB and DANDI teams have brainstormed several approaches, one of which is to maintain a human-readable, sidecar JSON file with the same name as the NWB file but with a .json suffix that contains the sequence of changes to be made to the data after reading it. The original NWB file would stay intact.

Other options should be explored too.

Checklist

• Have you ensured the feature or change was not already reported ?
• Have you included a brief and descriptive title?
• Have you included a clear description of the problem you are trying to solve?
• Have you included a minimal code snippet that reproduces the issue you are encountering?
• Have you checked our Contributing document?

[mavaylon1]

@rly where we in our thoughts on a sidecar?

[rly]

I think LINDI would be a better approach but let's discuss at the breakout session on Thursday and add notes here.