Convert ND-TIFF image plane metadata

[ziw-liu]

@ieivanov raised the need of converting all image plane metadata, which will be useful for the mantis project.

In a previous meeting we agreed that the easiest, fastest, and most robust way to preserve MM image plane metadata during conversion is to toggle on the option to store them as a separate TXT file in MM. However this may not be accessible from pycromanager. @ieivanov is this the reason why need such a feature? If so I propose that we only implement this for ND-TIFF, and write an output TXT file similar to what MM MDA would produce.

[ieivanov]

The motivation here is to be able to convert all of our tif datasets - ome.tif or ndtiff, to zarr for downstream processing and long-term storage. Ideally we'd be able to delete the original tif datasets after conversion to avoid data duplication. For that, we need to be 100% sure that we transfer all data and metadata.

[mattersoflight]

@ziw-liu @ieivanov we will not be able to delete TIFF files across all projects, particularly ones where Fiji/ImgaeJ are still tools of choice of the users.

Let's focus on capturing metadata for the mantis project, i.e., NDTIFF format. @ieivanov,
Does the NDTIFF python reader provide a metadata object that you can query during conversion?

Next question would be: what is a useful format for metadata that doesn't create many small files in one folder (here I am keeping in mind rsync efficiency)? Two ideas:

• Add a custom field in .zattrs, say acq-meta-<tczyx>.
• Dump metadata in machine parsable text files, and store them in singe tar.gz with the zarr store.

[ieivanov]

Yes, let's discuss today if we have enough time. The ndtiff reader does provide handles to all metadata. We should figure out how to store it efficiently.

MM does a bit of a word vomit when it comes to metadata. Most of these value don't change across the acquisition. We can figure out a way to store constant value and those that change only across certain dimensions of the data - e.g. z position during a z stack, channel params when imaging different channels, or elapsed time, which changes for every image.

[ziw-liu]

Upstream issue: ome/ngff#104

The current bioformats2raw behavior seems to be dumping the OME XML as a separate text file, although not officially part of the specification.
EDIT: the spec did make a special case for the OME-XML in NGFF v0.4.1. It probably warrants a closer look at their implementation then (i.e. whether the OME metadata is sufficient). Unfortunately direct reuse won't be possible since it's Java and GPL.

[ziw-liu]

I will try dumping all the image plane metadata into a single JSON blob in the top level of the output Zarr store.