bump ndtiff version to >1.12.1

[ieivanov]

ndtiff==1.12.0 and below have a bug which returned the wrong shape of dask arrays, see micro-manager/NDStorage#107

[ziw-liu]

Maybe we can finally get rid of the workaround in this:

https://github.com/czbiohub/iohub/blob/64083d45fa45527a171665c7700f6ad4f9204398/iohub/ndtiff.py#L119-L166

Also this may be related to #105.

[ieivanov]

Correct, you shouldn't need to reshape the array at the end anymore.

ndtiff now also sorts the axes in PTCZ order when Dataset.as_array(axes=None). Your implementation of axis sorting will error out or omit channels if custom axis names are present.

See https://github.com/micro-manager/NDTiffStorage/blob/d7645a418013017e49a2d38f89cf8b486e20468b/python/ndtiff/nd_tiff_current.py#L237

[ziw-liu]

ndtiff made a 2.0.0 release: https://pypi.org/project/ndtiff/#history

Not sure what the breaking changes (if any) are...

Edit: It happened in this PR: micro-manager/NDStorage#111, which doesn't seem to change how offline python reading works.

[ziw-liu]

ndtiff now also sorts the axes in PTCZ order when Dataset.as_array(axes=None). Your implementation of axis sorting will error out or omit channels if custom axis names are present.

At first glance (v2.0.0) calling NDTiffDataset.as_array() without axes does not return in PTCZ order:

from ndtiff import Dataset # v2.0.0
# pytest dataset with PTCYX=(2, 3, 2, 7, 128, 128)
dataset = Dataset("MM20_pycromanager/mm2.0-20210713_pm0.13.2_2p_3t_2c_7z_1/")
dataset.as_array(position=0).shape
# (2, 7, 3, 128, 128)=CZTYX
dataset.as_array().shape
# (2, 2, 7, 3, 128, 128)=PCZTYX

[ieivanov]

Axis sorting is only implemented for the "current" NDTiff format version whereas mm2.0-20210713_pm0.13.2_2p_3t_2c_7z_1 is NDTiff_v2_0, as defined here.

Just tested that axes are correctly sorted in PTCZ order for a new pycromanager dataset. We should update the iohub PM test dataset with data from the "current" format.

[ieivanov]

P.S. we could extend the NDTiff_v2_0 format to also do axis sorting. That's something that I had planned on, but didn't get around to implementing.

[ziw-liu]

We should update the iohub PM test dataset with data from the "current" format.

I agree that we should add the new format to tests. However the old one should still be supported to be able to read existing data. Then there will need to be branching logic for different versions like we have for OME-TIFF MMStacks? Does ndtiff provide an API for consistently accessing arrays from different versions?

[ieivanov]

Does ndtiff provide an API for consistently accessing arrays from different versions?

Yes, the ndtiff Dataset initializes the right reader such that the returned object has the same API, independent of the underlying dataset version.

[ziw-liu]

Yes, the ndtiff Dataset initializes the right reader such that the returned object has the same API, independent of the underlying dataset version.

This is what iohub is currently using. However using this method to access arrays from old format datasets still needs reordering axes and reshaping the array (current behavior), while for the new format:

Your implementation of axis sorting will error out or omit channels if custom axis names are present.

[ieivanov]

Just added axis sorting for v2 datasets in micro-manager/NDStorage#117. This way, you can use the same code for v2 or v3 ndtiff datasets. This means that you won't need to do axis sorting in iohub anymore. However, you'll still need to pad the array with singleton dimensions if they are missing in the dataset (as iohub currently does).