Cannot open certain stitched .nd2 files from Elements

[ColinDTaylor]

Hello Ruben,

I am having an issue that I noticed had come up some time in the past but was never fully resolved (in #6). When trying to open certain .nd2 files using nd2Reader I get the error:

ValueError: cannot reshape array of size 45686334 into shape (6759,6759)

This happens only on certain images from this run, and I can't seem to find any way to predict which images will open alright and which will not. We recently changed the stitching in Elements to overlap slightly more than it used to, and maybe that has some connection to this problem cropping up again?

Here is the object info:
<FramesSequenceND> Axes: 4 Axis 'x' size: 6759 Axis 'y' size: 6759 Axis 'c' size: 3 Axis 't' size: 1 Pixel Datatype: <class 'numpy.float64'>

bundle_axes:
['y', 'x']

iter_axes:
['c']

metadata:
{'height': 6759, 'width': 6759, 'date': datetime.datetime(2019, 10, 29, 13, 0, 1), 'fields_of_view': [0], 'frames': [0], 'z_levels': [], 'z_coordinates': [5246.4125], 'total_images_per_channel': 1, 'channels': ['Hoechst', 'AF488', 'AF555'], 'pixel_microns': 0.648678997775168, 'num_frames': 1, 'experiment': {'description': 'unknown', 'loops': []}, 'events': []}

Here is a traceback:

ValueError                                Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\nd2reader\parser.py in _get_raw_image_data(self, image_group_number, channel_offset, height, width)
    276         try:
--> 277             image_data = np.reshape(image_group_data[image_data_start::number_of_true_channels], (height, width))
    278         except ValueError:

<__array_function__ internals> in reshape(*args, **kwargs)

C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py in reshape(a, newshape, order)
    300     """
--> 301     return _wrapfunc(a, 'reshape', newshape, order=order)
    302 

C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py in _wrapfunc(obj, method, *args, **kwds)
     57     if bound is None:
---> 58         return _wrapit(obj, method, *args, **kwds)
     59 

C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\fromnumeric.py in _wrapit(obj, method, *args, **kwds)
     46         wrap = None
---> 47     result = getattr(asarray(obj), method)(*args, **kwds)
     48     if wrap:

ValueError: cannot reshape array of size 45686334 into shape (6759,6759)

[aaristov]

Same problem for me. Any updates?

As an alternative, pims_nd2 handles the same dataset without problems.

[bioimage-analysis]

Hi Ruben (@rbnvrw ),
I'm having a similar issue with a stitch image:

ValueError: cannot reshape array of size 1038565 into shape (1019,1019)

metadata:

{'height': 1019, 'width': 1019, 'date': datetime.datetime(2017, 6, 6, 11, 15, 6), 'fields_of_view': range(0, 1), 'frames': [0], 'z_levels': range(0, 21), 'z_coordinates': None, 'total_images_per_channel': 21, 'channels': ['CSU far red RNA', 'CSU red RNA', 'CSU green RNA', 'CSU blue RNA', 'CSU BF'], 'pixel_microns': 1.0153102411107793, 'num_frames': 1, 'experiment': {'description': 'ND Acquisition', 'loops': []}, 'events': []}

Dimension:

Dimensions: XY(1) x λ(5) x Z(21)

What I found in

def _get_raw_image_data(self, image_group_number, channel_offset, height, width):
[...]
image_group_data[image_data_start::number_of_true_channels]

the slicing is good for the first 5(channels) x 1019 pixels then there is an extra 0 that is added to the data that need to be removed, I added a very crude:

unwanted = []
        for i in range(5099, 5192828, 5095):
            unwanted.append(i)
        
        for ele in unwanted:
            del image_group_data[ele]

5099 is 1019x5+4(image_data_start), 5192828 is the length of image_group_data and 5095 is 1019x5. Obviously it was just me trying to figure out what was wrong.
I wonder if the XY(1) is Dimensions is an indication that something is wrong, but, unfortunately, I don't have more stitched images to try.

[rbnvrw]

Thank you for the addition Cedric. @bioimage-analysis @ColinDTaylor @aaristov could you maybe all add a link to a sample ND2 file that I can download?
I'd really like to solve this, but I don't have any files I can try to analyze. Thanks! 👍

[bioimage-analysis]

Hey @rbnvrw, mine is the sample_data from:
https://downloads.openmicroscopy.org/images/ND2/karl/

[rbnvrw]

This should now be fixed in the latest master version with PR #40 (thanks again @ggirelli !)
If anyone could test this with their images, that would be appreciated.

[aaristov]

Amazing! Now I can read my stitched nd2 files! Thanks a lot guys @rbnvrw @ggirelli ! Please bump the version number to signify this fix.

[ggirelli]

The fix seems to break things for 3D multi-channel images... I am looking into it.

[rbnvrw]

@ggirelli OK thank you. please keep us posted. Does it happen for images with (x,y,z,c) channels only or also (x,y,z,t) or (x,y,c,t) images for example?

[rbnvrw]

@ggirelli could you test this again with the latest fix from #50?

[ggirelli]

@rbnvrw sorry, I am back after a short break after submitting my thesis :)
I'll test this later this week and post here about it!

[rbnvrw]

@ggirelli congratulations on submitting your thesis! do you have time to test this now? :-)

[ggirelli]

Hi @rbnvrw , thanks and sorry, it's been crazy a crazy period before/after the defense :)
I will take care of these in the next couple of weeks!

[ggirelli]

@rbnvrw better late than ever, I just tested it on my files. All good on my end 😄 🎉 🎆

[FrickTobias]

Hi guys!

So I hate to ruin the good mood but I am running into a similar issue.

I'll copy-paste it here it for convenience and it's completely fine for me to share the file (36.6 MB) if you can help me. I saw you might want the metadata from NIS Viewer, I'll see if I can locate it.

Interestingly it is only one of the two channels that has the issue (so I can use images[0] as normal but iterating or images[1] will trigger the ValueError).

Some nd2 images has images seem to miss one data point, making it impossible to reshape their array which also makes the file non-iterable.

Notably 18105583 == 18105584 - 1, where 18105584 == 3892 * 4651 so it would seem one data point is missing.

NIS-viewer can open the file without issues.

Error

ValueError: cannot reshape array of size 18105583 into shape (3892,4651)

Code to reproduce

#! /bin/python3

from nd2reader import ND2Reader

INPUT = "/Users/tobiasfrick/PycharmProjects/demos/test-data/multiple-channels.nd2"


def iterate():
    with ND2Reader(INPUT) as images:
        for _ in images:
            pass


if __name__ == '__main__':
    iterate()

Versions

% python --version
Python 3.8.10
% conda list | grep nd2reader
nd2reader                 3.3.0              pyhd8ed1ab_0    conda-forge

Traceback

python main.py
Traceback (most recent call last):
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/nd2reader/parser.py", line 277, in _get_raw_image_data
    image_data = np.reshape(image_group_data[image_data_start::number_of_true_channels], (height, width))
  File "<__array_function__ internals>", line 5, in reshape
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 298, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 54, in _wrapfunc
    return _wrapit(obj, method, *args, **kwds)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 43, in _wrapit
    result = getattr(asarray(obj), method)(*args, **kwds)
ValueError: cannot reshape array of size 18105583 into shape (3892,4652)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 18, in <module>
    main()
  File "main.py", line 13, in main
    for i in images:
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/slicerator/__init__.py", line 227, in <genexpr>
    return (self._get(i) for i in self.indices)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/slicerator/__init__.py", line 207, in _get
    return self._ancestor[key]
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/slicerator/__init__.py", line 188, in __getitem__
    return self._get(indices)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/pims/base_frames.py", line 98, in __getitem__
    return self.get_frame(key)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/pims/base_frames.py", line 592, in get_frame
    result = self._get_frame_wrapped(**coords)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/nd2reader/reader.py", line 88, in get_frame_2D
    return self._parser.get_image_by_attributes(t, v, c, z, y, x)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/nd2reader/parser.py", line 103, in get_image_by_attributes
    timestamp, raw_image_data = self._get_raw_image_data(image_group_number, channel,
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/nd2reader/parser.py", line 280, in _get_raw_image_data
    image_data = np.reshape(image_group_data[image_data_start::number_of_true_channels], (height, new_width))
  File "<__array_function__ internals>", line 5, in reshape
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 298, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 54, in _wrapfunc
    return _wrapit(obj, method, *args, **kwds)
  File "/Users/tobiasfrick/miniconda3/envs/demos/lib/python3.8/site-packages/numpy/core/fromnumeric.py", line 43, in _wrapit
    result = getattr(asarray(obj), method)(*args, **kwds)
ValueError: cannot reshape array of size 18105583 into shape (3892,4651)

[FrickTobias]

Update

I'm getting the same error for any image I have taken where I am using more than one channel. So possibly this merits a separate issue as I can reproduce it even when I have non-stitched images.

For every image I test on from my most recent dataset, channel=0 always works whereas channel=1 never works (for example in a syntax like this image = images.get_frame_2D(c=channel)).