cnm.save() issue

[donglabimaging]

Windows 10 Jupyter Notebook

• Describe the issue that you are experiencing
  We modify the demo_pipeline_cnmfE to run our analysis. We are experiencing similar issue as cnm2.save fails #427. It does not happen all the time. It only happens for certain imaging files.

• Copy error log below
  ValueError Traceback (most recent call last)
  in
  ----> 1 cnm.save('28M_Day7_ImageJ_DS.hdf5')

~\Anaconda3\envs\caiman\lib\site-packages\caiman\source_extraction\cnmf\cnmf.py in save(self, filename)
663
664 if '.hdf5' in filename:
--> 665 save_dict_to_hdf5(self.dict, filename)
666 else:
667 raise Exception("Filename not supported")

~\Anaconda3\envs\caiman\lib\site-packages\caiman\utils\utils.py in save_dict_to_hdf5(dic, filename, subdir)
371
372 with h5py.File(filename, 'w') as h5file:
--> 373 recursively_save_dict_contents_to_group(h5file, subdir, dic)
374
375 def load_dict_from_hdf5(filename:str) -> Dict:

~\Anaconda3\envs\caiman\lib\site-packages\caiman\utils\utils.py in recursively_save_dict_contents_to_group(h5file, path, dic)
458 h5file[path + key] = np.array(item)
459 elif type(item).name in ['CNMFParams', 'Estimates']: # parameter object
--> 460 recursively_save_dict_contents_to_group(h5file, path + key + '/', item.dict)
461 else:
462 raise ValueError("Cannot save %s type for key '%s'." % (type(item), key))

~\Anaconda3\envs\caiman\lib\site-packages\caiman\utils\utils.py in recursively_save_dict_contents_to_group(h5file, path, dic)
440 h5file[path + key] = item
441 if not np.array_equal(h5file[path + key].value, item):
--> 442 raise ValueError('The data representation in the HDF5 file does not match the original dict.')
443 # save dictionaries
444 elif isinstance(item, dict):

ValueError: The data representation in the HDF5 file does not match the original dict.

[epnev]

@donglabimaging What's your version of CaImAn?

[epnev]

@donglabimaging
Please follow the procedure below so I can understand better where the saving fails:

1. Go to '~\Anaconda3\envs\caiman\lib\site-packages\caiman\utils\utils.py' and change line 442 from
   raise ValueError('The data representation in the HDF5 file does not match the original dict.')
   to
   raise ValueError('Error while saving {}'.format(key))
2. Relaunch python and reproduce the error
3. Paste the error log

[donglabimaging]

@epnev Thank you for the suggestion. I will reproduce the error and post the error log tomorrow. I believe that I am using Caiman 1.0

[donglabimaging]

@epnev Here is the error log.

ValueError Traceback (most recent call last)
in
----> 1 cnm.save(name + '_fr5.hdf5')

~\Anaconda3\envs\caiman\lib\site-packages\caiman\source_extraction\cnmf\cnmf.py in save(self, filename)
663
664 if '.hdf5' in filename:
--> 665 save_dict_to_hdf5(self.dict, filename)
666 else:
667 raise Exception("Filename not supported")

~\Anaconda3\envs\caiman\lib\site-packages\caiman\utils\utils.py in save_dict_to_hdf5(dic, filename, subdir)
371
372 with h5py.File(filename, 'w') as h5file:
--> 373 recursively_save_dict_contents_to_group(h5file, subdir, dic)
374
375 def load_dict_from_hdf5(filename:str) -> Dict:

~\Anaconda3\envs\caiman\lib\site-packages\caiman\utils\utils.py in recursively_save_dict_contents_to_group(h5file, path, dic)
458 h5file[path + key] = np.array(item)
459 elif type(item).name in ['CNMFParams', 'Estimates']: # parameter object
--> 460 recursively_save_dict_contents_to_group(h5file, path + key + '/', item.dict)
461 else:
462 raise ValueError("Cannot save %s type for key '%s'." % (type(item), key))

~\Anaconda3\envs\caiman\lib\site-packages\caiman\utils\utils.py in recursively_save_dict_contents_to_group(h5file, path, dic)
440 h5file[path + key] = item
441 if not np.array_equal(h5file[path + key].value, item):
--> 442 raise ValueError('Error while saving {}'.format(key))
443 # save dictionaries
444 elif isinstance(item, dict):

ValueError: Error while saving r_values

[epnev]

@donglabimaging Thanks, can you tell me what the value of cnm.estimates.r_values is?

[donglabimaging]

It is an array of values. I am just coping and pasting it, please let me know if you would like me to show it in a different way.

array([ 0.25331816, 0. , 0.06987047, 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0.10248296, 0.16001321, 0. ,
0. , 0.72161293, 0.81875956, 0.27681053, 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0.28551576, 0.21547383, 0. , 0.59137803,
0.28320134, 0.45537084, 0.14588687, 0. , 0.183382 ,
0. , 0. , 0.22468545, 0.31784287, 0. ,
0. , 0.29795268, 0. , 0.3089847 , 0.43607062,
0.30252713, 0.49532762, 0.06882336, 0. , 0. ,
0. , 0. , 0. , 0. , 0.50757527,
0.33767372, 0.28254986, 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , nan, 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0.20138496, 0. , 0.5223248 , 0. ,
0.25367451, 0.35996601, 0. , 0.45052961, 0.48375019,
0.18707378, 0. , 0. , 0.1733571 , 0.32300287,
0. , 0.25283217, 0.36614799, 0. , 0. ,
0.36429125, 0.23129036, 0.24599309, 0.14637537, 0.19116801,
0.38407889, 0.615848 , 0.57694715, 0.41712585, 0. ,
0. , -0.00337458, 0. , 0. , 0.66962522,
0.32184848, 0. , 0. , 0.60442084, 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0.42635268,
0.93063235, 0. , 0. , 0. , 0. ,
0. , 0. , 0.0545918 , 0. , 0. ,
0. , 0.45083597, 0.28487831, 0.27604222, 0.14588434,
0.47218999, 0.4220362 , 0.12468433, 0.35743007, 0.22593041,
0. , 0. , 0. , 0.89494008, -0.05834457,
0. , 0. , 0. , 0. , 0.27881563,
0.44523007, 0. , 0. , 0. , 0. ,
nan, 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0.39793313, 0. ,
0.28132957, 0.37366155, 0.22920035, 0.29099634, 0.41911834,
0.17553519, 0.12887871, 0.30515638, 0. , 0.40736854,
0.32520461, 0.425625 , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0.41244817, 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0.5082826 , 0.1133839 ,
0. , 0. , 0.29890239, 0. , 0. ,
0. , 0.17031549, 0. , 0.28174284, 0.31618845,
0.6437189 , 0. , 0. , 0.41657263, 0.34630808,
0. , 0.2167155 , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0. ,
0. , 0. , 0. , 0. , 0.41752833,
0. , 0. , 0.13488655, 0.32820418, 0.44589606,
0. , 0.16449337, 0.41150722, 0. , 0.55136633])

[epnev]

@donglabimaging No, this is helpful thanks. I'm wondering whether the existence of the NaN value causes the problems. Can you run the following command:

cnm.estimates.r_values = np.where(np.isnan(cnm.estimates.r_values), -1, cnm.estimates.r_values)

and then try to save the object again. Does that work?

[donglabimaging]

It worked! Thank you!

Mind if I ask why is there a NaN instead of 0?

[epnev]

Great! It's a good question why this happened; I don't have an immediate answer.
I'll need to look into this a bit and will provide a fix. Thanks for reporting

[donglabimaging]

Sounds good. Thanks again for the prompt response! I really appreciate your help!!