GUI tries to import .mmap files based off path during pipeline and not at data load.

[tvajtay]

For better support, please use the template below to submit your issue. When your issue gets resolved please remember to close it.

Sometimes errors while running CNMF occur during parallel processing which prevents the log to provide a meaningful error message. Please reproduce your error with setting dview=None.

If you need to upgrade CaImAn follow the instructions given in the wiki.

• Tell us about your operating system (Linux/macOS/Windows), Python version (3.x), working environment (Spyder/Jupyter Notebook/other), and (if applicable) which of the demo scripts you're using for your analysis
  Linux, 3.6.6, GUI

• Describe the issue that you are experiencing
  GUI uses the internal hdf5 object paths instead of path provided to GUI to hdf5 to find corresponding .mmap files. This becomes an issue when I analyze videos with an HPC and then download data to a local machine for analysis. GUI is trying to use HPC paths instead of my local one.

• Copy error log below

(caiman) tomi@tomi-HP-Pavilion-Laptop-15:~/CaImAn$ ipython caiman/gui/gui_pyqtgraph_layout.py
Gtk-Message: 11:12:13.655: Failed to load module "overlay-scrollbar"
Gtk-Message: 11:12:13.657: Failed to load module "atk-bridge"
Gtk-Message: 11:12:13.658: Failed to load module "canberra-gtk-module"
1
QObject::connect: invalid null parameter
/home/tomi/anaconda3/envs/caiman/lib/python3.6/site-packages/h5py/_hl/dataset.py:313: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
  "Use dataset[()] instead.", H5pyDeprecationWarning)
ERROR:root:File request:[/scratch/tjv55/Naomi_2d1/Test/memmap__d1_256_d2_398_d3_1_order_C_frames_14663_.mmap] not found!
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
~/CaImAn/caiman/gui/gui_pyqtgraph_layout.py in <module>
     50 # movie
     51 # mov = cm.load('/Users/agiovann/caiman_data/example_movies/memmap__d1_60_d2_80_d3_1_order_C_frames_2000_.mmap')
---> 52 mov = cm.load(cnm_obj.mmap_file)
     53 min_mov = np.min(mov)
     54 max_mov = np.max(mov)

~/CaImAn/caiman/base/movies.py in load(file_name, fr, start_time, meta_data, subindices, shape, var_name_hdf5, in_memory, is_behavior, bottom, top, left, right, channel, outtype)
   1426     else:
   1427         logging.error('File request:[' + str(file_name) + "] not found!")
-> 1428         raise Exception('File not found!')
   1429 
   1430     return movie(input_arr.astype(outtype), fr=fr, start_time=start_time, file_name=os.path.split(file_name)[-1], meta_data=meta_data)

Exception: File not found!

[epnev]

The problem is that the path to the memory mapped file is saved in the cnmf object and then it tries to read it from there (as shown in line 52 highlighted in your error log).
There is a couple of things one can do there:

• We can modify the gui script to search for the memory mapped file in the directory where the saved object is located. That could be risky in case there is more than one memory mapped files in that directory.
• We can load the object in python, modify the the path to the file save it, and then open in GUI. That will work but is kind of dump.
• We can create an additional file dialog.

In all cases, you will need to transfer the memory mapped from your HPC back to your local machine or re-create them. We'll need to think a bit about what's the best approach here.

[tvajtay]

So in a best case scenario I would just use the GUI with an interactive session on the HPC, but I get an oasis error. While oasis is available on the login node and allows me to install normally, I think it isn't on the compute nodes, so it gives an error. I thought this was because of an installation error, but now I see that oasis is not a part of the conda environment. Since I think only the libraries explicitly in the caiman environment get loaded, there is no oasis on the compute nodes. Which prevents me from opening the GUI.

[pgunn]

Do you have a shared filesystem between the compute nodes? If so, and if you're using conda, the oasis bits should just work.

[tvajtay]

We do have a shared file system, I'm just a bit confused as to how CaImAn can import oasis without it being explicitly installed in the environment. Unless it uses the local C compiler which may not be loaded on the compute nodes.

[tvajtay]

This is the output I'm currently getting on HPC https://pastebin.com/Y43UND7f

[pgunn]

The problem is that "cd". If you did a normal pip install . it places the built libraries in your normal python extension directories, and you cannot do work in your sourcedir (because python hardcodes looking in the current directory for libraries before looking in the normal library locations). For development, you have another option with pip that doesn't require you do reinstall every time you change the sourcedir (add the -e flag to pip). For other users, you should put your GUI launcher into the datadir somewhere.

On your other question, Oasis is a binary component that's built during pip install that's used by caiman. It lives in the conda lib dir. It's not a separate systems component, and so long as your shared systems have access to the conda dir it was installed into, it should work. (again, provided you're not running things inside your sourcedir)

[tvajtay]

So during installation I cd into CaImAn to use environment.yml and build the environment, activate the environment, cd back out, and then pip install .? I think most people will call pip install . inside the CaImAn directory based on the installation instructions.

[pgunn]

no - the pip install must be done inside the caiman dir. It's just that once you've done that you no longer should do anything inside the sourcedir.

[tvajtay]

Alright, some progress, the GUI pops up but now I'm hitting issues with loading in the .mmap, which is great, because that brings us back to the original issue. https://pastebin.com/YnU4uzE7
I think this may be due to my pipeline because I may have commented out Cn.

Also I have to load git as a module for directory searches?

[epnev]

    738         try:
--> 739             lc = subprocess.check_output(["git", "rev-parse", "HEAD"]).decode("utf-8").split("\n")[0]
    740             self.data['last_commit'] = lc
    741         except subprocess.CalledProcessError:

I wasn't excepting this to cause an error since it is inside a try/except clause. Feel free to comment it out; it just stores the head of the last commit as a means of knowing exactly what version of the code you're using.

[pgunn]

The reason this fails is that it doesn't catch the particular exception being thrown.
If another except block is added that catches FileNotFoundError (or is a generic except), that should stop it from failing.

[epnev]

@tvajtay see my response above. we'll introduce a fix.

[epnev]

should be ok now.

[tvajtay]

So, yes, I can now open CaImAn GUI on our HPC. WooHoo! Thanks for the help!

[tvajtay]

Wow, I am very impressed so far with the GUI btw. I think the .mmap issue is not that concerning now that it works on the HPC where all the data is anyway. Having the ability to specify a .mmap file with an additional dialog will probably be useful to people if they don't have interactive desktops?

[epnev]

Yes, we'll think about this. Glad it worked out!

[epnev]

@tvajtay d82bf7a enables the option to select the memory mapped file in case it is not found in the path. Can you test that it works?

[tvajtay]

Yes, I can test later today.

…

On Wed, Aug 7, 2019, 10:13 AM eftychios pnevmatikakis < ***@***.***> wrote: @tvajtay <https://github.com/tvajtay> d82bf7a <d82bf7a> enables the option to select the memory mapped file in case it is not found in the path. Can you test that it works? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#590?email_source=notifications&email_token=AECSXT4KRYRC3ARBI3ZB3CDQDLJ77A5CNFSM4IFZXQA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3YRITY#issuecomment-519115855>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AECSXT7EEE6RZRDGPRCXMLTQDLJ77ANCNFSM4IFZXQAQ> .

[epnev]

@tvajtay did you get a chance to try this?

[tvajtay]

@epnev Just did, works with current version on some older data I have analyzed. Analyzed on HPC, downloaded .h5 and .mmap, was able to open for analysis on local.

[epnev]

Great, thanks!