Having trouble training DECODE (ValueError, Runtime Error, Google Collab ModuleNotFoundError)

[nms6]

Hi,

We've been trying to train DECODE for some time now and have run into a few issues on several platforms.

The first is when running DECODE locally, we run into this error (ValueError: The data contains non-finite values). We've attached the image corresponding to the error we get.

In the line above the error, the message we get says the issue is with line 383 of decode_env\lib\site-packages\scipy\stats_continuous_distns.py. Because of this, we just commented out the corresponding lines and reran DECODE. We tried to do this on three different computers and ran into this issue on all three and also ran into this issue regardless of running on CPU or GPU. We also ran into this issue even when we tried using the default param_friendly.yaml and notebook_example.yaml from the sample data provided with Jupyter Notebooks.

After commenting out the line and rerunning DECODE, we were able to train for 60 epochs before running into the following error (RuntimeError: Trying to resize storage that is not resizable at ... ):

We then tried to use the Google Collab notebook and ran into an issue during Step 1 with installing DECODE:

We also wanted to attach the .yaml file we generated from SMAP and have been using in case there are any major errors there:

Thank you so much for your help and so sorry for all the questions! We are fairly new to machine learning and appreciate any help you might be able to offer!

[nms6]

Hi!

Just wanted to update this question!

We managed to get the model to train by looking through old discussion questions and changing the num_worker_train and torch_threads to 0 and 1. We still aren't sure why we had to comment out those lines of codes initially to fix the ValueError (or if that was what we were supposed to do) but it seemed to work. However, now that our model is trained, we still can't seem to fit it given that both Jupyter Notebooks and Google Collab are giving us errors when we try to import decode. Is there a way to fit with the conda terminal similar to how we can train with it? Or are Jupyter and Collab the only way? Thanks so much!!

[Haydnspass]

Dear @nms6 ,

Thanks for the question!

(1) Local: Sorry, there is a bug in the dependencies of decode at the moment; that's why you were getting the errors. I would recommend, in the meantime installing decode with the following conda command

conda create -n decode_v0_10_1rc1 -c turagalab/label/rc -c turagalab/label/dev -c pytorch -c conda-forge decode=0.10.1 cudatoolkit=11.3 jupyterlab ipykernel

here, both training and the jupyter notebooks should work without errors (possibly some warnings). We will release 0.10.1 soon such that you don't have to do that manually anymore.

(2) Colab: Google updated the python version, working on it as well.

Please let me know if there is something still not working with part (1) and the updated installation command.

Best,
Lucas

[nms6]

Thank you so much for the reply! We installed the updated version and were able to train and fit easily! :)

Quick question regarding training performance! We had relatively good data (sparse, bright, high SNR) that we had used to generate a param file from SMAP. Training was complete after ~1000 epochs and the JI converged to around 0.69. We were using the double helix PSF. We were wondering if we should expect convergence at a higher JI?

Thank you so much!

[Haydnspass]

Dear @nms6 ,

the absolute value of JI is not relevant as it will vary by the 'difficulty' of your simulation and is not necessary reflecting the JI expected from your data (e.g. less or more sparse, brighter / less bright etc.). The metrics in tensorboard are however very interesting to judge how much the training has converged, typically 250 epochs are 98% sufficient, so the marginal gain is not much after that.

Cheers,
Lucas

[Haydnspass]

Colab fixed.

[nms6]

Thanks so much!