Importing Develop main (#447)

- New feature - New ROI Detect function: LCCD added - You can rename Workflow now. - ROI Edit on VISUALIZE page (currently, only suite2p works) - Add pipeline sample - Bug fix - #377 - #378 - #379 - Update Documents --------- Co-authored-by: SangLv <sanglv@reactplus.jp> Co-authored-by: quanpython <quannd@reactplus.jp> Co-authored-by: sanglv <lsang2884@gmail.com> Co-authored-by: sanglevinh <109060170+sanglevinh@users.noreply.github.com> Co-authored-by: React Plus - Quan <95335204+quanpython@users.noreply.github.com> Co-authored-by: ut29 <45911784+ut29@users.noreply.github.com> Co-authored-by: tienday <> Co-authored-by: itutu-tienday <122338895+itutu-tienday@users.noreply.github.com>
oist · Mar 30, 2023 · 6e12286 · 6e12286
1 parent 85a9e96
commit 6e12286
Show file tree

Hide file tree

Showing 907 changed files with 1,744,232 additions and 767 deletions.
diff --git a/Dockerfile b/Dockerfile
@@ -1,23 +1,12 @@
-FROM python:3.9.7-slim
+FROM --platform=linux/amd64 python:3.8.16-slim
 
 COPY requirements.txt /app/requirements.txt
 
 RUN apt-get --allow-releaseinfo-change update && \
-    apt-get install --no-install-recommends -y git gcc g++ libgl1 libgl1-mesa-dev && \
+    apt-get install --no-install-recommends -y gcc g++ libgl1 libgl1-mesa-dev && \
     pip3 install --no-cache-dir --upgrade pip && \
     pip3 install --no-cache-dir -r /app/requirements.txt && \
-    pip3 install --no-cache-dir \
-        cython==0.29.30 \
-        holoviews==1.14.9 \
-        ipyparallel==8.4.1 \
-        matplotlib==3.5.2 \
-        opencv-python==4.6.0.66 \
-        scikit-image==0.18.0 \
-        scikit-learn==1.1.1 \
-        tensorflow==2.9.1 && \
-        watershed==2.2.2 \
-    pip3 install --no-cache-dir git+https://github.com/flatironinstitute/CaImAn.git@914324989443fac5d481ef32aad4f327701294a8#egg=caiman && \
-    apt-get purge git -y && apt-get autoremove -y && apt-get clean && rm -rf /root/.cache/pip/*
+    apt-get clean && rm -rf /root/.cache/pip/*
 
 RUN mkdir -p /root/miniconda3 && \
     apt-get install --no-install-recommends -y wget && \

diff --git a/Dockerfile-test b/Dockerfile-test
@@ -1,4 +1,4 @@
-FROM python:3.9.7-slim
+FROM --platform=linux/amd64 python:3.8.16-slim
 
 COPY requirements.txt /app/requirements.txt
 

diff --git a/docs/Makefile b/docs/Makefile
@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/docs/_static/installation/snakemake_use_conda.png b/docs/_static/installation/snakemake_use_conda.png
diff --git a/docs/_static/tutorials/caiman_sample_pipeline.png b/docs/_static/tutorials/caiman_sample_pipeline.png
diff --git a/docs/_static/tutorials/edit-roi/add_roi_clicked.png b/docs/_static/tutorials/edit-roi/add_roi_clicked.png
diff --git a/docs/_static/tutorials/edit-roi/box.png b/docs/_static/tutorials/edit-roi/box.png
diff --git a/docs/_static/tutorials/edit-roi/cell_roi_selected.png b/docs/_static/tutorials/edit-roi/cell_roi_selected.png
diff --git a/docs/_static/tutorials/edit-roi/roi_selected_merge_or_delete.png b/docs/_static/tutorials/edit-roi/roi_selected_merge_or_delete.png
diff --git a/docs/_static/tutorials/fig11_runall.png b/docs/_static/tutorials/fig11_runall.png
diff --git a/docs/_static/tutorials/fig12_cancel.png b/docs/_static/tutorials/fig12_cancel.png
diff --git a/docs/_static/tutorials/fig2.2_tags.png b/docs/_static/tutorials/fig2.2_tags.png
diff --git a/docs/_static/tutorials/fig2_open.png b/docs/_static/tutorials/fig2_open.png
diff --git a/docs/_static/tutorials/fig3_imagenode.png b/docs/_static/tutorials/fig3_imagenode.png
diff --git a/docs/_static/tutorials/fig5_imagenode2.png b/docs/_static/tutorials/fig5_imagenode2.png
diff --git a/docs/_static/tutorials/fig6_suite2p.png b/docs/_static/tutorials/fig6_suite2p.png
diff --git a/docs/_static/tutorials/fig7_suite2p2.png b/docs/_static/tutorials/fig7_suite2p2.png
diff --git a/docs/_static/tutorials/fig8_suite2pcaiman.png b/docs/_static/tutorials/fig8_suite2pcaiman.png
diff --git a/docs/_static/tutorials/fig9_workflowbranch.png b/docs/_static/tutorials/fig9_workflowbranch.png
diff --git a/docs/_static/tutorials/lccd_sample_pipeline.png b/docs/_static/tutorials/lccd_sample_pipeline.png
diff --git a/docs/_static/tutorials/sample_pipelines.png b/docs/_static/tutorials/sample_pipelines.png
diff --git a/docs/_static/tutorials/suite2p_sample_pipeline.png b/docs/_static/tutorials/suite2p_sample_pipeline.png
diff --git a/docs/gui/record.md b/docs/gui/record.md
@@ -1,21 +1,28 @@
 Record
 =================
-In the RECORD field, you can check the workflow status in your OPTINIST_DIR and manage your analysis pipeline. The table lists all the pipelines in your OPTINIST_DIR. By clicking on the mark on the 2nd column, you can show the details of each pipeline.
-
-OptiNiSt records and reproduces past workflow pipelines. It can download results in nwb format.
 <br>
 <p align="center">
 <img width="400px" src="../_static/record/whole.png" alt="Whole"/>
 </p>
 
 
+In the RECORD field, you can check the workflow status in your OPTINIST_DIR and manage your analysis pipeline. The table lists all the pipelines in your OPTINIST_DIR. By clicking on the mark on the 2nd column, you can show the details of each pipeline.
+
+OptiNiSt records and reproduces past workflow pipelines. It can download results in nwb format.
+
+
 OptiNiSt can:
 - Record past executed workflow
 - Reproduce past workflow
 - Download results as NWB files
 - Download snakemake config file
 
 ### Record Table
+
+<p align="center">
+<img width="400px" src="../_static/record/components/table.png" alt="Table"/>
+</p>
+
 - **Timestamp**: latest execution date and time.
 - **ID**: unique ID. This is the directory name for the whole results of the pipeline.
 - **Name**: user-defined workflow name.
@@ -25,18 +32,39 @@ OptiNiSt can:
 - **NWB**: button to copy the analysis results as NWB file to your download folder on your computer.
 - **Delete**: button to delete the workflow from the OPTINIST_DIR.
 
-<p align="center">
-<img width="400px" src="../_static/record/components/table.png" alt="Table"/>
-</p>
 
 
 ### Details
+
+<p align="center">
+<img width="400px" src="../_static/record/components/details.png" alt="Details"/>
+</p>
+
 - **Function**: names of the nods (function).
 - **nodeID**: unique ID. This is the directory name for the results of each node.
 - **Success**: success or failure (abortion with error) of execution of the node.
 - **NWB**: button to copy the analysis results as NWB file to your download folder on your computer.
 
 
-<p align="center">
-<img width="400px" src="../_static/record/components/details.png" alt="Details"/>
+### Reproduce Button
+
+<br>
+<p align="left">
+<img width="100px" src="../_static/tutorials/fig41_reproduce.png" alt="Whole" />
 </p>
+
+Clicking the Reproduce arrow retrieves the pipeline onto the workflow. This function is convenient when you restart the analysis after closing the browser. The reproduced pipeline needs to be `RUN` again (not `RUN ALL`) to make plots available.
+
+### Download Buttons
+
+<br>
+<p align="left">
+<img width="150px" src="../_static/tutorials/fig42_workflownwb.png" alt="Whole" />
+</p>
+
+The Download buttons for the Workflow column and the NWB column copy the Snakemake config or NWB file to your download folder. 
+
+- The Snakemake config file contains the workflow information and parameters for each node. 
+- The NWB file contains the data and its analysis results. 
+
+This function is convenient when users want to share the same analysis pipeline or inspect the output results.
diff --git a/docs/gui/visualize.md b/docs/gui/visualize.md
@@ -1,57 +1,172 @@
 Visualize
 =================
 
-
-OptiNiSt visualizes the analysis results by plotly. 
-<br>
 <p align="center">
 <img width="400px" src="../_static/visualize/whole.png" alt="Whole" />
 </p>
 
+OptiNiSt visualizes the analysis results by plotly. 
 
 
-### Adding a display box
-Click the "+" button to create a display box.   
-To add another display box below, click the "+" button shown below. 
+
+## Adding a display box
 
 <p align="center">
 <img width="400px" src="../_static/visualize/components/set_display_box.png" alt="SetDisplayBox" />
 </p>
 
-To add another display box to the right, click **︙** at the upper right side of the box and show the pulldown. Then, click `Insert into next columns`.
+Click the "+" button to create a display box.   
+To add another display box below, click the "+" button shown below. 
 
 <p align="center">
 <img width="400px" src="../_static/visualize/components/add_column.png" alt="AddColumn" />
 </p>
 
+To add another display box to the right, click **︙** at the upper right side of the box and show the pulldown. Then, click `Insert into next columns`.
 
 ### Selecting an item to show
-Pull down of the `Select Item` shows the available item to show. Select one of these items.
 
 <p align="center">
 <img width="400px" src="../_static/visualize/components/select_output_item.png" alt="SelectOutputItem" />
 </p>
 
+Pull down of the `Select Item` shows the available item to show. Select one of these items.
+
+### Checking movies
+
+<p align="center">
+<img width="200px" src="../_static/tutorials/fig21_loadmovie.png" alt="Whole" />
+</p>
+
+You may want to check some frames of the multi-page tiff files. Visualize page offers the way to check. After creating a plot box by clicking on + mark, Select the image using the SELECT IMAGE button on the left top.
+You can select the range of the frame by assigning 1st and last frame numbers. LOAD button starts loading the data.
+
+<p align="center">
+<img width="400px" src="../_static/tutorials/fig22_movie.png" alt="Whole" />
+</p>
+
+Click on the PLAY button within the plotting box to play the loaded movie.
+The number indicated on the right of PAUSE button is the frame interval in milliseconds. 
+
 
 ### Customizing visualization parameters
-Select one of the display boxes by clicking inside of the box. The blue highlight of the box indicates the current selection of the display box. The parameters shown on the left are attached to the currently selected box. Details of the parameter are explained here.
 
 <p align="center">
 <img width="400px" src="../_static/visualize/components/customize_param.png" alt="CustomizeParam" />
 </p>
 
+Select one of the display boxes by clicking inside of the box. The blue highlight of the box indicates the current selection of the display box. The parameters shown on the left are attached to the currently selected box. Details of the parameter are explained here.
 
 ## ROI and timecourse 
 There is a way to link ROI plots and fluorescence time series. 
 Create One box showing ROI and another box showing fluorescence. You can link two boxes by setting `ref image` in the fluorescence box to be the ID of the ROI box. (ID of the box is on the left upper side). By clicking on the ROI of a cell, you can visualize the corresponding fluorescence time course in the fluorescence box. By turning on the `drag select` in ROI box, you can select multiple cells in the image at once. 
 
 
 ## Ref Image
-TimeSeries plot can refer to a image plot to synthronize cell number index. Click or drag image plot so that cell number indexes are synthoronized in corresponding timeseries plot.
+
 <p align="center">
 <img width="500px" src="../_static/visualize/components/ref_image.png" alt="RefImage" />
 </p>
 
+TimeSeries plot can refer to a image plot to synthronize cell number index. Click or drag image plot so that cell number indexes are synthoronized in corresponding timeseries plot.
+
+
+## Showing ROI and time courses
+After running the ROI detection algorithms, the most often created plots are extracted cells' shape and fluorescence time series. To show the plot, prepare two plotting boxes.
+
+<br>
+<p align="center">
+<img width="600px" src="../_static/tutorials/fig23_twobox.png" alt="Whole" />
+</p>
+
+In one plotting box (ex, the one with ID:0), select a background image such as meanimg from the Select Item pulldowns.
+
+<br>
+<p align="center">
+<img width="200px" src="../_static/tutorials/fig24_selectitem.png" alt="Whole" />
+</p>
+
+In the same plotting box, select cell_roi from the Select Roi pull-downs. Both Suite2P and CaImAn include the process to drop the extracted ROIs that do not meet the criteria. In OptiNiSt, the cell ID is given to all the ROIs. Cell_roi is the ROIs that passed the criteria. 
+
+<br>
+<p align="center">
+<img width="200px" src="../_static/tutorials/fig25_selectroi.png" alt="Whole" />
+</p>
+
+The plotting box (ID:0) shows the background image and detected cells.
+<br>
+<p align="center">
+<img width="400px" src="../_static/tutorials/fig26_roi.png" alt="Whole" />
+</p>
+
+In another plotting box (ex, the one with ID:1), select fluorescence from the Select Item pulldown.
+And select 0(same ID with the plotting box of your ROI image) from the ref image pull down. By doing this,  the two plotting boxes are linked. 
+
+<br>
+<p align="center">
+<img width="400px" src="../_static/tutorials/fig27_fluo.png" alt="Whole" />
+</p>
+
+Now you can explore the ROI and time course. The color of ROI and corresponding time course is matched. You will know the cell ID by letting your mouse over the cell in the image. Clicking on the cell automatically adds the fluorescence time course of the clicked cell. 
+<br>
+<p align="center">
+<img width="600px" src="../_static/tutorials/fig28_roifluo.png" alt="Whole" />
+</p>
+
+If it is tiring to select the cell by clicking one by one, turn on the drag select button on the right in the plotting box of ROI. It enables selecting all the cells within the rectangular area you define.
+
+<br>
+<p align="center">
+<img width="600px" src="../_static/tutorials/fig29_dragselect.png" alt="Whole" />
+</p>
+
+
+## Editing ROI
+
+<p align="center">
+<img width="400px" src="../_static/tutorials/edit-roi/box.png" alt="Whole" />
+</p>
+
+To the edit roi, prepare a plotting box. 
+
+<p align="center">
+<img width="200px" src="../_static/tutorials/fig24_selectitem.png" alt="Whole" />
+</p>
+
+In one plotting box (ex, the one with ID:0), select a background image such as meanImg from the Select Item pulldowns.
+In the same plotting box, select cell_roi from the Select Roi pull-downs. 
+
+<br>
+<p align="center">
+<img width="200px" src="../_static/tutorials/fig25_selectroi.png" alt="Whole" />
+</p>
+The plotting box (ID:0) shows the background image and detected cells.
+<br>
+<p align="center">
+<img width="400px" src="../_static/tutorials/edit-roi/cell_roi_selected.png" alt="Whole" />
+</p>
+
+<p align="center">
+<img width="400px" src="../_static/tutorials/edit-roi/add_roi_clicked.png" alt="Whole" />
+</p>
+
+
+You can click the <strong>Add ROI</strong> button then drag drop, resize the white cirle to change the new ROI position and size.
+Press <strong>OK</strong> or <strong>Cancel</strong> button to Add or No
+
+<p align="center">
+<img width="400px" src="../_static/tutorials/edit-roi/roi_selected_merge_or_delete.png" alt="Whole" />
+</p>
+
+Or click on each cell ROI to delete ROI or merge ROIs (when you select 2 or more ROI cells)
+Press <strong>Merge ROI</strong> or <strong>Delete ROI</strong> or <strong>Cancel</strong> button to Merge or Delete or No.
+
+
 ## saving plots
-For your record, you can save the plots. Select the format and give the filename at the bottom of the parameters. Clicking on the camera mark in the box saves the plot figures to your download folder on your computer.
+
+<p align="center">
+<img width="200px" src="../_static/tutorials/fig30_saving.png" alt="Whole" />
+</p>
+
+You can save created plots in svg, png, jpeg, or webp format. Please select the format, decide the saving name in the lower area on the left panel, and click the camera mark in the plotting box. Svg format saves the plot as a vector-based graphical format which may be convenient when you need high-resolution figures.