Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Analyze Plate 4 and rerun analysis + IC with CellProfiler Parallel #17

Merged
merged 10 commits into from
Jul 17, 2023
Merged

Analyze Plate 4 and rerun analysis + IC with CellProfiler Parallel #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
bf1b731
add cellprofiler parallel
jenna-tomkinson Jul 14, 2023
68abec5
modifiy .gitignore file
jenna-tomkinson Jul 14, 2023
6e1667d
rerun nf1 ic with cellprofiler parallel
jenna-tomkinson Jul 14, 2023
84b30ae
run black
jenna-tomkinson Jul 14, 2023
fa5a1e9
edit pipeline names for analysis
jenna-tomkinson Jul 14, 2023
ccbdc5e
removes old pipeline names
jenna-tomkinson Jul 14, 2023
7880375
remove old SQLite files and add ones from new run
jenna-tomkinson Jul 17, 2023
eca0019
edit documentation and notebook + nbconvert
jenna-tomkinson Jul 17, 2023
82df230
modify .gitignore
jenna-tomkinson Jul 17, 2023
23a447b
edit specs
jenna-tomkinson Jul 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@
# ignore pycache
__pycache__

# ignore corrected plates if they are put into a separate folder
Corrected_Images

# ignore corrected plates images (like in download data)
1.cellprofiler_ic/Corrected_Plate_1
1.cellprofiler_ic/Corrected_Plate_2
Expand All @@ -17,6 +20,7 @@ __pycache__

# ignore log files
*logs
*.logs

# ignore Mac related files
.DS_Store
10 changes: 5 additions & 5 deletions ...llprofiler_ic/NF1_illum_Plates_1_2.cppipe → 1.cellprofiler_ic/NF1_illum_3channel.cppipe
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ GitHash:
ModuleCount:11
HasImagePlaneDetails:False

Images:[module_num:1|svn_version:'Unknown'|variable_revision_number:2|show_window:False|notes:['To begin creating your project, use the Images module to compile a list of files and/or folders that you want to analyze. You can also specify a set of rules to include only the desired files in your selected folders.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Images:[module_num:1|svn_version:'Unknown'|variable_revision_number:2|show_window:False|notes:['Images module is blank since we are adding the path to the images in the CLI']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
:
Filter images?:Images only
Select the rule criteria:and (extension does isimage) (directory doesnot containregexp "[\\\\/]\\.")

Metadata:[module_num:2|svn_version:'Unknown'|variable_revision_number:6|show_window:False|notes:['The Metadata module optionally allows you to extract information describing your images (i.e, metadata) which will be stored along with your measurements. This information can be contained in the file name and/or location, or in an external file.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Metadata:[module_num:2|svn_version:'Unknown'|variable_revision_number:6|show_window:False|notes:['No metadata is extracted during illumination correction']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Extract metadata?:No
Metadata data type:Text
Metadata types:{}
Expand All @@ -27,7 +27,7 @@ Metadata:[module_num:2|svn_version:'Unknown'|variable_revision_number:6|show_win
Metadata file name:None
Does cached metadata exist?:No

NamesAndTypes:[module_num:3|svn_version:'Unknown'|variable_revision_number:8|show_window:False|notes:['The NamesAndTypes module allows you to assign a meaningful name to each image by which other modules will refer to it.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
NamesAndTypes:[module_num:3|svn_version:'Unknown'|variable_revision_number:8|show_window:False|notes:['In this pipeline, we only have three channels to correct:', '', 'DAPI', 'GFP', 'RFP']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Assign a name to:Images matching rules
Select the image type:Grayscale image
Name to assign these images:DNA
Expand Down Expand Up @@ -60,7 +60,7 @@ NamesAndTypes:[module_num:3|svn_version:'Unknown'|variable_revision_number:8|sho
Set intensity range from:Image metadata
Maximum intensity:255.0

Groups:[module_num:4|svn_version:'Unknown'|variable_revision_number:2|show_window:False|notes:['The Groups module optionally allows you to split your list of images into image subsets (groups) which will be processed independently of each other. Examples of groupings include screening batches, microtiter plates, time-lapse movies, etc.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Groups:[module_num:4|svn_version:'Unknown'|variable_revision_number:2|show_window:False|notes:['We do not use the Groups module']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Do you want to group your images?:No
grouping metadata count:1
Metadata category:None
Expand Down Expand Up @@ -156,7 +156,7 @@ CorrectIlluminationApply:[module_num:8|svn_version:'Unknown'|variable_revision_n
Set output image values less than 0 equal to 0?:Yes
Set output image values greater than 1 equal to 1?:Yes

SaveImages:[module_num:9|svn_version:'Unknown'|variable_revision_number:16|show_window:False|notes:['Save DAPI images as same tiff bit depth']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
SaveImages:[module_num:9|svn_version:'Unknown'|variable_revision_number:16|show_window:False|notes:['Save DAPI images as same bit depth']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Select the type of image to save:Image
Select the image to save:CorrDAPI
Select method for constructing file names:From image filename
Expand Down
12 changes: 6 additions & 6 deletions ...er_ic/NF1_illum_Plate3_Plate3prime.cppipe → 1.cellprofiler_ic/NF1_illum_4channel.cppipe
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ GitHash:
ModuleCount:13
HasImagePlaneDetails:False

Images:[module_num:1|svn_version:'Unknown'|variable_revision_number:2|show_window:False|notes:['To begin creating your project, use the Images module to compile a list of files and/or folders that you want to analyze. You can also specify a set of rules to include only the desired files in your selected folders.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Images:[module_num:1|svn_version:'Unknown'|variable_revision_number:2|show_window:False|notes:['Images module is blank in the GUI since we are adding the image path in CLI.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
:
Filter images?:Images only
Select the rule criteria:and (extension does isimage) (directory doesnot containregexp "[\\\\/]\\.")

Metadata:[module_num:2|svn_version:'Unknown'|variable_revision_number:6|show_window:False|notes:['The Metadata module optionally allows you to extract information describing your images (i.e, metadata) which will be stored along with your measurements. This information can be contained in the file name and/or location, or in an external file.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Metadata:[module_num:2|svn_version:'Unknown'|variable_revision_number:6|show_window:False|notes:['We do not extract metadata when performing illumination correction']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Extract metadata?:No
Metadata data type:Text
Metadata types:{}
Expand All @@ -27,7 +27,7 @@ Metadata:[module_num:2|svn_version:'Unknown'|variable_revision_number:6|show_win
Metadata file name:None
Does cached metadata exist?:No

NamesAndTypes:[module_num:3|svn_version:'Unknown'|variable_revision_number:8|show_window:False|notes:['The NamesAndTypes module allows you to assign a meaningful name to each image by which other modules will refer to it.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
NamesAndTypes:[module_num:3|svn_version:'Unknown'|variable_revision_number:8|show_window:False|notes:['This pipeline is meant for correction with 4 channels specifically:', '', 'DAPI', 'GFP', 'RFP', 'CY5', '']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Assign a name to:Images matching rules
Select the image type:Grayscale image
Name to assign these images:DNA
Expand Down Expand Up @@ -66,7 +66,7 @@ NamesAndTypes:[module_num:3|svn_version:'Unknown'|variable_revision_number:8|sho
Set intensity range from:Image metadata
Maximum intensity:255.0

Groups:[module_num:4|svn_version:'Unknown'|variable_revision_number:2|show_window:False|notes:['The Groups module optionally allows you to split your list of images into image subsets (groups) which will be processed independently of each other. Examples of groupings include screening batches, microtiter plates, time-lapse movies, etc.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Groups:[module_num:4|svn_version:'Unknown'|variable_revision_number:2|show_window:False|notes:['We do not use the Groups module.']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Do you want to group your images?:No
grouping metadata count:1
Metadata category:None
Expand Down Expand Up @@ -171,7 +171,7 @@ CorrectIlluminationCalculate:[module_num:8|svn_version:'Unknown'|variable_revisi
Maximum number of iterations:40
Residual value for convergence:0.001

CorrectIlluminationApply:[module_num:9|svn_version:'Unknown'|variable_revision_number:5|show_window:False|notes:['Apply illumination correction functions to all channel groups']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
CorrectIlluminationApply:[module_num:9|svn_version:'Unknown'|variable_revision_number:5|show_window:False|notes:['Apply illumination correction functions to all channels']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Select the input image:OrigDAPI
Name the output image:CorrDAPI
Select the illumination function:IllumDAPI
Expand Down Expand Up @@ -251,7 +251,7 @@ SaveImages:[module_num:12|svn_version:'Unknown'|variable_revision_number:16|show
How to save the series:T (Time)
Save with lossless compression?:Yes

SaveImages:[module_num:13|svn_version:'Unknown'|variable_revision_number:16|show_window:False|notes:['Save CY5 images as the same bit-deph']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
SaveImages:[module_num:13|svn_version:'Unknown'|variable_revision_number:16|show_window:False|notes:['Save CY5 images as the same bit-depth']|batch_state:array([], dtype=uint8)|enabled:True|wants_pause:False]
Select the type of image to save:Image
Select the image to save:CorrMito
Select method for constructing file names:From image filename
Expand Down
24 changes: 24 additions & 0 deletions 1.cellprofiler_ic/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,16 @@
In this module, we perform illumination correction (IC) on images for each plate and save the corrected images into new folders.
Images are saved as 16-bit depth, which is the same as the raw data.

## CellProfiler Pipeline

Due to there being differences in channel number between plates, we have two different illumination correction pipelines.

1. [3 channel pipeline](./NF1_illum_3channel.cppipe) -> This pipeline is used only with Plates 1 and 2 since these first pilot plates only had 3 channels.
2. [4 channel pipeline](./NF1_illum_4channel.cppipe) -> This pipeline is used for all the rest of the plates (and future plates) as we have established the protocol for staining.

**Note:** The parameters for correction between pipelines in the same channel might be slightly different.
But, the parameters in both perform and output the best corrected images at this point

## Run the `nf1_ic` notebook

To calculate and apply an IC function on each channel, run the [nf1_ic.ipynb](nf1_ic.ipynb) notebook as a python script using the code block below:
Expand All @@ -15,6 +25,20 @@ cd 1.cellprofiler_ic
source nf1_ic.sh
```

## CellProfiler Parallel

To improve the speed for correcting the images, we have implemented `CellProfiler Parallel`, which utilizes multi-processing to run one plate per CPU core.

### Speed when running CellProfiler Parallel

To run **five plates** through illumination correction, it took approximately **1 hour** in total on a computer using Pop_OS! 22.04 LTS with an AMD Ryzen 7 3700X 8-Core Processor.

This means that we are saving 2 hours (assuming the fifth plate running sequentially would take another hour totalling 3 hours) when running five plates.

### Speed when running CellProfiler sequentially

In the past, we ran one command per plate in seqential order (e.g., one plate is ran and once it finishes the next plate starts).

For four plates, it took about 2 hours to calculate, apply, and save illumination corrected images on a computer using Pop_OS! 22.04 LTS with an AMD Ryzen 7 3700X 8-Core Processor.

For three plates, it took about 18 hours to calculate and save illumination corrected images on a computer using MacOS Ventura 13.2.1 with the M2 chip.
Loading