Super resolution prediction
Bacteriology suffers strongly from the diffraction limit of light, as many processes occur at subdiffraction scale. Therefore, gaining subdiffraction information from diffraction-limited images represents a very desirable task.
Up to date, there have been several publications demonstrating the potential of DL networks for this task, for example:
- Deep learning enables cross-modality super-resolution in fluorescence microscopy from the Aydogan Ozcan lab
- Content-aware image restoration: pushing the limits of fluorescence microscopy from the Eugene W. Myers lab
- Evaluation and development of deep neural networks for image super-resolution in optical microscopy from the Dong Li lab
- Three-dimensional residual channel attention networks denoise and sharpen fluorescence microscopy image volumes from the Hari Shroff lab
In these publications, super-resolution prediction was demonstrated for different technques, i.e. confocal-to-STED or widefield-to-SIM.
Here, we demonstrate widefield-to-SIM prediction for labeled membranes in live E. coli and S. aureus cells using CARE
Here are the links to the original articles/repositories of the networks we employ for prediction of super-resolution images:
CARE:
- Content-aware image restoration: pushing the limits of fluorescence microscopy by Martin Weigert et al.
- CARE original code and documentation is freely available in GitHub
Please also cite the original CARE paper when using it with our notebook.
-
Successful and robust prediction of super-resolution images using the CARE network requires properly registered image pairs. Typically, this is achieved by consecutive recording of a widefield and SR image (here: SIM). Depending on the dynamics of the target structure, this can pose a non-trivial problem, as motion blur and structural changes between recording of both images can introduce artifacts in network predictions. The recent publication by the Dong Li lab on single-image super resolution microscopy with DL highlights this issue for the highly dynamic nature of the endoplasmic reticulum. A typical strategy to prevent this is to train the network on images obtained from fixed cells, but this requires proper fixation controls.
-
Alignment of diffraction-limited and super-resolved images should be checked carefully. Misaligned image pairs (and even worse, a combination of well-aligned and misaligned images) can confuse the network, which can then lead to poor models (i.e. blurring of structures as a kind of "averaging") or artifacts. An overlay of both channels in ImageJ/Fiji can reveal mismatches, which can be corrected by image registration methods.
-
Training dataset size depends on the heterogeneity of the observed structure. Structures with strongly varying shapes likely require large datasets, allowing the network to learn all features and generalize well. If structures are more uniform, smaller datasets (several tens of image pairs) can be sufficient. Also here, data augmentation can help to increase the size of the training dataset. However, these images are not fully independent from the underlying images and thus only limitedly increase dataset diversity.
In DeepBacs, we provide two datasets:
- Live E. coli cells labeled with FM5-95 and recorded with structured illumination microscopy (SIM)
- Live S. aureus cells labeled with Nile Red and recorded with structured illumination microscopy (SIM)
For S. aureus experiments overnight cultures of S. aureus strain JE2 were back-diluted 1:500 in TSB and grown to mid-exponential phase (OD600 ≈ 0.5). One milliliter of the culture was incubated for 5 min (at 37°C) with the membrane dye Nile Red (5 µg ml−1, Invitrogen), washed once with phosphate buffered saline (PBS), subsequently pelleted and resuspended in 20 µl PBS.
For E. coli experiments overnight cultures of E. coli strain DH5α were back-diluted 1:500 in LB and grown to mid-exponential phase (OD600 ≈ 0.3). One milliliter of the culture was incubated for 10 min (at 37°C) with the membrane dye FM5-95 (10 µg ml−1, Invitrogen), washed once with PBS, subsequently pelleted and resuspended in 10 µl PBS.
One microliter of the labelled culture (E. coli and S. aureus) was then placed on a microscope slide covered with a thin layer of agarose (1.2% (w/v) in 1:1 PBS/TSB solution).
Images were acquired by structured illumination microscopy (SIM) or classical diffraction limited widefield microscopy in a GE HealthCare Deltavision OMX system (with temperature and humidity control, 37°C). The images were acquired using 2 PCO Edge 5.5 sCMOS cameras (one for DIC, one for fluorescence), an Olympus 60x 1.42NA Oil immersion objective (oil refractive index 1.522), Cy3 fluorescence filter sets (for the 561 nm laser) and DIC optics. Each time-point is a Z-stack of 3 epifluorescence images using either the 3D-SIM optical path (for SIM images) or classical widefield optical path (for non super-resolution images). These stacks were acquired with a Z step of 125 nm in order to use the 3D-SIM-reconstruction modality (for the SIM images) of Applied Precision's softWorx software (AcquireSRsoftWoRx), as this provides higher quality reconstructions. A 561 nm laser (100 mW) was used at 11–18 W cm-2 with exposure times of 10-30 ms. For single-acquisition S. aureus experiments, sample preparation and image acquisition was performed as mentioned above but single images were acquired.
Preparation of training datasets
To generate the paired training dataset for super-resolution prediction of E. coli menbranes, raw SIM images were averaged to obtain the diffraction limited widefield image, while the in-focus plane of the SIM reconstruction was used as corresponding high-resolution image. Diffraction-limited images were scaled 2x without interpolation in Fiji to match the pixel size of the reconstructed SIM image. The dataset was curated by removing defocused images and images with low signal resulting in reconstruction artifacts. In total, 55 training and 5 test image pairs were used.
For S. aureus, widefield images were recorded separately before acquiring the SIM images. Diffraction-limited images were scaled 2x without interpolation in Fiji to match the pixel size of the reconstructed SIM image. In total, 94 training and 5 test image pairs were used.
** Here are example images for the E.coli and S. aureus datasets: **
CARE models for the prediction of SIM images of E. coli and S. aureus cells were trained using the following parameters:
| Organism | Scaling | Network | Images (train/test) | Epochs | Steps | Image size | Patch size (#/img) | Batch size | LR | % Valid. | Augmentation | Train time |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| E. coli | 2x | CARE | 55/5 | 300 | 2500 | 1024 x 1024 px² | 80 x 80 px² (100) | 8 | 0.0004 | 10 | 4x (flip/rot.) | 6 h 6 min |
| S. aureus | 2x | CARE | 94/5 | 300 | 3000 | 1024 x 1024 px² | 80 x 80 px² (100) | 8 | 0.0004 | 10 | 4x (flip/rot.) | 7 h 23 min |
| Network | Link to datasets and pretrained models | Direct link to notebook in Colab |
|---|---|---|
| CARE (2D) | E. coli and S. aureus |  |