Artificial labeling
In artificial labeling or label-free prediction, a neural network is used to predict pseudo-fluorescent images from bright field images. The network is trained with paired image data, consisting of a brigh light images (typically 3D data) and corresponding fluorescence images. After training (if successful), the network is able to predict fluorescence signal from the input bright field images. The advantages are quite obvious:
- very gentle to the sample (no laser excitation required)
- improved multiplexing capabilities (the bright field image is the same, no matter what is stained; so different networks can be trained and used to predict the fluorescence image of several targets)
Artificial labeling was pioneered in two studies:
- Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy by Chawin Ounkomol et al. (also called fnet)
- In Silico Labeling: Predicting Fluorescent Labels in Unlabeled Images by Eric Christiansen et al.
These two studies used artificial labeling to stain multiple organelles or target structures in 3D brightfield images of mammalian cells. For microbiology, artificial labeling was not performed yet.
We use 2D implementations of fnet and CARE to predict the bacterial cell membrane in brightfield images.
Here are the links to the original articles/repositories of the networks we employ for artifical labeling:
CARE:
- Content-aware image restoration: pushing the limits of fluorescence microscopy by Martin Weigert et al.
- CARE original code and documentation is freely available in GitHub
fnet:
- Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy by Chawin Ounkomol et al.
- fnet code and documentation from the original publication (used here) is freely available in GitHub.
Please also cite these original papers when using U-Net with our notebook.
In artificial labeling, the network detects features in brightfield, transmitted light or phase contrast images and converts them into a pseudofluorescent image. Due to the blackbox character of the DL network, it is not possible to know whether training will be successful a priori. Even if no prominent features are obviously visible for the human eye, the network might still be able to find features that allow predicting the target structure. However, success is more likely when features are already detectable by eye.
For mammalian specimen, examples of such structures are:
- the nucleus, which is a very prominent feature in brightlight images (see fnet paper)
- mitochondria, which appear as darker or brighter structures (depending on the focus) due to high membrane packaging (see ZeroCostDL4Mic example dataset)
Brightlight images of bacteria are less feature-rich. The main target here is the cell envelop, as typically there is a prominent intensity gradient from background to the bacteria cytosol.
However, there are further structures that are eventually suited for artificial labeling:
- outer membrane vesicles (at least the larger ones)
- the nucleoid (which appears as bright structure at certain focal planes)
- inclusion bodies (appear as dark spots or areas due to their high packing)
- cell poles and invaginations (that can be detected due to their curvature, as done using conventional postprocessing i.e. in MicrobeJ)
In order to train a network for artificial labeling, a paired dataset of brightfield and fluorescent images is required. As for every DL approach, more data is better, as networks generalize better on diverse datasets. However, several 10s of images might suffice (data augmentation helps a lot here) and yield suitable models. The image pairs should be registered as good as possible, so that the network can learn the exact transformation. Improper registration may lead to poor performance or even introduction of artifacts. This can be checked by overlaying the brightfield and fluorescence image in ImageJ/Fiji. Mismatches should be corrected (often, simple translation of one channel can be used for this purpose).
Always set aside a few image pairs for validation of the trained model. These images may not be seen by the network and can be used to quantify model performance (see the corresponding ZeroCostDL4Mic page for further details)
Artificial labeling was used to predict Nile Red fluorescence signal in E. coli brightfield images on different scales.
- Diffraction limited images with 158 and 106 nm pixel size
- Super-resolved images based on PAINT (see here and here) images recorded with Nile Red
Sample preparation and data acquisition
E. coli K12 cells were grown in LB at 37°C and 220 rpm, fixed in mid-exponential phase (OD600 ~ 0.5) using a mixture of 2% formaldehyde and 0.1% glutaraldehyde, immobilized on poly-L-Lysine coated chamberslides and permeabilized with 0.5% TX-100 in PBS for 30 min (see Spahn et al., 2018). For the widefield dataset, membranes were stained using 100 nM Nile Red in PBS and small time series of 50 images were recorded for the brightfield and fluorescence channel (561 nm excitation) on a commercial Nikon N-STORM setup. For super-resolution membrane images, 400 pM Nile Red in PBS was added and PAINT time series (6,000 - 10,000 frames) were recorded on a custom built setup for single-molecule detection (Nikon Ti-E body equipped with a 100x Plan Apo TIRF 1.49 NA oil objective) using 561 nm excitation (~ 1 kW/cm²) or a commercial N-STORM system with a similar objective and imaging parameters. Two image datasets were recorded using either a 1x or 1.5x tube lens (158 and 106 nm pixel size, respectively). PAINT images were reconstructed using Picasso (Schnitzbauer et al., 2017) and exported at different magnifications (8x for 158 nm pixel [ ~19.8 nm/px] and 6x for 106 nm pixel size [17.7 nm/px]). Corresponding bright field images were scaled similarly in Fiji without interpolation and registered with the PAINT image. Multiple 512 x 512 px² image patches were extracted from these images and used for model training. For artificial labeling in drug-treated cells, cells were exposed to the following antibiotics: 100 µg/ml rifampicin for 10 min, 50 µg/ml Chloramphenicol for 60 min, 2 µg/ml Mecillinam for 60 min . Further sample preparation and imaging was performed similar to untreated cells.
Networks for artificial labeling of E. coli membranes were trained via the ZeroCostDl4Mic platform using the following parameters:
| Imaging Mode | Scaling | Network | Images (train/test) | Epochs | Steps | Image size | Patch size (#/img) | Batch size | LR | % Valid. | Augmentation | Train time |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Widefield | none | fnet | 56/8 | n.a. | 200000 | 256 x 256 px² | 128 x 128 px² (1) | 2 | 0.001 | 10 | 4x (flip/rot.) | 1 h 43 min |
| Widefield | none | CARE | 56/8 | 300 | 200 | 256 x 256 px² | 64 x 64 px² (4) | 4 | 0.0004 | 10 | 4x (flip/rot.) | 57 min |
| PAINT | 8x | fnet | 33/20 | n.a. | 200000 | 512 x 512 px² | 128 x 128 px² (1) | 4 | 0.0004 | 10 | 4x (flip/rot.) | 2 h 27 min |
| PAINT | 8x | CARE | 33/20 | 300 | 100 | 512 x 512 px² | 256 x 256 px² (1) | 4 | 0.0004 | 10 | 4x (flip/rot.) | 1 h 33 min |
The ZeroCostDL4Mic notebooks used for artificial labeling in DeepBacs can be accessed here
| Network | Link to datasets and pretrained models | Direct link to notebook in Colab |
|---|---|---|
| CARE (2D) | E. coli membrane labeling |  |
| Label-free prediction (fnet) 2D | E. coli membrane labeling | |