Meetings

[eddogola]

Fri Oct 20

• Composing models would be helpful e.g. SSD+UNet: Finetune object detector(e.g. SSD) on biological images(get crops) + run crops on instance segmentation model like UNet.
• Finetune using the methodology that makes the most sense: e.g. best to freeze initial layers and train last classification layer than to finetune entire model
• tfjs model to Layers model -> finetune(https://www.tensorflow.org/js/guide/conversion): try to combine inference-only parts with training parts (Essentially: fine-tuning in-browser?)
• What are the boundaries we can push?
  • What's the largest input image size before the browser crashes?
  • What's the maximum throughput we can exhaust?
  • Traditional metrics: memory footprint, time.

[eddogola]

Mon Nov 6

• Composing models has significant promise of improving user experience
  • In terms of composition, the first piece is object detection(if reasonably efficient) - however, this is nontrivial since the object of interest varies depending on the user and their needs; How can this be solved? ***
• User experience idea: ask user to do a crop of the object of interest instead of doing ML-based object detection methods
• Is overfitting something to consider in this context?
• How else can the Piximi experience be improved? Solution space obviously not exhausted.
• Model zoo: Assuming, negligible technical familiarity with segmentation models, give users the best model for their case from a model zoo.
• Human in the Loop training on Piximi
• Tendency in computational(?) biology to not try out things not already translated to the biology domain. In this context, what (segmentation) models haven't been used on biological images that could be ripe for this problem, and what fresh compositions of such models, and popular ones, can be made?

[eddogola]

Wed Nov 15

• Two different ways to think of composing(multi-stage fine-tuning vs model zoo):
  • Nodar: Modular MaskRCNN: SSD+its trainable head; NMS; semantic background, foreground segmentor+trainable head; classifier+trainable head
  • Beth: model zoo, run inference on different models, pick the best one; very expensive
  • Beth: similar to Nodar's but multi-class object detector, ask user to pick object(s) of interest; semantic bgfg; no classifier
• wasm/dynamic webgpu assignment might be a distraction since it's mostly engineering.

[gnodar01]

Papers with datasets:

CytoImageNet

• paper: https://arxiv.org/pdf/2111.11646.pdf
• data: https://www.kaggle.com/datasets/stanleyhua/cytoimagenet

CellTrackingChallenge

• paper: https://www.nature.com/articles/s41592-023-01879-y
• data (you can ignore tracking info): http://celltrackingchallenge.net/annotations/

DeepCellKiosk

• paper: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8759612/
• data: https://datasets.deepcell.org/

LIVECell

• paper: https://www.nature.com/articles/s41592-021-01249-6
• data: https://sartorius-research.github.io/LIVECell/

nuclear image annoted fluorescence image dataset (tissue)

• paper: https://www.nature.com/articles/s41597-020-00608-w
• data: https://www.ebi.ac.uk/biostudies/bioimages/studies/S-BSST265

nucleus segmentations: towards automated solutions

• paper: https://helda.helsinki.fi/bitstream/handle/10138/356032/Trends_in_Cell_Biology.pdf?sequence=1
• data: many discussed in paper, see table 1

[gnodar01]

whiteboard session image:

[eddogola]

Tue Dec 5

*Object detection piece options:
(
what to consider:
- one-stage vs two-stage
- proposal vs proposal-free
- anchor vs anchor free
)
* Yolov1
* RPN
* FPN
* SSD
* FCOS (checks all the right boxes from the criteria above)

• Segmentation piece
  • UNet piece would be ideal as it's CNN-based (doing this for starters)
  • other option is using the segmentation piece from Yolov8
    references: YOLACT
    Yolov8 arch

[eddogola]

Wed Dec 13

• varied data from allen institute
• fcos where/when to take output:
  • backbone feature embeddings output then implement vanila bbox regression layers with our own training criterion
  • get bbox regression outputs add correction layers w criterion
• segmentation piece
  • Frozen UNet with a tunable fully conv layer after
  • frozen unet with learning/tunable unet after
  • unet with one way skip connections, only during forward pass(and maybe that'll be enough) (1)
  • entirely tunable unet on the browser - in tfjs (2)
  • YOLACT as its fully conv, no skip connections so easy to implement (3)
• statistical approach?
  • clustering
  • learn intensity threshold thru some statistical algorithm, just with the image?

[eddogola]

Allen institute data: description data download

[gnodar01]

learn intensity threshold thru some statistical algorithm, just with the image?

@bethac07 and I discussed this a bit more.

See #13, for a base case we could just go with a non-learned statistical measure for thresholding. I.e. simple mean (per channel). For natural images that would be a pretty underperformant approach, but for biological images it gets us a pretty long way.

So we would have pretrained object detector -> trainable object detector -> crops -> simple threshold as our pipeline.

[eddogola]

Fri Feb 23

• More disciplined and consistent communication.
• clarification on next steps for first experiment:
  • FCOS detection model with (simple) trainable wrapper, then
  • YOLACT with trainable wrappers
• engineering next step: Implement (core) models in PyTorch(scrap away chained commands alternative)
• data:
  • BBBC022: nuclei images to be prepared & labeled by image analysts
  • allen institute: ignore for now

[eddogola]

@gnodar01 anything I missed?

[eddogola]

Fri Mar 1

[Agenda]

• YOLACT implementation
• Data
  • dimensions
• Goal: train on entire dataset, then freeze mid layers to fine-tune wrapper layers on slightly different objects

Meeting items:

• think of the core part of the pytorch model that gets converted to onnx as a black box, can't tweak
• we tack tfjs layers to the end and/or start of the onnx->tfjs model
  • at the start it could be, as Beth suggested, making a one channel picture to 3 channels, as the model expects
  • and at the end it could be regression layers that fine-tune the bbox predictions(for starters)
• data: 520x520 but could be any other size; the model expects 550x550; we could upsample/downsample as needed.

[eddogola]

Thur Apr 25

Meeting items:

• copy image from one dimension to others to make a 3-dim image, instead of using a zero tensor