Visium hires image segmentation

[chsher]

Thank you for creating this great tool. I am working on cell type deconvolution based on your tutorial (https://squidpy.readthedocs.io/en/latest/external_tutorials/tutorial_tangram.html), but I am running into an issue with cell segmentation. Namely, the Visium H&E image is not very high-resolution (2000 pixels x 1843 pixels), and I am having trouble finding a threshold to detect the nuclei based on your example (https://squidpy.readthedocs.io/en/latest/auto_examples/image/compute_segment_hne.html). Do you have any recommendations for segmenting the cells in a Visium H&E image?

[giovp]

Hi @chsher

thank you for trying out squidpy! Indeed segmentation is quite difficult on hne stain, especially if low resolution. You can play around with the threshold argument but can be difficult to find an heuristic. Do you mind sharing a crop of the image just to get a sense of how "visible" the nuclei are in your image?

[chsher]

Thanks for your speedy reply @giovp. Here is a crop of the image (left: original, right: smoothed).

[giovp]

mmh I see yes, it looks pretty difficult. Seems like each nuclei is only represented by a handful of pixels, and smoothing definitely doesn't help. I see it difficult also for DL based approaches. If you'd like to try one I would suggest you to start with this maybe https://github.com/MouseLand/cellpose

We provide a segmentation custom class that you can use with any callable (e.g. a DL model) here: https://squidpy.readthedocs.io/en/latest/classes.html#segmentationcustom

If that also doesn't work out of the box, then you probably have to pre-train the model with some ground truth that you'd have to segment yourself. It can be complicated and time consuming...

pinging @hspitzer who might have some smart tips.

I just want to mention that for Tangram you don't need to have segmentation masks at all costs, and the deconvolved proportions (as showed in the tutorial) works quite well already. In case you don't manage with segmenting it, could be interesting to try out tangram without it anyway.

[tpentim]

Hi @giovp ,

here is Tancredi from Rajewsky Lab, first of all: I'm grateful for this great tool!

I'm following up on this question as I've a similar problem with segmenting nuclei in fresh frozen HE images and I've tried cellpose and other DL segmentation tools (e.g. stardist: [https://github.com/stardist/stardist] ) achieving satisfactory results.

Now I'm facing problems when including these models in squidpy custom segmentation. What are exactly the requirements of this function? The function I'm inputting as 'method' is callable, takes as input a numpy array of shape (x, y, channels) and returns a numpy array of shape (x, y) with integers identifying pixels belonging segmented objects. Does this makes sense to you?

from stardist.models import StarDist2D 
from csbdeep.utils import Path, normalize


def stardist_segmentation(img):
    model = StarDist2D.from_pretrained('2D_versatile_he')
    
    img = normalize(img, 1,99.8)
    
    labels, details = model.predict_instances(img,
                                         prob_thresh= 0.3)
    return labels

import squidpy

img = sq.im.ImageContainer(chunks= 1000 ,img= 'A1_small.tiff')

sq.im.segment(
    img=img,
    layer_added= 'segmented_stardist',
    method=stardist_segmentation
)

The precise error message, which occurs only with custom segmentation, is:

Found model '2D_versatile_he' for 'StarDist2D'.
Loading network weights from 'weights_best.h5'.
Loading thresholds from 'thresholds.json'.
Using default values: prob_thresh=0.692478, nms_thresh=0.3.
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-28-544e4dcedf52> in <module>
----> 1 sq.im.segment(
      2     img=img,
      3     channel= 0,
      4     layer="image",
      5     layer_added= 'segmented_stardist',

~/miniconda3/envs/stardist/lib/python3.8/site-packages/squidpy/im/_segment.py in segment(img, layer, method, channel, size, layer_added, copy, show_progress_bar, n_jobs, backend, **kwargs)
    297     start = logg.info(f"Segmenting `{len(crops)}` crops using `{segmentation_model}` and `{n_jobs}` core(s)")
    298 
--> 299     crops: List[ImageContainer] = parallelize(  # type: ignore[no-redef]
    300         _segment,
    301         collection=crops,

~/miniconda3/envs/stardist/lib/python3.8/site-packages/squidpy/_utils.py in wrapper(*args, **kwargs)
    164             pbar, queue, thread = None, None, None  # type: ignore[assignment]
    165 
--> 166         res = jl.Parallel(n_jobs=n_jobs, backend=backend)(
    167             jl.delayed(runner if use_runner else callback)(
    168                 *((i, cs) if use_ixs else (cs,)),

~/miniconda3/envs/stardist/lib/python3.8/site-packages/joblib/parallel.py in __call__(self, iterable)
   1039             # remaining jobs.
   1040             self._iterating = False
-> 1041             if self.dispatch_one_batch(iterator):
   1042                 self._iterating = self._original_iterator is not None
   1043 

~/miniconda3/envs/stardist/lib/python3.8/site-packages/joblib/parallel.py in dispatch_one_batch(self, iterator)
    857                 return False
    858             else:
--> 859                 self._dispatch(tasks)
    860                 return True
    861 

~/miniconda3/envs/stardist/lib/python3.8/site-packages/joblib/parallel.py in _dispatch(self, batch)
    775         with self._lock:
    776             job_idx = len(self._jobs)
--> 777             job = self._backend.apply_async(batch, callback=cb)
    778             # A job can complete so quickly than its callback is
    779             # called before we get here, causing self._jobs to

~/miniconda3/envs/stardist/lib/python3.8/site-packages/joblib/_parallel_backends.py in apply_async(self, func, callback)
    206     def apply_async(self, func, callback=None):
    207         """Schedule a func to be run"""
--> 208         result = ImmediateResult(func)
    209         if callback:
    210             callback(result)

~/miniconda3/envs/stardist/lib/python3.8/site-packages/joblib/_parallel_backends.py in __init__(self, batch)
    570         # Don't delay the application, to avoid keeping the input
    571         # arguments in memory
--> 572         self.results = batch()
    573 
    574     def get(self):

~/miniconda3/envs/stardist/lib/python3.8/site-packages/joblib/parallel.py in __call__(self)
    260         # change the default number of processes to -1
    261         with parallel_backend(self._backend, n_jobs=self._n_jobs):
--> 262             return [func(*args, **kwargs)
    263                     for func, args, kwargs in self.items]
    264 

~/miniconda3/envs/stardist/lib/python3.8/site-packages/joblib/parallel.py in <listcomp>(.0)
    260         # change the default number of processes to -1
    261         with parallel_backend(self._backend, n_jobs=self._n_jobs):
--> 262             return [func(*args, **kwargs)
    263                     for func, args, kwargs in self.items]
    264 

~/miniconda3/envs/stardist/lib/python3.8/site-packages/squidpy/im/_segment.py in _segment(crops, model, layer, layer_new, channel, queue, **kwargs)
    339     segmented_crops = []
    340     for crop in crops:
--> 341         crop = model.segment(crop, layer=layer, channel=channel, **kwargs)
    342         crop._data = crop.data.rename({layer: layer_new})
    343         segmented_crops.append(crop)

~/miniconda3/envs/stardist/lib/python3.8/functools.py in _method(*args, **kwargs)
    910         def _method(*args, **kwargs):
    911             method = self.dispatcher.dispatch(args[0].__class__)
--> 912             return method.__get__(obj, cls)(*args, **kwargs)
    913 
    914         _method.__isabstractmethod__ = self.__isabstractmethod__

~/miniconda3/envs/stardist/lib/python3.8/site-packages/squidpy/im/_segment.py in _(self, img, layer, channel, **kwargs)
    100     def _(self, img: ImageContainer, layer: str, channel: int = 0, **kwargs: Any) -> ImageContainer:
    101         # simple inversion of control, we rename the channel dim later
--> 102         return img.apply(self.segment, layer=layer, channel=channel, **kwargs)
    103 
    104     @abstractmethod

~/miniconda3/envs/stardist/lib/python3.8/site-packages/squidpy/im/_container.py in apply(self, func, layer, channel, copy, **kwargs)
    827             arr = arr[{channel_dim: channel}]
    828 
--> 829         res = func(arr.values, **kwargs)
    830         if res.ndim == 2:
    831             res = res[..., np.newaxis]

~/miniconda3/envs/stardist/lib/python3.8/functools.py in _method(*args, **kwargs)
    910         def _method(*args, **kwargs):
    911             method = self.dispatcher.dispatch(args[0].__class__)
--> 912             return method.__get__(obj, cls)(*args, **kwargs)
    913 
    914         _method.__isabstractmethod__ = self.__isabstractmethod__

~/miniconda3/envs/stardist/lib/python3.8/site-packages/squidpy/im/_segment.py in _(self, img, **kwargs)
     88             raise ValueError(f"Expected only `1` channel, found `{img.shape[-1]}`.")
     89 
---> 90         arr = self._segment(img, **kwargs)
     91 
     92         if arr.ndim == 2:

~/miniconda3/envs/stardist/lib/python3.8/site-packages/squidpy/im/_segment.py in _segment(self, arr, **kwargs)
    164 
    165     def _segment(self, arr: np.ndarray, **kwargs: Any) -> np.ndarray:
--> 166         return np.asarray(self._model(arr, **kwargs))
    167 
    168     def __repr__(self) -> str:

<ipython-input-4-a8c40e912284> in stardist_segmentation(img)
      4     img = normalize(img, 1,99.8)
      5 
----> 6     labels, details = model.predict_instances(img,
      7                                          prob_thresh= 0.3)
      8     return labels

~/miniconda3/envs/stardist/lib/python3.8/site-packages/stardist/models/base.py in predict_instances(self, img, axes, normalizer, prob_thresh, nms_thresh, n_tiles, show_tile_progress, verbose, predict_kwargs, nms_kwargs, overlap_label)
    422         _shape_inst   = tuple(s for s,a in zip(_permute_axes(img).shape, _axes_net) if a != 'C')
    423 
--> 424         prob, dist = self.predict(img, axes=axes, normalizer=normalizer, n_tiles=n_tiles, show_tile_progress=show_tile_progress, **predict_kwargs)
    425         return self._instances_from_prediction(_shape_inst, prob, dist, prob_thresh=prob_thresh, nms_thresh=nms_thresh, overlap_label=overlap_label, **nms_kwargs)
    426 

~/miniconda3/envs/stardist/lib/python3.8/site-packages/stardist/models/base.py in predict(self, img, axes, normalizer, n_tiles, show_tile_progress, **predict_kwargs)
    303 
    304         channel = axes_dict(axes_net)['C']
--> 305         self.config.n_channel_in == x.shape[channel] or _raise(ValueError())
    306         axes_net_div_by = self._axes_div_by(axes_net)
    307 

~/miniconda3/envs/stardist/lib/python3.8/site-packages/csbdeep/utils/utils.py in _raise(e)
     88 
     89 def _raise(e):
---> 90     raise e
     91 
     92 

ValueError: 

Thanks in advance for your help!

[giovp]

Hi @tpentim !

thank you for your interest in squidpy!

The image container is currently undergoing bit of a refactoring (see here #296 ) shouldn't take long to get it merged. Furthermore, we are gonna provide extensive tutorials on how to use stardist and cellpose (currently worked on here theislab/squidpy_reproducibility#12 and here theislab/squidpy_reproducibility#11 ). Both of them will end up in the tutorials page in the docs.

Therefore, if you could give it 2-3 weeks, we'll provide all info necessary for reproducibility. I will ping you here when everything is merged. Thank you in advance for your patient 🙏

[tpentim]

Hi @giovp,

thanks for your reply. It's great news there will be tutorials to integrate stardist and cellpose!
Thanks for letting me know when it's ready and for your help.
Best,
Tancredi

[giovp]

hi @chsher , @tpentim

we are gonna release squidpy 1.2 sometime next week. Meanwhile, if you install it with pip install -e . from the dev branch, you can already try out the new tutorials.
We have added tutorials to work with Stardist and Cellpose:

squidpy.readthedocs.io/en/latest/external_tutorials/tutorial_cellpose_segmentation.html
squidpy.readthedocs.io/en/latest/external_tutorials/tutorial_stardist.html

let us know if that helps, they should also be general enough to help you plug in your own pretrained/fine tuned favourite model.

[giovp]

hi both, with the new release those notebooks are now stable
https://squidpy.readthedocs.io/en/latest/external_tutorials/tutorial_cellpose_segmentation.html
https://squidpy.readthedocs.io/en/latest/external_tutorials/tutorial_stardist.html

will close this but feel free to reopen if not addressed