Multi-label segmentation by segmentation model

[saeidsh1370]

Hi everyone,

I am currently working on segmenting various organs in Head and Neck CT images (approximately 25 organs). I have 30 labeled CT datasets (512x512 resolution). I modified the label names in the segmentation.py file and set use_pretrained_model to false. Despite using high epochs (1000 to 3000) with target spacing 3 and ROI size (96, 96, 96), my accuracy remains below 10% for both SegResNet and UNet networks (After experimenting with different spacings and ROI sizes, these values have shown the best results so far). However, I am encountering several issues:

1- Since I aim to segment tissues of varying sizes (from optic nerves to cerebrum and lungs), what would be the optimal ROI size and spacing? Is it feasible to segment all organs accurately with a constant size for these parameters?

2- The predicted labels are often incorrect and seem random, e.g., the brain is labeled as lung or muscle as thyroid. I've verified that the label numbers are consistent across my training datasets. What could be causing this issue?

3- Even when segmenting a single label, the accuracy is below 30% after training on 30 datasets for 100 epochs (This issue also persists when using the spleen dataset with the segmentation.py model without a pre-trained model). How many images are typically required to achieve high accuracy for single-label segmentation like spleen size?

4- How can I implement affine transformations in main.py to augment my dataset and increase the number of training samples?

Any insights or suggestions on these issues would be greatly appreciated. Thank you!

[lassoan]

In your screenshot above the image spacing is incorrect. Did you resample all the input images to have the same resolution?

You can also improve segmentation quality by segmenting more structures. You don't need to do it manually, but use pretrained models, such as whole head and upper body segmentation models of 3D Slicer's MONAIAuto3DSeg extension and TotalSegmentator extension.

If you can release your segmentations publicly then @diazandr3s may be able to help with the training and distributing via MONAI Auto3DSeg.

[diazandr3s]

Hi @saeidsh1370,

It seems an issue related to the indexes associated to each segment in the annotated dataset.

Can you please expand more on this?

I am currently working on segmenting various organs in Head and Neck CT images (approximately 25 organs). I have 30 labeled CT datasets (512x512 resolution).

How did you create these annotations?

1- Since I aim to segment tissues of varying sizes (from optic nerves to cerebrum and lungs), what would be the optimal ROI size and spacing? Is it feasible to segment all organs accurately with a constant size for these parameters?

Yes, it is feasible. I'd start with an ROI of 96x96x96

2- The predicted labels are often incorrect and seem random, e.g., the brain is labeled as lung or muscle as thyroid. I've verified that the label numbers are consistent across my training datasets. What could be causing this issue?

Can you please comment on how did you make sure the label indexes are consistent among the training set?

3- Even when segmenting a single label, the accuracy is below 30% after training on 30 datasets for 100 epochs (This issue also persists when using the spleen dataset with the segmentation.py model without a pre-trained model). How many images are typically required to achieve high accuracy for single-label segmentation like spleen size?

Which single region/segment did you use to verify this?

4- How can I implement affine transformations in main.py to augment my dataset and increase the number of training samples?

You could update the default pretransforms here: https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/trainers/segmentation.py#L75-L95

[saeidsh1370]

Thanks, @lassoan, for your prompt answer. All CT images in my dataset have the same resolution. Also, thanks, @diazandr3s, for your comment.

• My dataset contains 35 manually labeled head and neck images, annotated by clinical experts. I have checked all the labels and found that each organ has the same indicated number in each CT scan. Although some CT images have fewer labels, I believe this will not be an issue for MONAILabel. However, the organ numbers are consistent throughout the dataset (as you can see here). Following your tutorial, I changed the label names in segmentation.py to match my dataset.

• For simplification, I reduced my labels to 3: Cerebrum (label 1), Cerebellum (label 2), and Brainstem (label 3). I used the segmentation model for training over 200 epochs and 32 CT images. The results are still unsatisfactory and have incorrect labeling (e.g., the green area represents the Cerebrum but is labeled as Brainstem).

• Regarding single label segmentation, I experimented with different organs like the brain, eye, and lung. As a test, I used the segmentation (not segmentation_spleen) model with the SegResNet default settings (spacing 1.5x1.5x1.5, ROI size 96x96x95, pretrained: false, 100 epochs) for training on 30 CT scans from the Task09_spleen dataset. However, the results were not satisfactory, as shown in the image.

• I have another question about active learning. Can I train a model for my dataset using an active learning strategy? My idea is to train the model with a small number of labels initially, then gradually increase the number of labels in subsequent steps and update the segmentation model. If I keep my segmentation.pt model in the radiology model folder and use different steps with different labels, can I update the segmentation.pt model without negatively impacting the previously learned information?

Thank you in advance

[saeidsh1370]

Thanks @diazandr3s for response and comments

I checked the spreadsheet you shared here and that's good to keep track of the annotations. But unfortunately this is not enough to make sure the labels indexes are consistent thougthout the dataset. We need to check the annotation files directly. Can I ask which file format are you using to store the annotations? Is it nii.gz or NRRD?

My dataset is in .mat (MATLAB) files, and I save them in both .nii.gz and .nrrd formats. In both cases, the label indexes problem persisted. Another problem I encountered, which might be useful for others, was the lack of alignment between the original images and the segmentation results in NIFIT format. I solved this by using reader="ITKReader" as guidance here . Additionally, this problem can be solved without using ITKReader by changing the origin of the .nrrd file as follows:
origin = [-size(image, 1) / 2, -size(image, 2) / 2, -size(image, 3) / 2]

This is a good test and also an extra clue that the label indexes are not consistent :/ In the segmentation file, which number did you use to represent Cerebrum (label 1), Cerebellum (label 2), and Brainstem (label 3). Was it 1, 2 and 3? Did you make sure the brainstem is always represented by the number 3 in the training set files? These results hint that this is not the case, I'm afraid. Once this is checked and fixed, I'd recommend training for more epochs - maybe 600

That's a good point. Initially, I used the default label indexes (Cerebrum: 8, Cerebellum: 7, and BrainStem: 9). In this setup, I encountered an issue where, when loading labels via the "Import Label" tool in MONAILabel, all labels were rendered with the same color, as shown in the image.

However, the illustration of labels was correct when loading data via the "Data Import" tool of 3D Slicer or by dragging and dropping. To address the color issue, I changed the label numbers to Cerebrum: 1, Cerebellum: 2, and BrainStem: 3. This adjustment resolved the color issue when illustrating labels via the "Import Label" tool. However, the problem of incorrect labeling persisted, even after 600 epochs of training. Indeed, neither of these changes had any effect on the results.

Finally, after further investigation, I identified the root cause of the incorrect labeling problem. It turned out that my labels for each organ were not sorted alphabetically, while the resulting labels were being interpreted based on alphabetical sorting. To fix this, I changed all label values based on the alphabetical order of the organ names. This adjustment surprisingly corrected the label indications.

Thank you again :)

[diazandr3s]

Many thanks for sharing your feedback, @saeidsh1370

[lassoan]

If you store segmentations in .seg.nrrd files then you can use either the segment names - or even better, standard terminology - to create labelmap images where a segment is always mapped to a specific label value. See example here: https://github.com/lassoan/slicerio?tab=readme-ov-file#extract-selected-segments-with-chosen-label-values

[saeidsh1370]

Hi again,
First, I want to thank both @lassoan and @diazandr3s for your invaluable guidance. Following your advice, I’ve successfully implemented active learning for my dataset, achieving acceptable accuracy for larger anatomical structures such as the lungs, spinal cord, and bones.

However, I’m encountering a significant challenge with smaller, low-depth tissues like the eye lens and optic nerves (as seen in the image). Despite using a training set of 65 datasets and running 1000 epochs, my segmentation model consistently fails to achieve accuracy for these smaller structures, often resulting in zero accuracy.
I’ve experimented with various configurations, but the accuracy remains unchanged:

• self.roi_size = (96, 96, 16) (also tested with (8, 8, 8) and (32, 32, 32)).
• self.target_spacing = (1, 1, 1) (also tested with (0.8, 0.8, 0.8)).
• Increased network depth with blocks_down=(2, 3, 4, 4) and blocks_up=(2, 2, 2).
• Additionally, I removed the Gaussian smoothing operation (GaussianSmoothd(keys="image", sigma=0.4)) during both training and inference to avoid smoothing these regions.

Given these parameters, could you provide any insights or suggestions on how to improve accuracy for small organs, similar to the results I've obtained with larger structures?

Thank you in advance for your assistance.

[diazandr3s]

Hi @saeidsh1370,

Small regions are challenging to segment, especially in CT scans with few slices. What's the average slice count in your training dataset?

Regarding ROI sizes:

• Avoid 8 or 32 for X and Y dimensions
• Try 128x128x32 (requires more GPU memory)

Alternative: Expand your training data using the whole head model from Slicer Auto3DSeg to create more annotated samples for retraining.

Hope this helps,