Training and Validation Dice Values are 0

[DexterLuth]

Hi,
When training a model all training and validation dice values are zero. configs segmentation module has been tailored to our needs by changing label titles and setting pretrained model loading to false.
Here is some terminal output: terminal.txt

GPU: NVIDIA GeForce RTX 4090
MONAILabel version: 0.7.0
Numpy version: 1.24.1
Pytorch version: 2.1.0.dev20230523+cu117
MONAILabel rev id: bfb4f8c

Any help is appreciated!
Thanks!

[tangy5]

Yeah, the logs seem all the validation metrics are 0, several items that you can double check:

1. Whether Image-Label pairs are correct, metrics can be 0, if the data is not aligned or values are wrong. Becuase I see the training Dice is also 0, it might be caused by data.
2. I saw the training loss is decreasing in the first 6 epochs, means that the model is learning something. If the loss are correct, you may need to wait a little bit longer to see the metrics, there are digits limit shown in the logs.
3. If you modified the mtrics, you could double check the metric calculation logics, it could a reason of all 0s.

There could other reasons, hope those helps.

[DexterLuth]

Hey tangy5 thanks for the reply!

1. I checked the data and the data is aligned. What do you mean by the fact that the values could be wrong?
2. I have trained the model for longer periods of time and it remains 0, however, I will try again
3. I'm not sure if I modified the metrics, where is the metric calculation logics located?

[ahmedhosny]

Hello - I am hitting the same issue with decreasing loss and zero training and validation dice.
Only changes made to the segmentation model (to train from scratch):

 # Labels
 self.labels = {
     "tumor": 1,
 }
 # Download PreTrained Model
 if strtobool(self.conf.get("use_pretrained_model", "false")):

[DexterLuth]

Hey Ahmed, I think the problem for me was in the data as tangy5 mentioned above, so I would look into that.

[ahmedhosny]

Thanks! It happened that I needed 2-3x more samples to get any overlap between predicted and actual segmentations.. and ultimately a non-zero dice.

[diazandr3s]

Thanks for the feedback.

By default, MONAI Label uses 20% of the training set for testing. So you'll need at least 4 to 5 annotated samples to see a non-zero Dice score.

If you wish to change this percentage, you could do it in the GUI.

[Keofire]

Hello

I have similar problem. The training loss is decreasing, but the dice (and accuracy) remains 0. I did the labels with monailabel-slicer, from zero.
I have ~70 pairs, with consistent labeling.
I ran ~3000 epochs with no success.

I'm lost, don't know what to do.

[linfanluntan]

I got the same zero dice metrics problem in multi-class training, from my tests, I can confirm

1. the training works for image pair with only one single annotated label in data, when multiple annotated labels present in data, training comes with zero dice metrics
2. if load the pre-trained model, training works for image pair with multiple annotated labels present in data.
3. it seems the training stuck at the beginning for image pair with multiple annotated labels present in data if start from scratch.
4. it seems nothing wrong with training data, should be problems of server

Thank you!

[linfanluntan]

additionally, I was able to train model from scratch for multi-class segmentation in March of 2024, this month I created a new conda environment and installed monailabel, while the codes are the same as in March, the training is failed on the same training data. The only differences are the new conda environment and new monailabel

[linfanluntan]

when installing monailabel it did indicate missing some thing but I can't remember what they are

[linfanluntan]

another difference is in pytorch installation, cu113 (on monailabel doc as below) is no more available, and cu118 is used instead
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113

[lassoan]

What are the origin, spacing, axis direction of the CT and the segmentation?

CT: in Volumes module, Volume information section
Segmentation: in Data module, MRML node information -> Properties -> Segmentation / Segments / Representations / Binary labelmap / Origin, Spacing, IJKToRASDirections

[linfanluntan]

[lassoan]

You have selected the Scene object in the tree. Please select the segmentation node to get information on the segmentation.

[lassoan]

Screenshots are missing. If you have trouble locating the volume information in Volumes and Data module then you can save both the image and the segmentation in NRRD or NHDR format (do not use NIFTI) then you can use any text file viewer to see the image geometry. Please either provide screenshots or copy the NRRD file header here.

[lassoan]

OK, I see the segmentation geometry now. Can you copy the CT volume geometry, too?

[lassoan]

Well, these don't match.

CT:

space origin: (0,0,0) and space directions: (0.91992199420928966,0,0) (0,0.91992199420928966,0) (0,0,5.0000000000000018) is equivalent to:

Spacing: 0.919922 0.919922 5
IJKToRASDirections:
-1 0 0
0 -1 0
0 0 1
Origin: 0, 0, 0

Segmentation:

Spacing: 0.919922 0.919922 5
IJKToRASDirections:
-1 0 0
0 1 0
0 0 1
Origin: 235.04 -235.04 -232.5

Make sure you only use NRRD format and use a MONAI version that includes this fix: Project-MONAI/MONAI#8091

[lassoan]

If you cannot figure out from here then I would recommend to create a new issue or discussion where you describe everything from scratch.

[linfanluntan]

Ok, I'll check it out.

…

On Thu, Sep 26, 2024, 3:25 PM Andras Lasso ***@***.***> wrote: Well, these don't match. CT: space directions: (0.91992199420928966,0,0) (0,0.91992199420928966,0) (0,0,5.0000000000000018) is equivalent to: Spacing: 0.919922 0.919922 5 IJKToRASDirections: -1 0 0 0 -1 0 0 0 1 Segmentation: Spacing: 0.919922 0.919922 5 Extent: 0 511 0 511 0 93 Scalar type: unsigned char Number of components: 1 IJKToRASDirections: -1 0 0 0 1 0 0 0 1 Make sure you only use NRRD format and use a MONAI version that includes this fix: Project-MONAI/MONAI#8091 <Project-MONAI/MONAI#8091> — Reply to this email directly, view it on GitHub <#1495 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANFZ7PE7NN7EZCXHQJYX23ZYRUSTAVCNFSM6AAAAAA2QNQZR2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANZWHAYDANQ> . You are receiving this because you commented.Message ID: ***@***.*** com>

[linfanluntan]

Thanks, I will. BTW, if that means nii.gz won't work?

…

On Thu, Sep 26, 2024, 3:29 PM Andras Lasso ***@***.***> wrote: If you cannot figure out from here then I would recommend to create a new issue or discussion where you describe everything from scratch. — Reply to this email directly, view it on GitHub <#1495 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANFZ7LGDTY4HUCO7XNFDOLZYRVDZAVCNFSM6AAAAAA2QNQZR2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANZWHAYTAMA> . You are receiving this because you commented.Message ID: ***@***.*** com>