Prediction after cascade training failed

[MrMarkusJ]

Hi,

in short, what is working:

• trained a 3d_fullres net for data of the size: 280 x 240 x 400 (~ size is variable, but in this range)

Next, the idea was to work on high resolution data of the size 400 x 400 x 660 and to train
a CascadeFullRes net (because in #551 it is mentioned that the cascade-training can "handle" symmetric structures better than
3d_fullres).

Hence, training was performed (5 folds) for

• 3d_lowres
• 3d_fullres_cascade

Next, the command nnUNet_find_best_configuration -m 3d_cascade_fullres was applied; this didn´t worked
out because the "3d_lowres" was missed out; so the command was repeated nnUNet_find_best_configuration -m 3d_lowres 3d_cascade_fullres;
At this step it turned out that the "softmax outputs" were not generated since the option --npz was omitted. Also for this issue a solution is already offered, and for the low_res and the cascade_fullres the "softmax outputs" were calculated afterwards by calling nnUNet_train CONFIGURATION TRAINER_CLASS_NAME TASK_NAME_OR_ID FOLD -val --npz

Next, the command nnUNet_find_best_configuration -m 3d_lowres 3d_cascade_fullres was applied; which resulted in the following output

1. nnUNet_predict -i FOLDER_WITH_TEST_CASES -o OUTPUT_FOLDER_MODEL1 -tr nnUNetTrainerV2 -ctr nnUNetTrainerV2CascadeFullRes -m 3d_lowres -p nnUNetPlansv2.1 -t TaskXXX -z

2. nnUNet_predict -i FOLDER_WITH_TEST_CASES -o OUTPUT_FOLDER_MODEL2 -tr nnUNetTrainerV2 -ctr nnUNetTrainerV2CascadeFullRes -m 3d_cascade_fullres -p nnUNetPlansv2.1 -t TaskXXX -z

3. nnUNet_ensemble -f OUTPUT_FOLDER_MODEL1 OUTPUT_FOLDER_MODEL2 -o OUTPUT_FOLDER -pp /home/YYY/nnUNet_trained_models/nnUNet/ensembles/TaskXXX/ensemble_3d_lowres__nnUNetTrainerV2__nnUNetPlansv2.1--3d_cascade_fullres__nnUNetTrainerV2CascadeFullRes__nnUNetPlansv2.1/postprocessing.json

Unfortunately when the command 1.) was executed the prediction started and during the preprocessing the algorithm tried to allocate for each volume / image 1.7TiB; which is simply too hughe:

preprocessing RESULTS_FOLDER/.........
no separate z, order 3
error in ['nnUNet_raw_data_base/nnUNet_raw_data/TaskXYZ/imagesTs/01_ima01_0000.nii.gz']
Unable to allocate 1.71 TiB for an array with shape (8376, 5225, 5362) and data type float64

-> this image size is too big, since the 01_ima01_0000.nii.gz volume is ~ 250 MB (400 x 400 x 660).

What is the issue here?
Did I missed some steps or did I simply spoiled the model / training due to innitially missing the --npz option and aftwards mixing up the commands?

THANKS

Best,
Markus

[FabianIsensee]

Hey Markus, thanks for the very detailed issue description. I think the error is related to image spacing. nnU-Net determiens the size to which images need to be resampled based on that. My guess is that the image you are trying to predict has a wrong spacing value, causing nnU-Net to compute a way too large target image size for resampling (you can see from the text that it is trying to resample that image to (8376, 5225, 5362)). Please check that the spacing is correct.
On a side note: You seem to be only interested in the cascade. nnUNet_find_best_configuration doesn't really make sense if you only trained one configuration. Just pick that, then. Use nnUNet_find_best_configuration only if you trained multiple configs and want to know which one/which combination is best.
I would not agree with #551 that the cascade handles flips particularly well. Maybe it does, but not by design. I recommend using the noMirror trainer referenced there instead.
Best,
Fabian

[MrMarkusJ]

Dear Fabian,

voxel spacing: this was the important hint! Trained voxel spacing was 32µm³, and the test images had the wrong spacing of 1 mm³.

Regarding your side note: looks like that I have misunderstood the concept of the nnUNet_find_best_configuration command --> as you said the idea is to pick the best configuration for the case that several configurations (3d_lowRes, 3d_fullRes, 3d_cascade) were trained.
What I have noticed is the fact that also in the case of only one configuration trained (3d_fullRes) I need to run nnUNet_find_best_configuration because the postprocessing is missing. Looks like that during the training of the 5 folds this postprocessing is ommited.
Once calling the command nnUNet_find_best_configuration it is stated: running missing postprocessing for task xyz
and the postprocessing is running.
Afterwards the nnUNet_predict can be executed.

• how can this missing postprocessing already be included in the training?

Best,
Markus

[MrMarkusJ]

Supplement Side note: for the current task, segmenting symmetric structures, nnUNetTrainerV2_noMirroring dramatically improves the Dice result!

Thank you once again for the hint and of course for the GREAT code!

Best,
Markus