using the original labels, instead of inverted transformed labels for evaluation

[wyli]

Currently quite a few bundles use
the inverted version of preprocessed original labels as the reference during evaluation:

model-zoo/models/spleen_ct_segmentation/configs/evaluate.json

Lines 12 to 21 in 987f49f

	"_target_": "Invertd",
	"keys": [
	"pred",
	"label"
	],
	"transform": "@validate#preprocessing",
	"orig_keys": "image",
	"meta_key_postfix": "meta_dict",
	"nearest_interp": [
	false,

which is different from the approach taken in the tutorial using the original label directly during evaluation (https://github.com/Project-MONAI/tutorials/blob/main/3d_segmentation/spleen_segmentation_3d.ipynb)

would be great to consider the standard method used in the tutorial for bundle evaluate.json

cc @holgerroth @dongyang0122

(this needs an update as well) https://github.com/Project-MONAI/tutorials/tree/main/bundle/spleen_segmentation)

[dongyang0122]

It is necessary to validate without modifying the label data for (almost) all models in the model zoo.

[yiheng-wang-nv]

Hi @Nic-Ma @wyli , to avoid maintaining duplicate spleen bundles in tutorial and model zoo, should we remove the one in the tutorial, and add a step in the readme file of tutorial to do "download from model zoo"?

[Nic-Ma]

Hi @yiheng-wang-nv ,

I think that's OK, and just put a link to the source code of spleen bundle in the model-zoo.
@wyli What do you think?

Thanks.

[wyli]

sure, i think it'll be a separate new feature request in the tutorial repo, or is it related to this issue?

[wyli]

(this device='cpu' fix will be needed for Invertd Project-MONAI/tutorials#1042 for the monai v1.1 bundles)

[yiheng-wang-nv]

sure, i think it'll be a separate new feature request in the tutorial repo, or is it related to this issue?

Yes, it should be a separate new feature request in the tutorial repo, I just created one: Project-MONAI/tutorials#1043

As for the issue of using inverted labels, may need further discussions with @Nic-Ma @daguangxu @dongyang0122 @wyli

[yiheng-wang-nv]

For the issue, I checked and the following bundles need to be updated:

1. spleen_ct_segmentation (invert transformed labels for evaluation)
2. endoscopic_tool_segmentation (no invert in postprocessing)
3. pancreas_ct_dints_segmentation (invert transformed labels for evaluation)
4. spleen_deepedit_annotation (no invert in postprocessing)
5. swin_unetr_btcv_segmentation (invert transformed labels for evaluation)

[mingxin-zheng]

(this device='cpu' fix will be needed for Invertd Project-MONAI/tutorials#1042 for the monai v1.1 bundles)

Hi @wyli , I don't fully understand this requirement of device='cpu'. Would you mind explaining a bit more, for example, if we don't set it to cpu, will it break something in the spleen segmentation tutorial?

[wyli]

yes, it will raise this error when computing metrics: Project-MONAI/tutorials#1041

[mingxin-zheng]

@wyli Thanks!

[yiheng-wang-nv]

For the issue, I checked and the following bundles need to be updated:

1. spleen_ct_segmentation (invert transformed labels for evaluation)
2. endoscopic_tool_segmentation (no invert in postprocessing)
3. pancreas_ct_dints_segmentation (invert transformed labels for evaluation)
4. spleen_deepedit_annotation (no invert in postprocessing)
5. swin_unetr_btcv_segmentation (invert transformed labels for evaluation)

Update: to prevent unexpected errors, this issue will be addressed after our next release.