Issue with ClipBoxToImage When Using Negative Samples for Detection

[fanlimua]

I tried to use with both positive and negative samples for detection. However, when loading the dataset, I encountered an error message when using the ClipBoxToImage Class stating "IndexError: The shape of mask[0] at index 0 does not match the shape of index tensor[1] at index 0 match". I've provided the JSON file and part of the code below. Could you help to check if there's a problem with the ClipBoxToImage class implementation, or if there's something wrong with the format of my JSON file?

The JSON file:

{
  "training": [
    {
      "box": [
      ],
      "image": "IXI653-Guys-1122-T2.nii.gz",
      "cl_label": [
        0
      ]
    },
    {
      "box": [
        [
          3,
          20,
          36,
          48,
          73,
          89
        ]
      ],
      "image": "BRATS_004.nii.gz",
      "cl_label": [
        1
      ]
    }
  ]
}

The transforms I used:

train_transforms = Compose(
        [
            LoadImaged(keys=["image"], meta_key_postfix="meta_dict"),
            EnsureChannelFirstd(keys="image"),
            EnsureTyped(keys=["image"]),
            EnsureTyped(keys=["box","cl_label"]),
            StandardizeEmptyBoxd(box_keys=["box"], box_ref_image_keys="image"),
            Orientationd(keys=["image"], axcodes="RAS"),
            Spacingd(
                keys=["image"],
                pixdim=(1.0, 1.0, 1.0),
                mode=("bilinear", "nearest"),
            ),
            CenterSpatialCropd(
                keys=["image"],
                roi_size=(128, 128, 128),
            ),
            ClipBoxToImaged(
                box_keys="box",
                label_keys=["cl_label"],
                box_ref_image_keys="image",
                remove_empty=True,
            ),
        ]
    )

The Error Details:

Traceback (most recent call last):
  File "main.py", line 73, in <module>
    main(config)
  File "main.py", line 40, in main
    solver.swin_train()
  File "/home/fan/project/Medical-image-analysis/train.py", line 714, in swin_train
    total_loss, seg_epoch_loss, det_epoch_loss, cls_epoch_loss, box_reg_epoch_loss = self.train_one_epoch(epoch, self.train_loader, self.device)
  File "/home/fan/project/Medical-image-analysis/train.py", line 250, in train_one_epoch
    for batch_data in train_loader:
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/torch/_utils.py", line 694, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/apps/detection/transforms/dictionary.py", line 839, in __call__
    d[self.box_keys], clipped_labels = self.clipper(d[self.box_keys], labels, spatial_size)
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/apps/detection/transforms/array.py", line 396, in __call__
    return boxes_clip, select_labels(labels, keep)
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/apps/detection/transforms/box_ops.py", line 348, in select_labels
    labels_t = labels_t[keep_t, ...]
IndexError: The shape of the mask [0] at index 0 does not match the shape of the indexed tensor [1] at index 0

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/transforms/transform.py", line 141, in apply_transform
    return _apply_transform(transform, data, unpack_items, lazy, overrides, log_stats)
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/transforms/transform.py", line 98, in _apply_transform
    return transform(data, lazy=lazy) if isinstance(transform, LazyTrait) else transform(data)
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/transforms/compose.py", line 335, in __call__
    result = execute_compose(
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/transforms/compose.py", line 111, in execute_compose
    data = apply_transform(
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.apps.detection.transforms.dictionary.ClipBoxToImaged object at 0x7f4befa0d820>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 51, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/data/dataset.py", line 112, in __getitem__
    return self._transform(index)
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/data/dataset.py", line 98, in _transform
    return apply_transform(self.transform, data_i) if self.transform is not None else data_i
  File "/home/fan/anaconda3/envs/py38-2/lib/python3.8/site-packages/monai/transforms/transform.py", line 171, in apply_transform
    raise RuntimeError(f"applying transform {transform}") from e
RuntimeError: applying transform <monai.transforms.compose.Compose object at 0x7f4beea08310>

[KumoLiu]

Hi @fanlimua,
The error message "IndexError: The shape of mask[0] at index 0 does not match the shape of index tensor[1] at index 0" often arises when the array index is trying to access out-of-bounds, like when trying to access an element at an index which is greater than its number of elements.
In regards to the ClipBoxToImage, it truncates bounding boxes to ensure that they lie completely within the relevant image. Specifically, the bounding box dimensions should match the corresponding image dimensions.
Looking at your JSON file, I noticed:

• For the 'IXI653-Guys-1122-T2.nii.gz' entry, the 'box' field is empty.
• For the 'BRATS_004.nii.gz' entry, there is one bounding box which spans from index 3 to 20 in the first dimension, index 36 to 48 in the second dimension, and index 73 to 89 in the third dimension.
  Here are a couple of points for consideration to verify and possibly solve the issue:

1. Make sure the indices in 'box', especially for 'BRATS_004.nii.gz', are within the actual image dimensions.
2. Please verify whether your dataset's format matches with your implementation of ClipBoxToImage class and its corresponding data pipeline. Specifically, ensure that the dataset provides "boxes" (or modify the key_used to match the one in the dataset).

Thanks.

[fanlimua]

Hi @KumoLiu, thank you for your reply. Actually this issue occurs on "IXI653-Guys-1122-T2.nii.gz" which is a health brain.
From my understanding, this error is caused by "keep_t" from the box label and "labels_t" from classification label. The box label being empty while the classification label has a value that results in the index error. I attempted to set both the box and classification labels to empty, which resolved the index error, but negatively impacted the detection metric.
I'm confuse about this part, if the brain does not have a tumor, it would not have a box label, but it should still have a classification label. Could you provide guidance on how to properly set the box and classification labels for negative samples?

[KumoLiu]

Hi @fanlimua, looks like a similar issue here, hope you can find the solution.
#1197 (reply in thread)
Thanks.