cannot train detection and instance segmentation together with Mask-RCNN

[mashuai191]

If you do not know the root cause of the problem / bug, and wish someone to help you, please
post according to this template:

Instructions To Reproduce the Issue:

1. what exact command you run:
   previously i trained a mask-rcnn model with COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml, and now i added some new images with only bounding box annotation(no segmentation annotation) and re-train the model. What i intend to do is to combine detection and segmentation in one model, for example, some instance has only segmentation and some has bbox only. Theorically Mask-RCNN model should be ok to do this kind of thing but looks not supported yet in code, with following error logs printed. Anyone can help me for a workaroud to go forward? Thanks!

2. what you observed (including full logs):

KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/content/detectron2_repo/detectron2/data/common.py", line 41, in __getitem__
    data = self._map_func(self._dataset[cur_idx])
  File "/content/detectron2_repo/detectron2/utils/serialize.py", line 23, in __call__
    return self._obj(*args, **kwargs)
  File "/content/detectron2_repo/detectron2/data/dataset_mapper.py", line 134, in __call__
    annos, image_shape, mask_format=self.mask_format
  File "/content/detectron2_repo/detectron2/data/detection_utils.py", line 260, in annotations_to_instances
    segms = [obj["segmentation"] for obj in annos]
  File "/content/detectron2_repo/detectron2/data/detection_utils.py", line 260, in <listcomp>
    segms = [obj["segmentation"] for obj in annos]
KeyError: 'segmentation'

[mashuai191]

for example, there are two instances in an image, a person with polygon annotation, and a bicycle with bounding box annotation, because i care person's shape but don't care bicyle's, only care bicycle's location.
is it possible to train with images like this? if yes how to do that?

[ppwwyyxx]

In theory it's possible to train a model with such data.
But this means that the input format to the model will need to be slightly different. You can achieve this by following the tutorials to write new data loader and replace some of the existing model components to handle such new input format.