Keypoint + segmentation mask in Detecton2

[VGrondin]

Hi, I would like to do instance segmentation and keypoint detection at the same time. From issue #17 it says I need to enable both mask and keypoint in my config.

My question is: which model do you start from in model_zoo (eg. COCO Instance Segmentation Baselines with Mask R-CNN or COCO Person Keypoint Detection Baselines with Keypoint R-CNN) and is this the good way to enable keypoints (#1219 (comment))?

Thanks!

[kvnsng]

Have you gotten this to work? I'm currently trying to do the same, and I'm having some trouble.

I looked at both of those config files (FPN version) and combined them:

_BASE_: "../Base-RCNN-FPN.yaml"
MODEL:
  WEIGHTS: "detectron2://ImageNetPretrained/MSRA/R-50.pkl"
  MASK_ON: True
  KEYPOINT_ON: True
  RESNETS:
    DEPTH: 50
  ROI_HEADS:
    NUM_CLASSES: 3
  ROI_BOX_HEAD:
    SMOOTH_L1_BETA: 0.5  # Keypoint AP degrades (though box AP improves) when using plain L1 loss
  RPN:
    # Detectron1 uses 2000 proposals per-batch, but this option is per-image in detectron2.
    # 1000 proposals per-image is found to hurt box AP.
    # Therefore we increase it to 1500 per-image.
    POST_NMS_TOPK_TRAIN: 1500
    
SOLVER:
  STEPS: (210000, 250000)
  MAX_ITER: 270000

then I change some of the configs by running this:

cfg = get_cfg()
cfg.merge_from_file("/path/seg-keypoint-RCNN-FPN.yaml")

cfg.DATASETS.TRAIN = ("my_dataset",)
cfg.DATASETS.TEST = ()
cfg.DATALOADER.NUM_WORKERS = 2
cfg.MODEL.WEIGHTS = "detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl"
cfg.MODEL.ROI_KEYPOINT_HEAD.NUM_KEYPOINTS =6
cfg.TEST.KEYPOINT_OKS_SIGMAS = (1, 2, 3, 4 ,5, 6)
cfg.SOLVER.IMS_PER_BATCH = 10
cfg.SOLVER.BASE_LR = 0.00025
cfg.SOLVER.MAX_ITER = 5000    # 300 iterations seems good enough, but you can certainly train longer
cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 256   # faster, and good enough for this toy dataset
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 3  # has four class

This hasn't worked for me, but I think it has more to do with a data loading issue as I keep getting
ValueError: expected sequence of length 6 at dim 1 (got 0) error.

If you have any insight on this, that would be great!

[kvnsng]

Thanks for the response! I noticed you're using semantic segmentation. I'm trying to use instance segmentation with it, and I'm wondering if it's causing problems because of that. Have you tried using instanced segmentation with keypoint detection??

[kvnsng]

I think the problem might be that I have 3 different objects (1 with keypoints and 2 without). I thought maybe I could simultaneously run instance segmentations on those 2 objects without keypoints and keypoint detection for the 1 with, and I don't think Detectron2 is capable of doing that...? Please correct me if I'm wrong, haha.

[VGrondin]

If each object class can have a different number of keypoints, it is not supported by Detectron2 models (see: #1415 (comment)) . You can try the solution proposed in the link and implement one keypoint head per class. Personally, for your classes that don't have any keypoints, I would try to register dummy keypoints that you set the visibility to 1 (not visible, see cocodataset/cocoapi#130 (comment)), but I don't know if the network will still learn. Let me know if it works.

[kvnsng]

Ahhh, okay. I think the dummy keypoints might be interesting. Thanks for the discussion!

[ZhangYi1999]

What if I train the mask and keypoint rcnn on two datasets