Adding multiple classification heads to train

[Shobhit20]

I am trying to build a model with multiple classification heads. MaskRCNN already has 3 heads - mask, box and class
How can we add another custom head such as another classification head. Is there a possibility to extend the current implementation of detectron to cater to this problem statement?

Say for example I have a problem statement in which I am supposed to infer the location, bbox, class, color of fruit from an image.
For inferring the class and the color one way can be to club the two into one class making a combination of each class with each color but that creates somewhat imbalanced datasets. Can I rather do multiple classification heads where one predicts the class and the other predicts the color of the crop?

[Hartvi]

I am trying tackle exactly the same problem, the only difference being that I do not necessarily need a brand new head, which I have seen examples of in issue #841, but for the model to have another instance of a mask head, such as mask_head.py.

In our project we are running the basic Detectron2 with tuples and it works very well. We currently have object categories as a series of tuples (box, paper), (box, plastic), but we would like to separate them into (box)+(paper)/(plastic).

If I am not mistaken, then for creating a new head you need a regular nn.Module or nn.Sequential class that just implements the forward function, e.g. mask_head and deeplab.

Then in roi_heads.py we should implement the StandardROIHeads or the super class ROIHeads and add the new head to it.

If I wanted to simply add another mask-type head and run Detectron2, how would I go about doing that? Would I need to make a wholly new project like DensePose or just add custom scripts to load the desired heads? And what are the places that would need to be modified? Because I see that there are many references to the heads in lots of different .py places.

[Hartvi]

In addition to my previous comment, one needs to create a yaml file in the detectron2/configs folder to include additional heads.

I have dived into previous issues and documentation and here is a compilation of related links I've found

• Issues pertaining to multiple head training:
  • Training with different level of supervisions #45
  • Keypoint + segmentation mask in Detecton2 #2871
  • How to enable both mask and keypoint in your config ? #1219
  • mutil task training #17
  • cannot train detection and instance segmentation together with Mask-RCNN #1190
  • Training keypoints + detection heads jointly #276
• Documentation related to the issues:
  • Use Custom Datasets
  • Writing custom models: Write Models
  • Losses are calculated in FasterRCNN stuff: FastRCNNOutputLayers
  • For different formats of datasets there are needed custom data loaders: Write a Custom Dataloader

[Shobhit20]

@Hartvi thanks for putting all this together
Will try to add multiple heads using these resources and in the meantime, if I stumble across any resources for adding a custom mask head, will keep you posted.

[eaedk]

Hello, how to finetune on my own keypoints dataset ??? I don't find any tutorial that works for me. I need help please. I have a dataset with 4 keypoints per image

[yusiyohpolimi]

In our project we are running the basic Detectron2 with tuples and it works very well. We currently have object categories as a series of tuples (box, paper), (box, plastic), but we would like to separate them into (box)+(paper)/(plastic).

@Hartvi Hello, I am trying to run Detectron2 with category_id = [bulb_number, direction] which is not supported as default. I think you have done the same thing, could you share your project with me? I am a bit lost in the codes because there are many different scripts and following certain variables is quite hard.

My idea was to create a dataset with 2 category ids (a list, [bulb_number, direction]) for each annotation. Then, using binary cross-entropy loss + sigmoid function in the roi_head (as classification loss), obtain 2 category predictions from the model as given in the input format (since every annotation has 2 labels). However, as I said, there are many other functions/scripts/methods I need to change just to input my custom multi-label dataset (and also need to change loss, metrics etc.). Here is the issue that I opened #3519

To summarize, I will be very happy if you can help me :) Also, my mail address is: yusufcan.simsek@mail.polimi.it
Thank you.

[Hartvi]

Hi, @yusiyohpolimi. Actually what we did in our project was just to refactor and combine some existing datasets into one and instead of multi-labeling as it was easier for our project to have tuples (category_id, material_id) as single outputs instead of separate outputs in a list, like you are trying to do.

I did have the same problem following what is actually going on inside detectron2, since none of the IDEs, such as pycharm and others were able to tell what's being used where. The solution I eventually came to was to have another classification network take the detectron2 outputs as inputs. I cut out the bounding boxes from detectron's output and fed them into a mobilenet trained on the smaller MINC material classification dataset.

In case you want to see some of the stuff I did manage to do, it was training detectron2 with an additional "material" head, but without actually calculating gradients for the additional head: https://github.com/Hartvi/detectron2/tree/master/projects/ipalm
The additional head is a directory deeper in ipalm/material_head.py

[yusiyohpolimi]

Hi, @yusiyohpolimi. Actually what we did in our project was just to refactor and combine some existing datasets into one and instead of multi-labeling as it was easier for our project to have tuples (category_id, material_id) as single outputs instead of separate outputs in a list, like you are trying to do.

I did have the same problem following what is actually going on inside detectron2, since none of the IDEs, such as pycharm and others were able to tell what's being used where. The solution I eventually came to was to have another classification network take the detectron2 outputs as inputs. I cut out the bounding boxes from detectron's output and fed them into a mobilenet trained on the smaller MINC material classification dataset.

In case you want to see some of the stuff I did manage to do, it was training detectron2 with an additional "material" head, but without actually calculating gradients for the additional head: https://github.com/Hartvi/detectron2/tree/master/projects/ipalm The additional head is a directory deeper in ipalm/material_head.py

Thank you for your reply. So if I understood well, you have actually multi-label classification as both category_id and material_id but you divided them as 2 single-label tasks and implemented 2 heads for each. However, I did not get why you do not calculate gradients for the material head. In addition, how did you define the cost function in this case? 2 heads should have the same weights for the cost I think (I could not check yet your code).
Thank you for sharing the code. I will check that. For now, I have to implement as I told, a multi-label detection model (thus I need to change many functions in detectron2 from single label to multi-label). However, your implementation is another good approach for the same task. I may try that if I cannot make other way work.
Could you share your mail with me? If it is okay for you, I may ask you some questions from time to time and do not spam here. Thanks again.

[frederick0291]

Anyone got any lead on this? I am trying to do the same but I am trying to get different number of classes for each superclass. Thanks.

[yusiyoh]

@frederick0291 I suggest you to use double classifier heads consecutively. The first one will classify the superclass, then the second one will classify according to each superclass. Thus, the heads should not be parallel but back to back instead. Your task is also known as hierarchical classification I think.