Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding multiple classification heads to train #3321

Open
Shobhit20 opened this issue Aug 4, 2021 · 9 comments
Open

Adding multiple classification heads to train #3321

Shobhit20 opened this issue Aug 4, 2021 · 9 comments
Labels
enhancement Improvements or good new features

Comments

@Shobhit20
Copy link

I am trying to build a model with multiple classification heads. MaskRCNN already has 3 heads - mask, box and class
How can we add another custom head such as another classification head. Is there a possibility to extend the current implementation of detectron to cater to this problem statement?

Say for example I have a problem statement in which I am supposed to infer the location, bbox, class, color of fruit from an image.
For inferring the class and the color one way can be to club the two into one class making a combination of each class with each color but that creates somewhat imbalanced datasets. Can I rather do multiple classification heads where one predicts the class and the other predicts the color of the crop?

@Shobhit20 Shobhit20 added the enhancement Improvements or good new features label Aug 4, 2021
@Hartvi
Copy link

Hartvi commented Aug 9, 2021

I am trying tackle exactly the same problem, the only difference being that I do not necessarily need a brand new head, which I have seen examples of in issue #841, but for the model to have another instance of a mask head, such as mask_head.py.

In our project we are running the basic Detectron2 with tuples and it works very well. We currently have object categories as a series of tuples (box, paper), (box, plastic), but we would like to separate them into (box)+(paper)/(plastic).

If I am not mistaken, then for creating a new head you need a regular nn.Module or nn.Sequential class that just implements the forward function, e.g. mask_head and deeplab.

Then in roi_heads.py we should implement the StandardROIHeads or the super class ROIHeads and add the new head to it.

If I wanted to simply add another mask-type head and run Detectron2, how would I go about doing that? Would I need to make a wholly new project like DensePose or just add custom scripts to load the desired heads? And what are the places that would need to be modified? Because I see that there are many references to the heads in lots of different .py places.

@Hartvi
Copy link

Hartvi commented Aug 12, 2021

In addition to my previous comment, one needs to create a yaml file in the detectron2/configs folder to include additional heads.

I have dived into previous issues and documentation and here is a compilation of related links I've found

@Shobhit20
Copy link
Author

@Hartvi thanks for putting all this together
Will try to add multiple heads using these resources and in the meantime, if I stumble across any resources for adding a custom mask head, will keep you posted.

@eaedk
Copy link

eaedk commented Sep 13, 2021

Hello, how to finetune on my own keypoints dataset ??? I don't find any tutorial that works for me. I need help please. I have a dataset with 4 keypoints per image

@yusiyohpolimi
Copy link

In our project we are running the basic Detectron2 with tuples and it works very well. We currently have object categories as a series of tuples (box, paper), (box, plastic), but we would like to separate them into (box)+(paper)/(plastic).

@Hartvi Hello, I am trying to run Detectron2 with category_id = [bulb_number, direction] which is not supported as default. I think you have done the same thing, could you share your project with me? I am a bit lost in the codes because there are many different scripts and following certain variables is quite hard.

My idea was to create a dataset with 2 category ids (a list, [bulb_number, direction]) for each annotation. Then, using binary cross-entropy loss + sigmoid function in the roi_head (as classification loss), obtain 2 category predictions from the model as given in the input format (since every annotation has 2 labels). However, as I said, there are many other functions/scripts/methods I need to change just to input my custom multi-label dataset (and also need to change loss, metrics etc.). Here is the issue that I opened #3519

To summarize, I will be very happy if you can help me :) Also, my mail address is: yusufcan.simsek@mail.polimi.it
Thank you.

@Hartvi
Copy link

Hartvi commented Oct 6, 2021

Hi, @yusiyohpolimi. Actually what we did in our project was just to refactor and combine some existing datasets into one and instead of multi-labeling as it was easier for our project to have tuples (category_id, material_id) as single outputs instead of separate outputs in a list, like you are trying to do.

I did have the same problem following what is actually going on inside detectron2, since none of the IDEs, such as pycharm and others were able to tell what's being used where. The solution I eventually came to was to have another classification network take the detectron2 outputs as inputs. I cut out the bounding boxes from detectron's output and fed them into a mobilenet trained on the smaller MINC material classification dataset.

In case you want to see some of the stuff I did manage to do, it was training detectron2 with an additional "material" head, but without actually calculating gradients for the additional head: https://github.com/Hartvi/detectron2/tree/master/projects/ipalm
The additional head is a directory deeper in ipalm/material_head.py

@yusiyohpolimi
Copy link

Hi, @yusiyohpolimi. Actually what we did in our project was just to refactor and combine some existing datasets into one and instead of multi-labeling as it was easier for our project to have tuples (category_id, material_id) as single outputs instead of separate outputs in a list, like you are trying to do.

I did have the same problem following what is actually going on inside detectron2, since none of the IDEs, such as pycharm and others were able to tell what's being used where. The solution I eventually came to was to have another classification network take the detectron2 outputs as inputs. I cut out the bounding boxes from detectron's output and fed them into a mobilenet trained on the smaller MINC material classification dataset.

In case you want to see some of the stuff I did manage to do, it was training detectron2 with an additional "material" head, but without actually calculating gradients for the additional head: https://github.com/Hartvi/detectron2/tree/master/projects/ipalm The additional head is a directory deeper in ipalm/material_head.py

Thank you for your reply. So if I understood well, you have actually multi-label classification as both category_id and material_id but you divided them as 2 single-label tasks and implemented 2 heads for each. However, I did not get why you do not calculate gradients for the material head. In addition, how did you define the cost function in this case? 2 heads should have the same weights for the cost I think (I could not check yet your code).
Thank you for sharing the code. I will check that. For now, I have to implement as I told, a multi-label detection model (thus I need to change many functions in detectron2 from single label to multi-label). However, your implementation is another good approach for the same task. I may try that if I cannot make other way work.
Could you share your mail with me? If it is okay for you, I may ask you some questions from time to time and do not spam here. Thanks again.

@frederick0291
Copy link

Anyone got any lead on this? I am trying to do the same but I am trying to get different number of classes for each superclass. Thanks.

@yusiyoh
Copy link

yusiyoh commented Dec 6, 2021

@frederick0291 I suggest you to use double classifier heads consecutively. The first one will classify the superclass, then the second one will classify according to each superclass. Thus, the heads should not be parallel but back to back instead. Your task is also known as hierarchical classification I think.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improvements or good new features
Projects
None yet
Development

No branches or pull requests

6 participants