-
Notifications
You must be signed in to change notification settings - Fork 7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC] Rotated Bounding Boxes #2761
Comments
Hi, This is a good thing to consider adding to torchvision, but as you mentioned it might involve quite a few changes so it won't be easy to do. For those interested in having support for rotated bounding boxes, please "thumbs up" the main post. |
Hi @fmassa , @oke-aditya , I noticed that there are implementations such as
Maybe this is an opportune moment to do this? cc @ppwwyyxx |
Let's discuss this @zhiqwang There are a few points about this what I think.
For Rotated boxes, we would need to implement these common operations. Both torchvision and detectron2 represent bounding boxes as Detectron2 represents rotated boxes as Some proposals for ops
I think Rotated bounding box is a more generic case of non rotated case. Case where angle is 0 represents normal boxes. (Please correct if I'm wrong).
My thoughts might be naive. I guess this needs better thoughts. |
Hi @oke-aditya , the points you highlight and the PR #2710 makes me clear of the implementation of boxes conversion in
I think that one reason both And as @vfdev-5 , @pmeier and @oke-aditya mentioned in #2753 :
But if there are operations like |
Let's have more thoughts from @fmassa |
Displaying bounding boxes for rotated rectangles.Given the rotated blue rectangular region, how to show the label? If the labels are rotated as well, it will be quite challenging to rotate your head everytime to read the labels. On the other hand, if the label is shown withing the non-rotated bounding box (green), which also contains the target rotated rectangle (blue), then it is clear where the rotated bounding box is and is also easy to read. So, each bounding box then is composed of two things:
But, any format specification should be adequate to determine both bounding boxes and draw them as well. |
Are there plans to port the rotated ops from detectron2 to torchvision? |
Any update on this? This would be really beneficial to the remote sensing community since, in RS imagery, objects of various sizes & orientations can be clustered together, and rotated object detectors generally outperform default detectors in those cases. If the rotated ops in |
@pmeier I guess with the new Features API this may be possible in near future? |
Yes, in theory we could have
This actually rotates the coordinates and thus the result is a "proper" (for the lack of a better term) bounding box. Thus, the bounding box may change shape. Excuse my ignorance here, but is rotation invariance that important? What about all the other transformations that change the shape of bounding boxes, like affine, elastic, perspective, and so on? From my, admittedly limited, perspective it seems weird that rotation is singled out here. Or to put differently, having a |
@pmeier |
Maybe I misunderstood the context here: are we talking about rotated bounding boxes at the transforms or at the model stage? |
At the modelling stage. You would need rotated variants for NMS, IOU calculation, RPNs, ROIAlign etc. Just like in detectron2. |
Ok, in that case the prototype features won't really help here as the models will only work with plain tensors. Basically, in the transform stage the I don't know what the priority for this would be. From what I understand is that our roadmap for H2 this year is already full, but I'll let @datumbox chime in here. |
Since I had created the issue, my initial idea was that it isn't exactly to do with transforms but we would need a few small transforms like say horizontal flip etc that need both image and the box. But other than that mostly it's on the ops side. |
This is a bigger piece of work that we should consider after H2. Philip is right to say that our roadmap is currently full and this is quite a sizeable piece of work to squeeze in. As Aditya said, this is not just about adding support for transforms but also for operators. I would be in favour of reviewing the support of this as a whole as we would need to provide Transforms, Ops and Models that are compatible. IMO we should avoid adding more partial support for features in TorchVision; we already have quite a few things that are partially supported (Keypoints, Video Decoding etc). We are currently investing on closing any high priority gaps on the already supported tasks. Once this is done, we should stack rank big proposed features to see which of them are worth investing into. The situation I would like to avoid is having some support of Rotated BBoxes in some Transforms/Ops/Models but not on others. This is likely to make our Detection API more complex and harder to revamp. |
🚀 Feature
A bit unsure about this feature.
Support Rotated Bounding Boxes in Torchvision.
Motivation
There is recent research on Rotated Bounding boxes which provides better detection results. I am not able to find highly cited results but a few of them are
I'm not sure for more papers. I think this is slightly new topic, and needs a bit more research.
Pitch
If possible we can also support Rotated models. In my opinion it might not be very feasible.
Since it will take a lot of time and maintenance for these models.
Alternatives
Currently, Detectron2 has great support for all the above. These operations are implemented in C++.
Additional context
Looks tricky and challenging. I guess it might also be too early to think about this. I think it needs a bit more research and some baseline.
cc @pmeier
The text was updated successfully, but these errors were encountered: