Feature Request: Extend SAM Interaction in CVAT with Bounding Box Inputs

[descilla]

Hi,

Firstly, I'd like to thank you for the rapid incorporation of SAM into CVAT and the newly added ability to continue image interaction directly in the browser. This has brought about a tremendous boost in performance.

As previously commented by me in another issue, I am interested in extending the interaction possibilities with SAM.

The playbook provided by Facebook Research presents several options that I would also like to use in CVAT.

Currently, only the use of positive and negative points is possible (as far as I understand). As shown in the playbook, SAM also offers the ability to use boxes exclusively or a combination of boxes and positive and negative points. As seen at the very bottom of the playbook, multiple boxes can be inputted as well.

@bsekachev This issue is a response to your request for a dedicated issue on the topic of bounding boxes and why I believe it would be beneficial to incorporate them into CVAT.

In numerous use cases of mine, I've been able to achieve significantly quicker and more robust results with the use bounding boxes or the combination of bounding boxes and points than with points alone.

If implementing my request would be a piece of cake for anyone here, I'd be thankful if you could help me out. Otherwise, I will try to find time to implement this feature myself. However, I must note that I'm more of a C++ person, and my last activities with TypeScript were a few years ago. ;)

[descilla]

I've implemented a proof of concept today, as you can see in my mentioned repository. I've used the startswith_box functionality and override it dynamically by clicking a switch in the ui:

This seems to be suitable for a proof of concept. But it lacks of consideration whether the selected model supports box-only or not.
Furthermore it is only possible to draw a single box for each annotation object.

However, I was able to meet my requirements and SAM can be used in de following three ways now:

• Draw boxes only to annotate objects
• Draw a box followed by positive and negative points
• Work with the points only

[Once88]

Is there any progress?

[AHansoe]

Wondering about the same ^