Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Video support for SAM2 #8

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Conversation

lab176344
Copy link

@lab176344 lab176344 commented Sep 26, 2024

https://imgur.com/a/jIxYTMf

Added the ability to track objects over time with SAM2 and grounding DINO, the idea of it from IDEA-Research Grounded SAM implementation.

@lab176344 lab176344 marked this pull request as draft September 26, 2024 16:32
@lab176344
Copy link
Author

output_video.mp4
Added the ability to track objects over time with SAM2 and grounding DINO, the idea of it from IDEA-Research Grounded SAM implementation.

Approach is based on the idea of running SAM2 Image predictor on each frame and propogating the mask and box prompts. A global data structure is tracking frame by frame change of ids, Ids are combined if the IOU between masks are greater than a threshold, and new Ids are added. It is an expensive approach as you have to run the image propogation is SAM2 for every frame.

@lab176344 lab176344 changed the title WIP - Video support for SAM2 Video support for SAM2 Sep 26, 2024
@lab176344 lab176344 marked this pull request as ready for review September 26, 2024 16:51
@lab176344
Copy link
Author

@capjamesg let me know what you think

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants