sam2 real time inference?

[Karmik25]

can someone explain to me how to use sam2 for real-time inference, such that the model can predict objects while my webcam is open.

[heyoeyo]

The existing code doesn't support live/streaming video. Instead you'd have to first record a video on webcam and then process that video using the existing video predictor and/or web demo. It is possible to handle streaming video by modifying the code. It's mostly a matter of re-implementing the existing frame loop outside of the sam2_video_predictor.py script, and handling the frame reading (e.g. from a webcam) in that new loop.

If you don't want to implement this manually, there is an existing repo that has done this:
https://github.com/Gy920/segment-anything-2-real-time
I also have a script that works with webcams (though it's worth noting the repo above has code much more similar to the original, in case that's important):
https://github.com/heyoeyo/muggled_sam?tab=readme-ov-file#run-video-or-webcam

[277845265]

https://www.bilibili.com/video/BV1TNkEYrEnb/?spm_id_from=333.33.top_right_bar_window_dynamic.content.click&vd_source=1aa85c9fa1298c2b3a033e3bd0932f3a