CoTracker3: Better Point Tracking for Frigate?

[glossyio]

I wanted to plant the seed here of improved motion tracking algorithms. I was perusing some AI articles and came across https://cotracker3.github.io/ that has some very cool methodology for tracking objects, even as they become occluded by other objects.

My mind boggles at the implementation complexity vs. the current Frigate motion detection and the necessary compute requirements. However, the additional capabilities seem like they are worth a look.

Am I right in thinking this would have some value over the current motion detection in Frigate? (Or are there even better ones?)
If it were valuable, would it even be a potential item for a far-off roadmap?

Food for thought!

[NickM-27]

Frigate already has an object tracker, using norfair

I think there would need to be a good reason to consider implementing another solution as it took many months of debugging during the beta to fix different user issues and the object tracking works very well in most cases. If anything, we will look at using embeddings to improve the existing tracking implementation

[NickM-27]

Frigate uses a simple motion detector and uses object detection to track objects, using motion based tracking would be going in the opposite direction.

[glossyio]

Good points, well taken. I really appreciate the thoughtfulness of the current capabilities. I have a passable understanding of Frigate's object detection + motion-based tracking, and I think it's a smart process! Anything new would definitely have to improve on that. I often see the flashy new capabilities and think of where it can be used, and as a corollary, I often think of ways to improve Frigate with new tech.

The current implementation works really well for me, so no big complaints. I have edge cases where I get many multiple counts if an object is occluded for a short time; e.g. a parked car on the other side of the road is occluded for a few seconds/mins by a delivery box truck, it will often end the parked car object and start it new again (counting the parked car at least twice). But it's a minor complaint that has been covered in the discussions, so I wouldn't be out of options even with the current implementation.

[NickM-27]

right, the occlusion would likely be handled by using embeddings so we have a visual way to distinguish between two objects