Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Detections] - enable from_mmdetection to extract tracker_id from MMDetection result #1381

Open
2 tasks done
rolson24 opened this issue Jul 18, 2024 · 12 comments
Open
2 tasks done
Assignees
Labels
api:detection enhancement New feature or request

Comments

@rolson24
Copy link
Contributor

Search before asking

  • I have searched the Supervision issues and found no similar feature requests.

Description

This is somewhat similar to #1308 in that it would be modifying from_mmdetection() to support TrackDataSamples. This would allow for easy integration with the MMTracking library, which has several trackers implemented in it, as well as integration with the Match and Segment Anything (MASA) model.

Use case

No response

Additional

I imagine it could be pretty simple and would only require checking if the input object has the pred_track_instances attribute, and then it could simply add the tracks with their track ID's into a Detections object.

Are you willing to submit a PR?

  • Yes I'd like to help by submitting a PR!
@rolson24 rolson24 added the enhancement New feature or request label Jul 18, 2024
@SkalskiP
Copy link
Collaborator

Hi @rolson24 I just managed to plug in your MASA code into my football demo. I'd never be able to do it without your help. As for MASA: not bad, but I was hoping for a slightly better result. Would that allow us to plug in all MMTracking models?

d82e7a_5-masa-result-rainbow-optimized.mp4

@rolson24
Copy link
Contributor Author

I don't think so. It seems like MASA is built on the trackers in the MMDet library, which are set up slightly different than the trackers in MMTracking. The trackers in MMTracking seem to output dicts of numpy arrays, while the trackers in MMDet output objects called TrackDataSample. Also I tried to test MMTracking and it seems that it does not support the newer versions of their own libraries. (Max MMCV version is 2.1.0 and newest is 2.2.0, and max MMDet version is 3.0.0 and newest is 3.3.0) So in order to use MMTracking with any recent version of CUDA and pytorch, a user would have to compile their own binaries. MMTracking has not had a commit since last April, so it seems like OpenMMLab doesn't have the bandwidth to support it right now.

So maybe not worth trying to support it if its a dead library anyway.

@rolson24
Copy link
Contributor Author

As for the MASA performance, I also noticed it was pretty bad at handling occlusion and creating new track IDs. I looked a bit into the code and it seems like the actual tracking is a very simple tracking algorithm with no motion modeling. I think MASA has some pretty good potential if motion modeling was added, and a more strict matching pipeline was incorporated. I think the real benefit of MASA is its ability to plug into foundational open world detectors, and doesn't hold up as well in specific applications. I might try to play around and see if I can integrate it with a better tracker like BoTSORT with MASA as the ReID model.

@rolson24
Copy link
Contributor Author

Maybe we should wait on the MASA integration because the library isn't quite there yet, and the tracking output will probably change.

If we wanted to integrate a strong tracking library that is still being maintained, I think BoxMOT could be a pretty good option. It supports some pretty good trackers and several ReID models and has a clean install process. I am also in the process of adding some new SOTA trackers to it.

@SkalskiP
Copy link
Collaborator

@rolson24 If we disregard MASA for the moment, do other trackers in MMTrack have a consistent output format?

@rolson24
Copy link
Contributor Author

Yes. They output a dict containing track_results and det_results which are each dicts containing bbox_results and mask_results which in turn are lists of 2d numpy arrays where each array corresponds to a class. Track_ids are in bbox_results[0] and bboxes are in bbox_results[1].

The main issue with MMTracking is that it hasn't been updated since 2022, and so none of the dependancies are updated to work with any of the newer libraries.

It seems like all of the trackers from MMTracking have now been merged into MMDetection, which uses the same output format as MASA, and MMDetection has least had a release this year, so the dependancies are much more recent.

@SkalskiP
Copy link
Collaborator

@rolson24 in that case, could we expand from_mmdetection to consume tracking results as well?

@rolson24
Copy link
Contributor Author

Ya it would be pretty easy. I can work on it right now.

@SkalskiP
Copy link
Collaborator

@rolson24 that's awesome! thanks a lot! 🙏🏻

@SkalskiP SkalskiP changed the title Add support for MMDetection tracking objects [Detections] - enable from_mmdetection to extract tracker_id from MMDetection result Jul 25, 2024
@rolson24
Copy link
Contributor Author

rolson24 commented Jul 25, 2024

Ok I made the change and have been testing it, but the trackers and object detectors in MMDetection are not perfectly set up and the pre-trained weights seem to perform very badly on my test video.

from faster-RCNN and DeepSORT pretrained:
image

I can open the PR, but I the experience of using the trackers is less than ideal right now.

@SkalskiP
Copy link
Collaborator

Hm... I'm becoming increasingly convinced that tracking is the domain we should explore. If a library like MMDetection does not have trackers that work well out of the box, we could do something useful here.

@mikel-brostrom
Copy link

I am open the lend a hand here if you plan to integrate BoxMOT into Supervision @SkalskiP . Reach out with any questions you may have.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api:detection enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants