TIDE outputs vs. what pycocotools outputs

[kdk2612]

Hi, first things first.. This lib is amazing and helped a lot to understand the errors related to the detections.
I was doing using this project for the initial evaluation but since there was no support for Recall, I decided to use the pycocotools for evaluation as well.

Now, during the comparison I got different results for the AP[0.50-0.95]
pycocotools gives- 0.460
TIDE gives - 41.33

Also,
pycocotools gives - AP @ 50 = 0.804
TIDE gives - AP @ 50 = 70.93 (extracted from the summary table)

I was wondering where the difference comes from, exploring how the TP, FP, FN are calculated for now.

[dbolya]

Hi, and thanks for making this issue. TIDE should have 100% parity with pycocotools when it comes to mAP calculation, so this difference shouldn't be happening.

Is this on COCO or a custom COCO-style dataset? I'm thinking if it's a custom dataset, there may be some edge case that TIDE and pycocotools are handling differently which results in the different mAP.

[kdk2612]

Hi, This is on custom COCO-style dataset. I am converting the data into the COCO format to run the evaluation. I identified the issue anyways. For using the pycocotools I had set the "useCats" to 1, which ignores the category labels. Seems to be working the same after setting it to 0.

[dbolya]

Nice! Then I'll close this issue.

[BartvanMarrewijk]

Maybe also good to mention that I had the same problem, but my solution was different. If your custom coco format annotation id starts with 0, then this annotation will not be taken into account in the coco format!!! The easiest solution is just make sure that the annotation id starts with 1. For more information see following git issue #cocodataset/cocoapi#507

[willyd]

I had a similar issue where both my groundtruth and predictions where in the COCO annotations format so I loaded both with tidecv.datasets.COCO. This ignores the score field in the detections which have to be loaded using tidecv.datasets.COCOResult.