Duplicated instance segmentation with different classes

[gibosler]

Hi community!
I'm using a custom trained instance segementation model based on X101 model zoo to segment an "high dense" picture with a lot of objects of the same type (vegetables). Now we are trying to add a new label to segment vegetables of "type1" and "type2" but on inference we get two masks on the same object with different class. There's a way to avoid this?
I'm trying to post-process plygons with opencv but the segmentation polygon (computed from mask) is not the same so it's difficult to find without a lot of compute expensing procedure.

Thanks in advance for your help!

[RGuven]

Hi, @gibosler

1-) Did you try to increase the prediction threshold? With this threshold you can filter out the lower prediction one.

cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.9 # set a custom testing threshold

#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-#-

2- From bitmask_to_polygon conversation

If you really need to do this, you can use this function

OpenCV - '3.4.18' version

`

def bitmask_to_polygons(bit_mask, epsilon=0.004):

ret, thresh = cv2.threshold(bit_mask, 0.5, 255, 0)
image, contours, hierarchy = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
# Sort the contours by their areas
sorted_contours = sorted(contours, key=cv2.contourArea, reverse=True)
# Select the largest contour
largest_contour = sorted_contours[0]

approx = cv2.approxPolyDP(largest_contour, epsilon * cv2.arcLength(largest_contour, True), True)
return np.squeeze(approx)

`

then use this function here (you can fine tune the epsilon constant depending on your problem)

`
#convert from bitmap mask to polygonal
bit_masks = instances.pred_masks.cpu().numpy().astype("uint8")

polygons = [bitmask_to_polygons(bit_mask) for bit_mask in bit_masks]
`

[gibosler]

Hi and thank you for your reply.
Yes the threshold was the first parameters i was playing on but if i reduce the threshold i reduce a lot the object detected. With a reasonable threshold (0.7-0.8) i get double class object with similar mask but not the same.
About the polygons conversion i'm actually using more or less the same strategy but my idea was to process polygons to remove duplicated once, maybe there's a better way to do this on masks? Working to find overallapping polygons with near centroid is a bit compute expensive..

The double label detection is a normal behaviour on inference? Can be caused by an imbalance on training dataset? I'm actually investigating on RepeatFactorTrainingSampler...

Thank yoy for you help!