SAM model doesn't always get the border pixels

[davodogster]

Hi I'm running SAM on CVAT.ai free version and often it won't segment the border pixels even if I add point prompts for them.

Also, does CVAT.ai run SAM on the GPU or CPU? We are considering purchasing the Cloud Pro version for our organisation.

Thank you, Sam

[bsekachev]

@davodogster

CVAT.ai runs SAM on GPU.

[davodogster]

Thanks! @bsekachev and do you know why SAM doesn't segment the pixels along the border of the image? Sometimes it misses the top/bottom borders and other times the left/right as shown in the images above.

[bsekachev]

So, I have two ideas here.

1. Apparantly part of reasons related to roundings issues between integers and floats, low and high resolution images. I tried to optimize some operations in ONNX decoder in this PR Accelerated SAM decoder, fixed serverless for EXIF rotated images #6275 and I believe it works a little bit better. Some issues with border pixels disappeared.
2. But not all the issues disappeared. The neural network itself does not guarantee returning masks where points are located.
   Even unchanged by other operations output from the decoder has negative values on the top left pixel (where marker is located BTW):

Also, here you can see, that on the same image, another element can be fully covered by mask on border pixels:

Finally on the latest screenshot we have two images with the same size, but different content obviously (the right one picture is just filled by blue color). You can see how the right side of the right image easily covered by mask. At the same time right side of the left image hardly ever can be covered.

[davodogster]

@bsekachev Thanks!!