SAM model takes too long to return remotely with low internet bandwidth

[vadimstasiev]

My actions before raising this issue

• Read/searched the docs
• Searched past issues

I have installed CVAT with the SAM model successfully on a server, and made sure it runs on the GPU, which is an RTX 3090, my upload speed to this server is 35Mbps and download speed is about 6Mbps,

Steps to Reproduce (for bugs)

1. Load an image with 4000x3000 resolution, 1.3MB
2. Click wand and toggle on convert masks to polygons, segment anything model.
3. Click element in the image
4. Took 40 seconds and shows 24.1MB downloaded in the network console (this time, sometimes takes longer), and clearly shows GPU usage during this process
5. Click the wand again to finalize that selection
6. Add a new element, same as before using SAM model
7. Took 1m13sec and 24.1MB downloaded again
8. Tried it locally on the actual server itself and again, also returns around 24MB but this time it only took 3 seconds no matter what I selected.

Expected Behaviour

It shouldn't take this long to return the result when using remotely.

Current Behaviour

It takes a long time when using it remotely, and 24MB seems to me far too much for a couple of points, but even so at 35mbps it should only take about 6 seconds

Context

My connection to this machine is using Tailscale and the network connection is a bit poor but even so it shouldn't take that long it should only take about 6 seconds in theory,

[bsekachev]

Hello,

You can try this patch #6019. The problem is solved there.
It is not finalized yet, but in quite good state.

[vadimstasiev]

Hi, thank you for your response, I tried the latest version of that branch and for the most part it seemed that the points weren't being returned when I clicked on something, and at times it returned some points but with the wrong selection, and I could still see 24MB being transferred.
I also tried your earlier version from the commit 3 weeks ago, when you did the PR and with that version when I use the SAM model I just get the error in the ui: "TypeError: Cannot read properties of undefined (reading 'length')"

[bsekachev]

Hello again, a couple of moments.

Client part should be built with additional argument, like CLIENT_PLUGINS='plugins/sam_plugin' yarn start:cvat-ui. I forgot to mention this, sorry. Actually it was possible only in dev environment. But now I added default value for the argument to the branch above and it should work without an argument.

Serverless function is also must be rebuild with ./deploy_gpu.sh pytorch/facebookresearch/sam/nuclio/.

[vadimstasiev]

Hi, thank you for the follow up, I have tried it again but I got the same error in the ui: "TypeError: Cannot read properties of undefined (reading 'length')"

Steps to reproduce

git clone https://github.com/opencv/cvat

git checkout bs/sam_ui

docker compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d

nuctl deploy --project-name cvat \ --path serverless/openvino/dextr/nuclio \ --volume pwd/serverless/common:/opt/nuclio/common \ --platform local

nuctl deploy --project-name cvat \ --path serverless/openvino/omz/public/yolo-v3-tf/nuclio \ --volume pwd/serverless/common:/opt/nuclio/common \ --platform local

cd serverless && ./deploy_gpu.sh pytorch/facebookresearch/sam/nuclio/

nuctl get function
Output:

 NAMESPACE | NAME                           | PROJECT | STATE | REPLICAS | NODE PORT
 nuclio    | openvino-dextr                 | cvat    | ready | 1/1      | 32771
 nuclio    | openvino-omz-public-yolo-v3-tf | cvat    | ready | 1/1      | 32772
 nuclio    | pth-facebookresearch-sam-vit-h | cvat    | ready | 1/1      | 32773

Then using the sam model in the ui I get:
"TypeError: Cannot read properties of undefined (reading 'length')"

[bsekachev]

@vadimstasiev

docker compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d

This command does not rebuild CVAT. Need add --build argument and dev yaml file to build not the latest release.

Try this:

docker compose -f docker-compose.yml -f docker-compose.dev.yml -f components/serverless/docker-compose.serverless.yml up -d --build

Also, please share the commit hash you are trying to build from.

[vadimstasiev]

Hi, yeah, that worked, and this now works so much better, it takes a little longer the first time but then for any succeeding ones it works instantly, thank you

[bsekachev]

Great, the first time image embeddings are calculated.

Keep look on this PR, it is improving continiously.