Cannot install SAM in CVAT | macOS

[DP1701]

Hello together,

My actions before raising this issue

I have read and followed the following instructions. CVAT, Nuclio and Servless support are installed and working.
SAM in CVAT
CVAT with servless support

Steps to Reproduce (for bugs)

1. Installation of CVAT (Docker previously installed) with docker compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d --build
2. Create a superuser with: docker exec -it cvat_server bash -ic 'python3 ~/manage.py createsuperuser'
3. Logging into CVAT (checking whether everything is working) -> Works for me
4. Nuclio installation with: sudo chmod +x nuctl-<version>-linux-amd64 sudo ln -sf $(pwd)/nuctl-<version>-linux-amd64 /usr/local/bin/nuctl
5. Install SAM with: cd serverless && ./deploy_cpu.sh pytorch/facebookresearch/sam/nuclio/

Expected Behaviour

After the installation of SAM, the account line should actually show the status ready.

Current Behaviour

But I see this:

cd serverless && ./deploy_cpu.sh pytorch/facebookresearch/sam/nuclio/
[+] Building 1.2s (9/9) FINISHED                           docker:desktop-linux
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 418B                                       0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/openvino/ubuntu20_runtime:2022  1.2s
 => [1/4] FROM docker.io/openvino/ubuntu20_runtime:2022.3.0@sha256:0cdf91  0.0s
 => [internal] load build context                                          0.0s
 => => transferring context: 66B                                           0.0s
 => CACHED [2/4] RUN apt-get update     && apt-get -y --no-install-recomm  0.0s
 => CACHED [3/4] RUN pip install --no-cache-dir opencv-python-headless pi  0.0s
 => CACHED [4/4] COPY model_loader.py shared.py /opt/nuclio/common/openvi  0.0s
 => exporting to image                                                     0.0s
 => => exporting layers                                                    0.0s
 => => writing image sha256:a0aa876f787609eff990b86b907ed7a096896e604dd24  0.0s
 => => naming to docker.io/library/cvat.openvino.base                      0.0s

What's Next?
  View summary of image vulnerabilities and recommendations → docker scout quickview
23.07.07 10:34:41.702                     nuctl (I) Project created {"Name": "cvat", "Namespace": "nuclio"}
./deploy_cpu.sh: line 15: shopt: globstar: invalid shell option name

Possible Solution

I have already tried to update the bash and activated the globstar option. It does not work.
Instruction

Context

Your Environment

• Git hash commit (git log -1):

git log -1
commit 4b179be1ee98caf4cc701a864ca6ec7bcecc7805 (HEAD -> develop, origin/develop, origin/HEAD)
Author: Andrey Zhavoronkov <andrey@cvat.ai>
Date:   Wed Jul 5 19:42:17 2023 +0300

    Updated CHANGELOG.md, bump version (#6426)

• Docker version docker version (e.g. Docker 17.0.05):

Client:
 Cloud integration: v1.0.35
 Version:           24.0.2
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        cb74dfc
 Built:             Thu May 25 21:51:16 2023
 OS/Arch:           darwin/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.21.1 (114176)
 Engine:
  Version:          24.0.2
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       659604f
  Built:            Thu May 25 21:52:17 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

• Are you using Docker Swarm or Kubernetes?

Docker

• Operating System and version (e.g. Linux, Windows, MacOS):

macOS 12.6.7

• Code example or link to GitHub repo or gist to reproduce problem:

• Other diagnostic information / logs:

  Logs from `cvat` container

[bsekachev]

Hi @DP1701

Try to deploy without the script:

# from serverless directory:
nuctl deploy --project-name cvat --path "`pwd`/pytorch/facebookresearch/sam/nuclio/" --file "`pwd`/pytorch/facebookresearch/sam/nuclio/function.yaml" --platform local

[DP1701]

Hi @bsekachev

The installation worked. However, I get the following message in CVAT when I want to use SAM.

This happens when I try to work with the interactor. Do you have any idea what this could be?

[azhavoro]

Probably it is a problem on the nuclio side, try deleting the model first

nuctl delete function pth-facebookresearch-sam-vit-h

and deploy as previously mentioned.

[DP1701]

Now I get the following message:

$ nuctl deploy --project-name cvat --path "`pwd`/pytorch/facebookresearch/sam/nuclio/"  --file "`pwd`/pytorch/facebookresearch/sam/nuclio/function.yaml" --platform local

Error - Project does not exist
    ...//nuclio/pkg/platform/abstract/platform.go:401

Call stack:
Project existence validation failed
    ...//nuclio/pkg/platform/abstract/platform.go:401
Failed to validate a function configuration
    /nuclio/pkg/platform/local/platform.go:1255
Failed to enrich and validate a function configuration
    /nuclio/pkg/platform/local/platform.go:166

[DP1701]

I am in the .../cvat/serverless folder

[DP1701]

Now it works for me. Weird. Here's what I did:

1. CVAT completely reinstalled with servless support
2. cd serverless && ./deploy_cpu.sh pytorch/facebookresearch/sam/nuclio/
3. nuctl deploy --project-name cvat --path "pwd/pytorch/facebookresearch/sam/nuclio/" --file "pwd/pytorch/facebookresearch/sam/nuclio/function.yaml" --platform local
4. nuctl delete function pth-facebookresearch-sam-vit-h
5. nuctl deploy --project-name cvat --path "pwd/pytorch/facebookresearch/sam/nuclio/" --file "pwd/pytorch/facebookresearch/sam/nuclio/function.yaml" --platform local

After that, I increased the RAM memory at Docker to 16 GB.

If I stop at point 3, I get the error message "Interaction error occured".

[sawhney-medha]

Hey @DP1701
I am running into the same error as you but still not able to get it to work. Do you also have an M1 Mac? When do you install nuclio?

[sushilkhadkaanon]

I'm getting this error. Please help @DP1701
My system is Mac M1.

[sushilkhadkaanon]

@DP1701 I somehow manged to follow you footsteps. But I'm still getting Interaction error occurred
Request failed with status code 503. It is not working for me

[DP1701]

@sushilkhadkaanon Have you increased the amount of RAM allowed in the Docker Engine? The default is 8GB but I had to increase it to 16GB.

[DP1701]

@sawhney-medha I installed nuclio before I installed cvat. I have an Intel-based Mac.

To install nuclio, check out this guide.
https://opencv.github.io/cvat/docs/administration/advanced/installation_automatic_annotation/

[SidPad03]

@sushilkhadkaanon I used "darwin" instead of "linux". It seemed to work. But now I ran into a new issue

[SidPad03]

@DP1701 did you figure out how to get shopt: globstar: to work?

[apple2373]

I've finally managed to get SAM running on an M1 Max MacBook Pro. Below are a few tips to help you achieve the same:

1. Setting Up Docker RAM:

   • I set the Docker RAM to 32GB. Though I've seen recommendations suggesting that 16GB is sufficient (and it is), I went with 32GB since my machine has 64GB RAM. By default, Docker is set to use 8GB, which is inadequate. The CPU limit I used was 10.

2. Starting with Serverless Mode:

   • First, stop all docker instances:

     docker stop $(docker ps -q)
     

   • Then, start with serverless mode:

     docker compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d
     

3. Install the Appropriate nuctl (1.11.24-darwin-amd64):

   • Use the following commands:

     wget https://github.com/nuclio/nuclio/releases/download/1.11.24/nuctl-1.11.24-darwin-amd64 
     mv nuctl-1.11.24-darwin-amd64  nuctl
     chmod +x nuctl
     sudo mv nuctl /usr/local/bin/
     nuctl

   • If you encounter the error /usr/local/bin/nuctl: cannot execute binary file, it means the installation wasn't done correctly.

4. Handling PyTorch Compatibility Issue:

   • The latest PyTorch version (2.1) is incompatible with Docker on Apple Silicon with Segmentation Fault. Therefore, you need to manually downgrade.
   • Modify this line here to:

     value: pip3 install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2  opencv-python pycocotools matplotlib onnxruntime onnx
     

   • Source: https://stackoverflow.com/questions/77290003/segmentation-fault-when-using-sentencetransformer-inside-docker-container

5. Manual Deployment Steps:

   • Avoid using deploy_cpu.sh. Instead, follow these manual deployment steps:

     cd serverless
     sudo nuctl deploy --project-name cvat --path "`pwd`/pytorch/facebookresearch/sam/nuclio/" --file "`pwd`/pytorch/facebookresearch/sam/nuclio/function.yaml" --platform local

6. Resetting the Setup:

   • If you wish to start afresh, stop the Docker instance and delete the nuctl function.

     docker stop <instance id of cvat.pth.facebookresearch.sam.vit_h>
     nuctl delete function pth-facebookresearch-sam-vit-h -n nuclio

   • If the above doesn't work, navigate to localhost:8070 using Chrome and manually delete the cvat function.
   • After performing the above steps, nuctl get function should return nothing.

7. Known Issue:

   • I encountered the error below but decided to ignore it since SAM still functions correctly:

     Interaction error occurred
     I.Mat is not a constructor
     

[zhanghaihangandliuli]

Please tell me how to deal with this error
Error - exit status 1
/nuclio/pkg/cmdrunner/shellrunner.go:114

Call stack:
stdout:

stderr:
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile.processor
#1 transferring dockerfile: 1.45kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/ubuntu:22.04
#2 DONE 1.4s

#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s

#4 [ 1/13] FROM docker.io/library/ubuntu:22.04@sha256:adbb90115a21969d2fe6fa7f9af4253e16d45f8d4c1e930182610c4731962658
#4 DONE 0.0s

#5 [ 2/13] WORKDIR /opt/nuclio/sam
#5 CACHED

#6 [ 3/13] RUN apt-get update && apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6
#6 CACHED

#7 [internal] load build context
#7 transferring context: 7.65kB done
#7 DONE 0.0s

#8 [ 4/13] RUN pip3 install torch torchvision torchaudio pycocotools matplotlib onnxruntime onnx
#8 21.51 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/torch/
#8 63.88 ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
#8 63.88 ERROR: No matching distribution found for torch
#8 ERROR: process "/bin/sh -c pip3 install torch torchvision torchaudio pycocotools matplotlib onnxruntime onnx" did not complete successfully: exit code: 1

[ 4/13] RUN pip3 install torch torchvision torchaudio pycocotools matplotlib onnxruntime onnx:
21.51 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='pypi.org', port=443): Read timed out. (read timeout=15)")': /simple/torch/
63.88 ERROR: Could not find a version that satisfies the requirement torch (from versions: none)
63.88 ERROR: No matching distribution found for torch

---

Dockerfile.processor:24

22 | RUN apt-get update && apt-get -y install curl git python3 python3-pip ffmpeg libsm6 libxext6
23 |
24 | >>> RUN pip3 install torch torchvision torchaudio pycocotools matplotlib onnxruntime onnx
25 |
26 | RUN pip3 install git+https://github.com/facebookresearch/segment-anything.git

ERROR: failed to solve: process "/bin/sh -c pip3 install torch torchvision torchaudio pycocotools matplotlib onnxruntime onnx" did not complete successfully: exit code: 1

/nuclio/pkg/cmdrunner/shellrunner.go:114

Failed to build
/nuclio/pkg/dockerclient/shell.go:116
Failed to build docker image
.../pkg/containerimagebuilderpusher/docker.go:70
Failed to build processor image
/nuclio/pkg/processor/build/builder.go:264
Failed to deploy function
...//nuclio/pkg/platform/abstract/platform.go:227