Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added SiamMask CUDA implementation [2x faster on Full HD] #3571

Merged
merged 23 commits into from
Sep 29, 2021

Conversation

bsekachev
Copy link
Member

@bsekachev bsekachev commented Aug 19, 2021

General

  • Added CUDA support for SiamMask tracker

  • Tracker is able to track different objects in one server request

  • "Trackable" objects are introduced (an object will be tracked automatically if the previous frame was a latest keyframe for the object)

  • Added tracker indication on canvas
    image

  • Added tracker switcher on sidebar
    image

  • Tracking with non-blocking UI
    image

Motivation and context

Resolved #2949
Resolved #2773
Related #2936

How has this been tested?

Checklist

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below)
# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

@@ -18,10 +18,20 @@ spec:

build:
image: cvat/pth.foolwood.siammask
baseImage: continuumio/miniconda3
baseImage: ubuntu:20.04
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed base image to have the same set of preCopy directives.
But finally it was nesessary to add
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
to function-gpu.yml, so it is not the same now anyway. Let me know if you think that better to leave this file as is.

@bsekachev bsekachev changed the title [WIP] Added SiamMask CUDA implementation [2x faster on Full HD] Added SiamMask CUDA implementation [2x faster on Full HD] Aug 31, 2021
@rohitsaluja22
Copy link

rohitsaluja22 commented Sep 2, 2021

hi @bsekachev i am new to cvat, request you explain how to run SiamMask CUDA implementation from starting?
Do I need to modify serverless/pytorch/foolwood/siammask/nuclio/function.yaml, which other files should i modify? I am unable to find function-gpu.yml in the code I downloaded from github.

@bsekachev
Copy link
Member Author

Hi @rohitsaluja22

Probably you use develop branch. This PR is not merged yet. To get function-gpu.yml you need to checkout to branch bs/siammask_gpu. Please, refer to git documentation.

To deploy the function you either run serverless/deploy_gpu.sh (to deploy all the gpu functions), or rename function-gpu.yml to function.yml and run nuctl deploy --project-name cvat --path `pwd`/serverless/pytorch/foolwood/siammask/nuclio/ --volume `pwd`/serverless/common:/opt/nuclio/common --platform local from cvat root directory.

ActiveChooN
ActiveChooN previously approved these changes Sep 6, 2021
Copy link
Contributor

@ActiveChooN ActiveChooN left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, but not able to test GPU version. And one question: is there option to autoupdate function with updating CVAT images?

@ActiveChooN
Copy link
Contributor

Successfully built serverless function with GPU support under WSL, but eventually got an error when called a function. Probably it's a problem with my setup.

Nuclio function log
21.09.06 22:50:48.605 processor (I) Starting processor {"version": "Label: 1.6.3, Git commit: 20b43c6d5637434ff9732c7fa8ce28c70d3805ce, OS: linux, Arch: amd64, Go version: go1.14.3"}

21.09.06 22:50:48.605 processor (D) Read configuration {"config": "{\n \"metadata\": {\n \"name\": \"pth-foolwood-siammask\",\n \"namespace\": \"nuclio\",\n \"labels\": {\n \"nuclio.io/project-name\": \"cvat\"\n },\n \"annotations\": {\n \"framework\": \"pytorch\",\n \"name\": \"SiamMask\",\n \"spec\": \"\",\n \"type\": \"tracker\"\n }\n },\n \"spec\": {\n \"description\": \"Fast Online Object Tracking and Segmentation\",\n \"handler\": \"main:handler\",\n \"runtime\": \"python:3.6\",\n \"env\": [\n {\n \"name\": \"PYTHONPATH\",\n \"value\": \"/opt/nuclio/SiamMask:/opt/nuclio/SiamMask/experiments/siammask_sharp\"\n }\n ],\n \"resources\": {\n \"limits\": {\n \"nvidia.com/gpu\": \"1\"\n }\n },\n \"image\": \"cvat/pth.foolwood.siammask:latest\",\n \"targetCPU\": 75,\n \"triggers\": {\n \"myHttpTrigger\": {\n \"class\": \"\",\n \"kind\": \"http\",\n \"name\": \"myHttpTrigger\",\n \"maxWorkers\": 2,\n \"workerAvailabilityTimeoutMilliseconds\": 10000,\n \"attributes\": {\n \"maxRequestBodySize\": 33554432\n }\n }\n },\n \"volumes\": [\n {\n \"volume\": {\n \"name\": \"volume-1\",\n \"hostPath\": {\n \"path\": \"/home/achoon/wsl_projects/cvat/serverless/common\"\n }\n },\n \"volumeMount\": {\n \"name\": \"volume-1\",\n \"mountPath\": \"/opt/nuclio/common\"\n }\n }\n ],\n \"build\": {\n \"functionConfigPath\": \"/home/achoon/wsl_projects/cvat/serverless/pytorch/foolwood/siammask/nuclio/function-gpu.yaml\",\n \"image\": \"cvat/pth.foolwood.siammask\",\n \"baseImage\": \"nvidia/cuda:11.1-devel-ubuntu20.04\",\n \"directives\": {\n \"preCopy\": [\n {\n \"kind\": \"ENV\",\n \"value\": \"PATH=\\\"/root/miniconda3/bin:${PATH}\\\"\"\n },\n {\n \"kind\": \"ARG\",\n \"value\": \"PATH=\\\"/root/miniconda3/bin:${PATH}\\\"\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"apt update \\u0026\\u0026 apt install -y --no-install-recommends wget git ca-certificates libglib2.0-0 libsm6 libxrender1 libxext6 \\u0026\\u0026 rm -rf /var/lib/apt/lists/*\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh \\u0026\\u0026 chmod +x Miniconda3-latest-Linux-x86_64.sh \\u0026\\u0026 ./Miniconda3-latest-Linux-x86_64.sh -b \\u0026\\u0026 rm -f Miniconda3-latest-Linux-x86_64.sh\"\n },\n {\n \"kind\": \"WORKDIR\",\n \"value\": \"/opt/nuclio\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"conda create -y -n siammask python=3.7\"\n },\n {\n \"kind\": \"SHELL\",\n \"value\": \"[\\\"conda\\\", \\\"run\\\", \\\"-n\\\", \\\"siammask\\\", \\\"/bin/bash\\\", \\\"-c\\\"]\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"git clone https://github.com/foolwood/SiamMask.git\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"pip install -r SiamMask/requirements.txt jsonpickle\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"conda install -y gcc_linux-64\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"cd SiamMask \\u0026\\u0026 bash make.sh \\u0026\\u0026 cd -\"\n },\n {\n \"kind\": \"RUN\",\n \"value\": \"wget -P SiamMask/experiments/siammask_sharp http://www.robots.ox.ac.uk/~qwang/SiamMask_DAVIS.pth\"\n },\n {\n \"kind\": \"ENTRYPOINT\",\n \"value\": \"[\\\"conda\\\", \\\"run\\\", \\\"-n\\\", \\\"siammask\\\"]\"\n }\n ]\n },\n \"codeEntryType\": \"image\",\n \"timestamp\": 1630967944\n },\n \"platform\": {\n \"attributes\": {\n \"mountMode\": \"volume\",\n \"restartPolicy\": {\n \"maximumRetryCount\": 3,\n \"name\": \"always\"\n }\n }\n },\n \"readinessTimeoutSeconds\": 60,\n \"securityContext\": {},\n \"eventTimeout\": \"30s\"\n },\n \"PlatformConfig\": null\n}", "platformConfig": "{\n \"kind\": \"local\",\n \"webAdmin\": {\n \"enabled\": true,\n \"listenAddress\": \":8081\"\n },\n \"healthCheck\": {\n \"enabled\": true,\n \"listenAddress\": \":8082\"\n },\n \"logger\": {\n \"sinks\": {\n \"stdout\": {\n \"kind\": \"stdout\"\n }\n },\n \"system\": [\n {\n \"level\": \"debug\",\n \"sink\": \"stdout\"\n }\n ],\n \"functions\": [\n {\n \"level\": \"debug\",\n \"sink\": \"stdout\"\n }\n ]\n },\n \"metrics\": {},\n \"scaleToZero\": {},\n \"autoScale\": {},\n \"cronTriggerCreationMode\": \"processor\",\n \"ingressConfig\": {},\n \"kube\": {\n \"defaultServiceType\": \"ClusterIP\"\n },\n \"local\": {\n \"FunctionContainersHealthinessEnabled\": false,\n \"FunctionContainersHealthinessTimeout\": 5000000000,\n \"FunctionContainersHealthinessInterval\": 30000000000\n },\n \"imageRegistryOverrides\": {}\n}"}

21.09.06 22:50:48.606 cessor.healthcheck.server (I) Listening {"listenAddress": ":8082"}

21.09.06 22:50:48.606 processor.http (D) Creating worker pool {"num": 2}

21.09.06 22:50:48.607 sor.http.w1.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-c4r9mi0n0a9vutmvhrng.sock"}

21.09.06 22:50:48.607 sor.http.w0.python.logger (D) Creating listener socket {"path": "/tmp/nuclio-rpc-c4r9mi0n0a9vutmvhro0.sock"}

21.09.06 22:50:48.607 sor.http.w1.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}

21.09.06 22:50:48.607 sor.http.w0.python.logger (D) Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}

21.09.06 22:50:48.607 sor.http.w1.python.logger (D) Using Python handler {"handler": "main:handler"}

21.09.06 22:50:48.607 sor.http.w0.python.logger (D) Using Python handler {"handler": "main:handler"}

21.09.06 22:50:48.607 sor.http.w1.python.logger (D) Using Python executable {"path": "/root/miniconda3/envs/siammask/bin/python3"}

21.09.06 22:50:48.607 sor.http.w0.python.logger (D) Using Python executable {"path": "/root/miniconda3/envs/siammask/bin/python3"}

21.09.06 22:50:48.607 sor.http.w1.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/SiamMask:/opt/nuclio/SiamMask/experiments/siammask_sharp"}

21.09.06 22:50:48.607 sor.http.w0.python.logger (D) Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/SiamMask:/opt/nuclio/SiamMask/experiments/siammask_sharp"}

21.09.06 22:50:48.607 sor.http.w1.python.logger (D) Running wrapper {"command": "/root/miniconda3/envs/siammask/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-c4r9mi0n0a9vutmvhrng.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger --decode-event-strings"}

21.09.06 22:50:48.607 sor.http.w0.python.logger (D) Running wrapper {"command": "/root/miniconda3/envs/siammask/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-c4r9mi0n0a9vutmvhro0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger --decode-event-strings"}

21.09.06 22:50:53.750 sor.http.w1.python.logger (I) Wrapper connected {"wid": 1, "pid": 312}

21.09.06 22:50:53.750 sor.http.w0.python.logger (I) Wrapper connected {"wid": 0, "pid": 313}

l{"datetime": "2021-09-06 22:50:53,750", "level": "info", "message": "Replacing logger output", "with": {"handler_name": "default", "worker_id": "0"}}

21.09.06 22:50:53.750 sor.http.w0.python.logger (D) Waiting for start

21.09.06 22:50:53.750 sor.http.w1.python.logger (D) Waiting for start

l{"datetime": "2021-09-06 22:50:53,750", "level": "info", "message": "Replacing logger output", "with": {"handler_name": "default", "worker_id": "1"}}

21.09.06 22:50:53.752 sor.http.w1.python.logger (I) Init context... 0% {"worker_id": "1"}

21.09.06 22:50:53.752 sor.http.w0.python.logger (I) Init context... 0% {"worker_id": "0"}

21.09.06 22:51:33.349 sor.http.w1.python.logger (W) Failed to read from connection {"err": "EOF"}

21.09.06 22:51:33.414 sor.http.w1.python.logger (E) Unexpected termination of child process {"error": null, "status": "signal: killed"}



ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['processor']' command failed. (See above for error)

[2021-09-06 22:39:16,152-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:39:16,154-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:39:16,159-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:39:16,162-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:39:16,259-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

[2021-09-06 22:39:16,264-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

[2021-09-06 22:39:26,795-rk0-load_helper.py# 25] remove prefix 'module.'

[2021-09-06 22:39:26,795-rk0-load_helper.py# 25] remove prefix 'module.'

[2021-09-06 22:39:26,802-rk0-load_helper.py# 18] used keys:356

[2021-09-06 22:39:26,802-rk0-load_helper.py# 18] used keys:356

panic: Wrapper process for worker 1 exited unexpectedly with: signal: killed


goroutine 36 [running]:

github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc0003826c0)

/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576

created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper

/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5

--- Logging error ---

Traceback (most recent call last):

File "/opt/nuclio/_nuclio_wrapper.py", line 107, in serve_requests

event_message_length = self._resolve_event_message_length()

File "/opt/nuclio/_nuclio_wrapper.py", line 213, in _resolve_event_message_length

raise WrapperFatalException('Client disconnected')

WrapperFatalException: Client disconnected


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1029, in emit

self.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1009, in flush

self.stream.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe

Call stack:

File "/opt/nuclio/_nuclio_wrapper.py", line 383, in <module>

run_wrapper()

File "/opt/nuclio/_nuclio_wrapper.py", line 377, in run_wrapper

wrapper_instance.serve_requests()

File "/opt/nuclio/_nuclio_wrapper.py", line 121, in serve_requests

self._on_serving_error(exc)

File "/opt/nuclio/_nuclio_wrapper.py", line 241, in _on_serving_error

self._log_and_response_error(exc, 'Exception caught while serving')

File "/opt/nuclio/_nuclio_wrapper.py", line 250, in _log_and_response_error

self._logger.error_with(error_message, exc=str(exc), traceback=traceback.format_exc())

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 107, in error_with

self._update_bound_vars_and_log(logging.ERROR, message, *args, **kw_args)

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 116, in _update_bound_vars_and_log

self._logger._log(level, message, args, extra={"with": kw_args})

Message: 'Exception caught while serving'

Arguments: ()


ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['processor']' command failed. (See above for error)

[2021-09-06 22:43:40,250-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:43:40,252-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:43:40,252-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:43:40,254-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:43:40,329-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

[2021-09-06 22:43:40,330-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

panic: Wrapper process for worker 0 exited unexpectedly with: signal: killed


goroutine 13 [running]:

github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00002e000)

/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576

created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper

/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5

[2021-09-06 22:43:56,864-rk0-load_helper.py# 25] remove prefix 'module.'

[2021-09-06 22:43:56,933-rk0-load_helper.py# 18] used keys:356

--- Logging error ---

Traceback (most recent call last):

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1029, in emit

self.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1009, in flush

self.stream.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe

Call stack:

File "/opt/nuclio/_nuclio_wrapper.py", line 383, in <module>

run_wrapper()

File "/opt/nuclio/_nuclio_wrapper.py", line 367, in run_wrapper

args.decode_event_strings)

File "/opt/nuclio/_nuclio_wrapper.py", line 91, in __init__

getattr(entrypoint_module, 'init_context')(self._context)

File "/opt/nuclio/main.py", line 14, in init_context

context.logger.info("Init context...100%")

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 89, in info

self._update_bound_vars_and_log(logging.INFO, message, *args)

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 116, in _update_bound_vars_and_log

self._logger._log(level, message, args, extra={"with": kw_args})

Message: 'Init context...100%'

Arguments: ()

--- Logging error ---

Traceback (most recent call last):

File "/opt/nuclio/_nuclio_wrapper.py", line 367, in run_wrapper

args.decode_event_strings)

File "/opt/nuclio/_nuclio_wrapper.py", line 97, in __init__

self._write_packet_to_processor('s')

File "/opt/nuclio/_nuclio_wrapper.py", line 202, in _write_packet_to_processor

self._processor_sock_wfile.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1029, in emit

self.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1009, in flush

self.stream.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe

Call stack:

File "/opt/nuclio/_nuclio_wrapper.py", line 383, in <module>

run_wrapper()

File "/opt/nuclio/_nuclio_wrapper.py", line 372, in run_wrapper

traceback=traceback.format_exc())

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 107, in error_with

self._update_bound_vars_and_log(logging.ERROR, message, *args, **kw_args)

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 116, in _update_bound_vars_and_log

self._logger._log(level, message, args, extra={"with": kw_args})

Message: 'Caught unhandled exception while initializing'

Arguments: ()


ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['processor']' command failed. (See above for error)

[2021-09-06 22:44:10,883-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:44:10,884-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:44:10,885-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:44:10,885-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:44:10,950-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

[2021-09-06 22:44:10,954-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

[2021-09-06 22:44:24,165-rk0-load_helper.py# 25] remove prefix 'module.'

[2021-09-06 22:44:24,168-rk0-load_helper.py# 25] remove prefix 'module.'

[2021-09-06 22:44:24,218-rk0-load_helper.py# 18] used keys:356

[2021-09-06 22:44:24,219-rk0-load_helper.py# 18] used keys:356

panic: Wrapper process for worker 1 exited unexpectedly with: signal: killed


goroutine 65 [running]:

github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc0003826c0)

/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576

created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper

/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5

--- Logging error ---

Traceback (most recent call last):

File "/opt/nuclio/_nuclio_wrapper.py", line 107, in serve_requests

event_message_length = self._resolve_event_message_length()

File "/opt/nuclio/_nuclio_wrapper.py", line 213, in _resolve_event_message_length

raise WrapperFatalException('Client disconnected')

WrapperFatalException: Client disconnected


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1029, in emit

self.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1009, in flush

self.stream.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe

Call stack:

File "/opt/nuclio/_nuclio_wrapper.py", line 383, in <module>

run_wrapper()

File "/opt/nuclio/_nuclio_wrapper.py", line 377, in run_wrapper

wrapper_instance.serve_requests()

File "/opt/nuclio/_nuclio_wrapper.py", line 121, in serve_requests

self._on_serving_error(exc)

File "/opt/nuclio/_nuclio_wrapper.py", line 241, in _on_serving_error

self._log_and_response_error(exc, 'Exception caught while serving')

File "/opt/nuclio/_nuclio_wrapper.py", line 250, in _log_and_response_error

self._logger.error_with(error_message, exc=str(exc), traceback=traceback.format_exc())

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 107, in error_with

self._update_bound_vars_and_log(logging.ERROR, message, *args, **kw_args)

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 116, in _update_bound_vars_and_log

self._logger._log(level, message, args, extra={"with": kw_args})

Message: 'Exception caught while serving'

Arguments: ()


ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['processor']' command failed. (See above for error)

[2021-09-06 22:50:55,085-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:50:55,088-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:50:55,119-rk0-features.py# 66] Current training 0 layers:


[2021-09-06 22:50:55,122-rk0-features.py# 66] Current training 1 layers:


[2021-09-06 22:50:55,243-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

[2021-09-06 22:50:55,267-rk0-load_helper.py# 31] load pretrained model from /opt/nuclio/SiamMask/experiments/siammask_sharp/SiamMask_DAVIS.pth

panic: Wrapper process for worker 1 exited unexpectedly with: signal: killed


goroutine 42 [running]:

github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00035e6c0)

/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576

created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper

/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5

[2021-09-07 06:06:36,120-rk0-load_helper.py# 25] remove prefix 'module.'

[2021-09-07 06:06:36,566-rk0-load_helper.py# 18] used keys:356

--- Logging error ---

Traceback (most recent call last):

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1029, in emit

self.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1009, in flush

self.stream.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe

Call stack:

File "/opt/nuclio/_nuclio_wrapper.py", line 383, in <module>

run_wrapper()

File "/opt/nuclio/_nuclio_wrapper.py", line 367, in run_wrapper

args.decode_event_strings)

File "/opt/nuclio/_nuclio_wrapper.py", line 91, in __init__

getattr(entrypoint_module, 'init_context')(self._context)

File "/opt/nuclio/main.py", line 14, in init_context

context.logger.info("Init context...100%")

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 89, in info

self._update_bound_vars_and_log(logging.INFO, message, *args)

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 116, in _update_bound_vars_and_log

self._logger._log(level, message, args, extra={"with": kw_args})

Message: 'Init context...100%'

Arguments: ()

--- Logging error ---

Traceback (most recent call last):

File "/opt/nuclio/_nuclio_wrapper.py", line 367, in run_wrapper

args.decode_event_strings)

File "/opt/nuclio/_nuclio_wrapper.py", line 97, in __init__

self._write_packet_to_processor('s')

File "/opt/nuclio/_nuclio_wrapper.py", line 202, in _write_packet_to_processor

self._processor_sock_wfile.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1029, in emit

self.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/logging/__init__.py", line 1009, in flush

self.stream.flush()

File "/root/miniconda3/envs/siammask/lib/python3.7/socket.py", line 607, in write

return self._sock.send(b)

BrokenPipeError: [Errno 32] Broken pipe

Call stack:

File "/opt/nuclio/_nuclio_wrapper.py", line 383, in <module>

run_wrapper()

File "/opt/nuclio/_nuclio_wrapper.py", line 372, in run_wrapper

traceback=traceback.format_exc())

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 107, in error_with

self._update_bound_vars_and_log(logging.ERROR, message, *args, **kw_args)

File "/root/miniconda3/envs/siammask/lib/python3.7/site-packages/nuclio_sdk/logger.py", line 116, in _update_bound_vars_and_log

self._logger._log(level, message, args, extra={"with": kw_args})

Message: 'Caught unhandled exception while initializing'

Arguments: ()

@ActiveChooN
Copy link
Contributor

@rohitsaluja22, also you can not rename a file and just add the flag --file to nuctl deploy command with the path to function-gpu.yml file.

@nmanovic
Copy link
Contributor

@bsekachev , if annotations are removed using "Remove annotations", tracked objects always are reinitialized.

@nmanovic nmanovic merged commit dbdcd4f into develop Sep 29, 2021
@nmanovic nmanovic deleted the bs/siammask_gpu branch September 29, 2021 09:31
@bsekachev
Copy link
Member Author

bsekachev commented Sep 29, 2021

@aschernov

Andrey, could your team update tracking documentation?
Thanks

@aschernov
Copy link
Contributor

@bsekachev, sure, we'll do it.

@siddtmb
Copy link

siddtmb commented Mar 19, 2024

I dont know why automatic annotation for "Allow Siammask active tracker to run for more frames" was closed, at the moment it can only run on the frame you are on and is incredibly slow

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
6 participants