fbrs semi automatic annotation nuclio function starts unhealhy

[nikste]

My actions before raising this issue

• [x ] Read/searched the docs
• [ x] Searched past issues

fbrs network for semi automatic annotations does not start with the current nuclio function code.

Am i doing something wrong? (please check steps to repeat)

Expected Behaviour

fbrs network for semi automatic annotation works after running deploy_cpu.sh

Current Behaviour

fbrs network for semi automatic annotation just says unhealthy when running nuctl get functions
when trying to access in cvat it returns error code 500.

Possible Solution

it says something with `pickle key´ not found in docker logs.
Did something change with the download url for the checkpoint?

Steps to Reproduce (for bugs)

1. install prerequisites for nucio
2. run cvat with docker compose on develop branch:
   docker-compose -f docker-compose.yml -f components/serverless/docker-compose.serverless.yml up -d
3. cd serverless
4. deploy_cpu.sh (or only start fbrs network)
5. wait till finished
6. wait a bit more till function startup runs into error
7. nuctl get functions
8. check for pth-saic-vul-fbrs, it says unhealthy

Context

cant do semi automatic annotation

Your Environment

ubuntu 22.04

Next steps

You may join our Gitter channel for community support.

[simone-viozzi]

Same thing,
on the nuclio dashboard, there are some logs:

 [13:16:32.906]  (I)  Deploying function    
 [13:16:32.906]  (I)  Building   [versionInfo: "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"] 
 [13:16:33.322]  (I)  Cleaning up before deployment   [functionName: "pth-saic-vul-fbrs"] 
 [13:16:33.368]  (I)  Function already exists, deleting function containers   [functionName: "pth-saic-vul-fbrs"] 
 [13:16:33.410]  (I)  Staging files and preparing base images    
 [13:16:33.478]  (W)  Failed to create a function; setting the function status  Failed to prepare staging dir  [errCauses: [{"error":"Failed to prepare staging dir","errorCauses":[{"error":"Failed to copy handler object","errorCauses":[{"error":"open /dev/core: permission denied"}],"errorVerbose":"
Error - open /dev/core: permission denied
    /nuclio/pkg/processor/build/builder.go:966

Call stack:
Failed to copy handler object
    /nuclio/pkg/processor/build/builder.go:966
Failed to copy handler object"}],"errorVerbose":"
Error - open /dev/core: permission denied
    /nuclio/pkg/processor/build/builder.go:966

the logs of the container are:

22.08.04 11:20:34.933 �[0;37m                processor�[0m �[0;34m(I)�[0m Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:34.933 �[0;37m                processor�[0m �[0;32m(D)�[0m Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:34.933 �[0;37mcessor.healthcheck.server�[0m �[0;34m(I)�[0m Listening {"listenAddress": ":8082"}
22.08.04 11:20:34.933 �[0;37m           processor.http�[0m �[0;32m(D)�[0m Creating worker pool {"num": 2}
22.08.04 11:20:34.933 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn0jhhegdmp159dug.sock"}
22.08.04 11:20:34.933 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn0jhhegdmp159dv0.sock"}
22.08.04 11:20:34.933 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:34.933 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:34.933 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:34.933 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:34.933 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:34.933 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:34.933 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:34.933 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:34.933 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn0jhhegdmp159dug.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:34.933 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn0jhhegdmp159dv0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
/usr/local/lib/python3.6/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /opt/nuclio/fbrs/isegm/utils/cython/_get_dist_maps.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
/usr/local/lib/python3.6/site-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /opt/nuclio/fbrs/isegm/utils/cython/_get_dist_maps.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
In file included from /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
                 from /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /root/.pyxbld/temp.linux-x86_64-3.6/pyrex/isegm/utils/cython/_get_dist_maps.cpp:760:
/usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
In file included from /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
                 from /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from /root/.pyxbld/temp.linux-x86_64-3.6/pyrex/isegm/utils/cython/_get_dist_maps.cpp:760:
/usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it with " \
  ^~~~~~~
22.08.04 11:20:38.159 �[0;37msor.http.w0.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 0, "pid": 23}
22.08.04 11:20:38.159 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Waiting for start
22.08.04 11:20:38.159 �[0;37msor.http.w1.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 1, "pid": 22}
22.08.04 11:20:38.159 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:38,159", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:38,159", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:38,160", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:38,160", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:38,160", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:38,160", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:38.160 �[0;37msor.http.w0.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:20:38.160 �[0;37msor.http.w1.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:20:38.399 �[0;37msor.http.w0.python.logger�[0m �[0;31m(E)�[0m Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 0 exited unexpectedly with: exit status 1

goroutine 40 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc0006f0000)
	/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
	/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:20:39.038 �[0;37m                processor�[0m �[0;34m(I)�[0m Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:39.038 �[0;37m                processor�[0m �[0;32m(D)�[0m Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:39.038 �[0;37mcessor.healthcheck.server�[0m �[0;34m(I)�[0m Listening {"listenAddress": ":8082"}
22.08.04 11:20:39.038 �[0;37m           processor.http�[0m �[0;32m(D)�[0m Creating worker pool {"num": 2}
22.08.04 11:20:39.039 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn1rhhegdmp1pn7c0.sock"}
22.08.04 11:20:39.039 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn1rhhegdmp1pn7cg.sock"}
22.08.04 11:20:39.039 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:39.039 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:39.039 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:39.039 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:39.039 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:39.039 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:39.039 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn1rhhegdmp1pn7cg.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:39.039 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:39.039 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:39.039 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn1rhhegdmp1pn7c0.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:40.440 �[0;37msor.http.w1.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 1, "pid": 24}
22.08.04 11:20:40.440 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:40,440", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
22.08.04 11:20:40.441 �[0;37msor.http.w0.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 0, "pid": 23}
22.08.04 11:20:40.441 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:40,441", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:40,441", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:40,442", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:40.442 �[0;37msor.http.w1.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
l{"datetime": "2022-08-04 11:20:40,442", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:40,442", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:20:40.443 �[0;37msor.http.w0.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:20:40.603 �[0;37msor.http.w0.python.logger�[0m �[0;31m(E)�[0m Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 0 exited unexpectedly with: exit status 1

goroutine 13 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00003c0c0)
	/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
	/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:20:41.208 �[0;37m                processor�[0m �[0;34m(I)�[0m Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:41.208 �[0;37m                processor�[0m �[0;32m(D)�[0m Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:41.208 �[0;37mcessor.healthcheck.server�[0m �[0;34m(I)�[0m Listening {"listenAddress": ":8082"}
22.08.04 11:20:41.208 �[0;37m           processor.http�[0m �[0;32m(D)�[0m Creating worker pool {"num": 2}
22.08.04 11:20:41.208 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn2bhhegdmp48qat0.sock"}
22.08.04 11:20:41.208 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn2bhhegdmp48qatg.sock"}
22.08.04 11:20:41.208 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:41.208 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:41.208 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:41.208 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:41.208 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:41.208 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:41.208 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:41.208 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn2bhhegdmp48qat0.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:41.208 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:41.209 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn2bhhegdmp48qatg.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:42.587 �[0;37msor.http.w0.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 0, "pid": 24}
22.08.04 11:20:42.587 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:42,587", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:42,588", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:42,589", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:20:42.589 �[0;37msor.http.w0.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:20:42.594 �[0;37msor.http.w1.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 1, "pid": 23}
22.08.04 11:20:42.594 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:42,594", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:42,595", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:42,595", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:42.595 �[0;37msor.http.w1.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:20:42.761 �[0;37msor.http.w0.python.logger�[0m �[0;31m(E)�[0m Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 0 exited unexpectedly with: exit status 1

goroutine 81 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00059a000)
	/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
	/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:20:43.525 �[0;37m                processor�[0m �[0;34m(I)�[0m Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:43.525 �[0;37m                processor�[0m �[0;32m(D)�[0m Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:43.525 �[0;37mcessor.healthcheck.server�[0m �[0;34m(I)�[0m Listening {"listenAddress": ":8082"}
22.08.04 11:20:43.525 �[0;37m           processor.http�[0m �[0;32m(D)�[0m Creating worker pool {"num": 2}
22.08.04 11:20:43.525 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn2rhhegdmp7ls99g.sock"}
22.08.04 11:20:43.525 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn2rhhegdmp7ls9a0.sock"}
22.08.04 11:20:43.526 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:43.526 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:43.526 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:43.526 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:43.526 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:43.526 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:43.526 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:43.526 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:43.526 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn2rhhegdmp7ls99g.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:43.526 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn2rhhegdmp7ls9a0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:44.942 �[0;37msor.http.w0.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 0, "pid": 22}
22.08.04 11:20:44.942 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:44,942", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:44,943", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
22.08.04 11:20:44.943 �[0;37msor.http.w1.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 1, "pid": 23}
22.08.04 11:20:44.943 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:44,943", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:44,944", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:20:44.944 �[0;37msor.http.w0.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
l{"datetime": "2022-08-04 11:20:44,944", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:44,945", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:44.945 �[0;37msor.http.w1.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:20:45.119 �[0;37msor.http.w1.python.logger�[0m �[0;31m(E)�[0m Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 1 exited unexpectedly with: exit status 1

goroutine 28 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc000130900)
	/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
	/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:20:46.268 �[0;37m                processor�[0m �[0;34m(I)�[0m Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:46.268 �[0;37m                processor�[0m �[0;32m(D)�[0m Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:46.268 �[0;37mcessor.healthcheck.server�[0m �[0;34m(I)�[0m Listening {"listenAddress": ":8082"}
22.08.04 11:20:46.268 �[0;37m           processor.http�[0m �[0;32m(D)�[0m Creating worker pool {"num": 2}
22.08.04 11:20:46.268 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn3jhhegdmp0k8le0.sock"}
22.08.04 11:20:46.268 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn3jhhegdmp0k8leg.sock"}
22.08.04 11:20:46.268 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:46.268 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:46.268 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:46.268 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:46.268 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:46.268 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:46.268 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:46.268 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn3jhhegdmp0k8le0.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:46.268 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:46.268 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn3jhhegdmp0k8leg.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:47.736 �[0;37msor.http.w1.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 1, "pid": 22}
22.08.04 11:20:47.736 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:47,736", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:47,737", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:47,737", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:47.737 �[0;37msor.http.w1.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:20:47.764 �[0;37msor.http.w0.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 0, "pid": 23}
22.08.04 11:20:47.764 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:47,765", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:47,766", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:47,766", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:20:47.766 �[0;37msor.http.w0.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:20:47.907 �[0;37msor.http.w1.python.logger�[0m �[0;31m(E)�[0m Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 1 exited unexpectedly with: exit status 1

goroutine 56 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00003c9c0)
	/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
	/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:20:49.871 �[0;37m                processor�[0m �[0;34m(I)�[0m Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:49.872 �[0;37m                processor�[0m �[0;32m(D)�[0m Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:49.872 �[0;37mcessor.healthcheck.server�[0m �[0;34m(I)�[0m Listening {"listenAddress": ":8082"}
22.08.04 11:20:49.872 �[0;37m           processor.http�[0m �[0;32m(D)�[0m Creating worker pool {"num": 2}
22.08.04 11:20:49.872 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn4bhhegdmp4gmp70.sock"}
22.08.04 11:20:49.872 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn4bhhegdmp4gmp7g.sock"}
22.08.04 11:20:49.872 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:49.872 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:49.872 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:49.872 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:49.872 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:49.872 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:49.872 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn4bhhegdmp4gmp70.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:49.872 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:49.872 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:49.872 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn4bhhegdmp4gmp7g.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:51.324 �[0;37msor.http.w1.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 1, "pid": 20}
22.08.04 11:20:51.324 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:51,324", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:51,325", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:51,325", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:51.326 �[0;37msor.http.w1.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:20:51.413 �[0;37msor.http.w0.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 0, "pid": 21}
22.08.04 11:20:51.413 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:51,413", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:51,414", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:51,415", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:20:51.415 �[0;37msor.http.w0.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:20:51.492 �[0;37msor.http.w1.python.logger�[0m �[0;31m(E)�[0m Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 1 exited unexpectedly with: exit status 1

goroutine 65 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00003c9c0)
	/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
	/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:20:55.089 �[0;37m                processor�[0m �[0;34m(I)�[0m Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:20:55.089 �[0;37m                processor�[0m �[0;32m(D)�[0m Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:20:55.089 �[0;37mcessor.healthcheck.server�[0m �[0;34m(I)�[0m Listening {"listenAddress": ":8082"}
22.08.04 11:20:55.089 �[0;37m           processor.http�[0m �[0;32m(D)�[0m Creating worker pool {"num": 2}
22.08.04 11:20:55.089 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn5rhhegdmp1l8kig.sock"}
22.08.04 11:20:55.089 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn5rhhegdmp1l8kj0.sock"}
22.08.04 11:20:55.089 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:55.089 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:55.089 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:20:55.089 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:55.089 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:55.089 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:20:55.089 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn5rhhegdmp1l8kig.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:55.089 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:20:55.089 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:20:55.089 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn5rhhegdmp1l8kj0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:20:56.544 �[0;37msor.http.w0.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 0, "pid": 22}
22.08.04 11:20:56.544 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:56,544", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:56,545", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:20:56,546", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:20:56.546 �[0;37msor.http.w0.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:20:56.548 �[0;37msor.http.w1.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 1, "pid": 21}
22.08.04 11:20:56.548 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:20:56,548", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:56,549", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:20:56,550", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:20:56.550 �[0;37msor.http.w1.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:20:56.718 �[0;37msor.http.w0.python.logger�[0m �[0;31m(E)�[0m Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 0 exited unexpectedly with: exit status 1

goroutine 40 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00003c0c0)
	/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
	/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:21:03.489 �[0;37m                processor�[0m �[0;34m(I)�[0m Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:21:03.489 �[0;37m                processor�[0m �[0;32m(D)�[0m Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:21:03.489 �[0;37mcessor.healthcheck.server�[0m �[0;34m(I)�[0m Listening {"listenAddress": ":8082"}
22.08.04 11:21:03.489 �[0;37m           processor.http�[0m �[0;32m(D)�[0m Creating worker pool {"num": 2}
22.08.04 11:21:03.489 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn7rhhegdmp29ioi0.sock"}
22.08.04 11:21:03.489 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqn7rhhegdmp29ioig.sock"}
22.08.04 11:21:03.489 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:21:03.489 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:21:03.489 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:21:03.489 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:21:03.489 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:21:03.489 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:21:03.489 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:21:03.489 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn7rhhegdmp29ioig.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:21:03.489 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:21:03.489 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqn7rhhegdmp29ioi0.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:21:04.888 �[0;37msor.http.w1.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 1, "pid": 23}
22.08.04 11:21:04.888 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:21:04,888", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:21:04,889", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:21:04,890", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:21:04.890 �[0;37msor.http.w1.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:21:04.896 �[0;37msor.http.w0.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 0, "pid": 22}
22.08.04 11:21:04.896 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:21:04,896", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:21:04,898", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:21:04,898", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:21:04.898 �[0;37msor.http.w0.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:21:05.053 �[0;37msor.http.w1.python.logger�[0m �[0;31m(E)�[0m Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 1 exited unexpectedly with: exit status 1

goroutine 28 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00003c9c0)
	/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
	/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:21:18.204 �[0;37m                processor�[0m �[0;34m(I)�[0m Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:21:18.204 �[0;37m                processor�[0m �[0;32m(D)�[0m Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:21:18.205 �[0;37mcessor.healthcheck.server�[0m �[0;34m(I)�[0m Listening {"listenAddress": ":8082"}
22.08.04 11:21:18.205 �[0;37m           processor.http�[0m �[0;32m(D)�[0m Creating worker pool {"num": 2}
22.08.04 11:21:18.205 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqnbjhhegdmp08targ.sock"}
22.08.04 11:21:18.205 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqnbjhhegdmp08tas0.sock"}
22.08.04 11:21:18.205 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:21:18.205 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:21:18.205 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:21:18.205 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:21:18.205 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqnbjhhegdmp08tas0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:21:18.205 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:21:18.205 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:21:18.205 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:21:18.205 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:21:18.205 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqnbjhhegdmp08targ.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:21:19.635 �[0;37msor.http.w1.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 1, "pid": 23}
22.08.04 11:21:19.635 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:21:19,635", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:21:19,636", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:21:19,636", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:21:19.636 �[0;37msor.http.w1.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:21:19.670 �[0;37msor.http.w0.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 0, "pid": 22}
22.08.04 11:21:19.670 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:21:19,670", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:21:19,671", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:21:19,672", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:21:19.672 �[0;37msor.http.w0.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:21:19.789 �[0;37msor.http.w1.python.logger�[0m �[0;31m(E)�[0m Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 1 exited unexpectedly with: exit status 1

goroutine 69 [running]:
github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc00003c9c0)
	/nuclio/pkg/processor/runtime/rpc/abstract.go:464 +0x576
created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper
	/nuclio/pkg/processor/runtime/rpc/abstract.go:236 +0x1b5
22.08.04 11:21:45.780 �[0;37m                processor�[0m �[0;34m(I)�[0m Starting processor {"version": "Label: 1.5.16, Git commit: ae43a6a560c2bec42d7ccfdf6e8e11a1e3cc3774, OS: linux, Arch: amd64, Go version: go1.14.3"}
22.08.04 11:21:45.780 �[0;37m                processor�[0m �[0;32m(D)�[0m Read configuration {"config": "{\n    \"metadata\": {\n        \"name\": \"pth-saic-vul-fbrs\",\n        \"namespace\": \"nuclio\",\n        \"labels\": {\n            \"nuclio.io/project-name\": \"cvat\"\n        },\n        \"annotations\": {\n            \"animated_gif\": \"https://raw.githubusercontent.com/openvinotoolkit/cvat/0fbb19ae3846a017853d52e187f0ce149adced7d/site/content/en/images/fbrs_example.gif\",\n            \"framework\": \"pytorch\",\n            \"help_message\": \"The interactor allows to get a mask for an object using positive points, and negative points\",\n            \"min_neg_points\": \"0\",\n            \"min_pos_points\": \"1\",\n            \"name\": \"f-BRS\",\n            \"spec\": \"\",\n            \"type\": \"interactor\"\n        }\n    },\n    \"spec\": {\n        \"description\": \"f-BRS interactive segmentation\",\n        \"handler\": \"main:handler\",\n        \"runtime\": \"python:3.6\",\n        \"env\": [\n            {\n                \"name\": \"PYTHONPATH\",\n                \"value\": \"/opt/nuclio/fbrs\"\n            }\n        ],\n        \"resources\": {},\n        \"image\": \"cvat/pth.saic-vul.fbrs:latest\",\n        \"targetCPU\": 75,\n        \"triggers\": {\n            \"myHttpTrigger\": {\n                \"class\": \"\",\n                \"kind\": \"http\",\n                \"name\": \"myHttpTrigger\",\n                \"maxWorkers\": 2,\n                \"workerAvailabilityTimeoutMilliseconds\": 10000,\n                \"attributes\": {\n                    \"maxRequestBodySize\": 33554432\n                }\n            }\n        },\n        \"volumes\": [\n            {\n                \"volume\": {\n                    \"name\": \"volume-1\",\n                    \"hostPath\": {\n                        \"path\": \"/home/ubuntu/cvatV2/serverless/common\"\n                    }\n                },\n                \"volumeMount\": {\n                    \"name\": \"volume-1\",\n                    \"mountPath\": \"/opt/nuclio/common\"\n                }\n            }\n        ],\n        \"build\": {\n            \"image\": \"cvat/pth.saic-vul.fbrs\",\n            \"baseImage\": \"python:3.6.11\",\n            \"directives\": {\n                \"preCopy\": [\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"git clone https://github.com/saic-vul/fbrs_interactive_segmentation.git fbrs\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio/fbrs\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"fileid=1Z9dQtpWVTobEdmUBntpUU0pJl-pEXUwR\"\n                    },\n                    {\n                        \"kind\": \"ENV\",\n                        \"value\": \"filename=resnet101_dh256_sbd.pth\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -c ./cookie -s -L \\\"https://drive.google.com/uc?export=download\\u0026id=${fileid}\\\"\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"curl -Lb ./cookie \\\"https://drive.google.com/uc?export=download\\u0026confirm=`awk '/download/ {print $NF}' ./cookie`\\u0026id=${fileid}\\\" -o ${filename}\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"apt update \\u0026\\u0026 apt install -y libgl1-mesa-glx\"\n                    },\n                    {\n                        \"kind\": \"RUN\",\n                        \"value\": \"pip3 install -r requirements.txt\"\n                    },\n                    {\n                        \"kind\": \"WORKDIR\",\n                        \"value\": \"/opt/nuclio\"\n                    }\n                ]\n            },\n            \"codeEntryType\": \"image\",\n            \"timestamp\": 1659612034\n        },\n        \"platform\": {\n            \"attributes\": {\n                \"mountMode\": \"volume\",\n                \"restartPolicy\": {\n                    \"maximumRetryCount\": 3,\n                    \"name\": \"always\"\n                }\n            }\n        },\n        \"readinessTimeoutSeconds\": 60,\n        \"securityContext\": {},\n        \"eventTimeout\": \"30s\"\n    },\n    \"PlatformConfig\": null\n}", "platformConfig": "{\n    \"kind\": \"local\",\n    \"webAdmin\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8081\"\n    },\n    \"healthCheck\": {\n        \"enabled\": true,\n        \"listenAddress\": \":8082\"\n    },\n    \"logger\": {\n        \"sinks\": {\n            \"stdout\": {\n                \"kind\": \"stdout\"\n            }\n        },\n        \"system\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ],\n        \"functions\": [\n            {\n                \"level\": \"debug\",\n                \"sink\": \"stdout\"\n            }\n        ]\n    },\n    \"metrics\": {},\n    \"scaleToZero\": {},\n    \"autoScale\": {},\n    \"cronTriggerCreationMode\": \"processor\",\n    \"ingressConfig\": {},\n    \"kube\": {\n        \"defaultServiceType\": \"ClusterIP\"\n    },\n    \"imageRegistryOverrides\": {}\n}"}
22.08.04 11:21:45.780 �[0;37mcessor.healthcheck.server�[0m �[0;34m(I)�[0m Listening {"listenAddress": ":8082"}
22.08.04 11:21:45.781 �[0;37m           processor.http�[0m �[0;32m(D)�[0m Creating worker pool {"num": 2}
22.08.04 11:21:45.781 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqnibhhegdmp3puiqg.sock"}
22.08.04 11:21:45.781 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Creating listener socket {"path": "/tmp/nuclio-rpc-cblqnibhhegdmp3puir0.sock"}
22.08.04 11:21:45.781 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:21:45.781 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
22.08.04 11:21:45.781 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:21:45.781 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python handler {"handler": "main:handler"}
22.08.04 11:21:45.781 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:21:45.781 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:21:45.781 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqnibhhegdmp3puir0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:21:45.781 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Using Python executable {"path": "/usr/local/bin/python3"}
22.08.04 11:21:45.781 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio:/opt/nuclio/fbrs"}
22.08.04 11:21:45.781 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Running wrapper {"command": "/usr/local/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --socket-path /tmp/nuclio-rpc-cblqnibhhegdmp3puiqg.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger"}
22.08.04 11:21:47.252 �[0;37msor.http.w1.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 1, "pid": 24}
22.08.04 11:21:47.252 �[0;37msor.http.w1.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:21:47,252", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:21:47,253", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "1"}}
l{"datetime": "2022-08-04 11:21:47,254", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "1"}}
22.08.04 11:21:47.254 �[0;37msor.http.w1.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:21:47.272 �[0;37msor.http.w0.python.logger�[0m �[0;34m(I)�[0m Wrapper connected {"wid": 0, "pid": 23}
22.08.04 11:21:47.272 �[0;37msor.http.w0.python.logger�[0m �[0;32m(D)�[0m Waiting for start
l{"datetime": "2022-08-04 11:21:47,272", "level": "info", "message": "Init context...  0%", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:21:47,273", "level": "error", "message": "Exception raised while running init_context", "with": {"worker_id": "0"}}
l{"datetime": "2022-08-04 11:21:47,274", "level": "error", "message": "Caught unhandled exception while initializing", "with": {"err": "invalid load key, '<'.", "traceback": "Traceback (most recent call last):\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 350, in run_wrapper\n    args.trigger_name)\n  File \"/opt/nuclio/_nuclio_wrapper.py\", line 80, in __init__\n    getattr(entrypoint_module, 'init_context')(self._context)\n  File \"/opt/nuclio/main.py\", line 14, in init_context\n    model = ModelHandler()\n  File \"/opt/nuclio/model_handler.py\", line 41, in __init__\n    state_dict = torch.load(model_path, map_location='cpu')\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 608, in load\n    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)\n  File \"/usr/local/lib/python3.6/site-packages/torch/serialization.py\", line 777, in _legacy_load\n    magic_number = pickle_module.load(f, **pickle_load_args)\n_pickle.UnpicklingError: invalid load key, '<'.\n", "worker_id": "0"}}
22.08.04 11:21:47.274 �[0;37msor.http.w0.python.logger�[0m �[0;33m(W)�[0m Failed to read from connection {"err": "EOF"}
22.08.04 11:21:47.447 �[0;37msor.http.w0.python.logger�[0m �[0;31m(E)�[0m Unexpected termination of child process {"error": null, "status": "exit status 1"}
panic: Wrapper process for worker 0 exited unexpectedly with: exit status 1

[amteusch]

I ran into this error a few months ago and forgot to make a PR for it. Google Drive changed and so the previous method to download the model weights no longer works. I tested with my fix today and FBRS worked, however I could not get the Docker image for IOG (which has the same problem) to build properly. I may try to solve the problem and make a PR next week, but if you want to try the changes on your own you can see my branch here