Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error occurs when auto-annotation is executed using serverless yolov7-gpu #8503

Closed
2 tasks done
prnr opened this issue Oct 3, 2024 · 1 comment · Fixed by #8824
Closed
2 tasks done

Error occurs when auto-annotation is executed using serverless yolov7-gpu #8503

prnr opened this issue Oct 3, 2024 · 1 comment · Fixed by #8824
Labels
bug Something isn't working

Comments

@prnr
Copy link

prnr commented Oct 3, 2024

Actions before raising this issue

  • I searched the existing issues and did not find anything similar.
  • I read/searched the docs

Steps to Reproduce

  1. deploy the model with ./serverless/deploy_gpu.sh serverless/onnx/wongkinyiu/yolov7
  2. check the correction of function of nuclio with localhost:8070
  3. start auto annotation

Expected Behavior

No response

Possible Solution

No response

Context

nuclio-nuclio-onnx-wongkinyiu-yolov7 log on docker

2024-10-03 17:25:11
2024-10-03 17:25:11 ==========
2024-10-03 17:25:11 == CUDA ==
2024-10-03 17:25:11 ==========
2024-10-03 17:25:11
2024-10-03 17:25:11 CUDA Version 11.8.0
2024-10-03 17:25:11
2024-10-03 17:25:11 Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2024-10-03 17:25:11
2024-10-03 17:25:11 This container image and its contents are governed by the NVIDIA Deep Learning Container License.
2024-10-03 17:25:11 By pulling and using the container, you accept the terms and conditions of this license:
2024-10-03 17:25:11 https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
2024-10-03 17:25:11
2024-10-03 17:25:11 A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
2024-10-03 17:25:11
2024-10-03 17:25:11 24.10.03 08:25:11.376 (I) processor Starting processor {"version": "Label: 1.13.0, Git commit: c4422eb772781fb50fbf017698aae96199d81388, OS: linux, Arch: amd64, Go version: go1.21.7"}
2024-10-03 17:25:11 24.10.03 08:25:11.376 (D) processor Read configuration {"config": "{\n "metadata": {\n "name": "onnx-wongkinyiu-yolov7",\n "namespace": "nuclio",\n "labels": {\n "nuclio.io/project-name": "cvat"\n },\n "annotations": {\n "framework": "onnx",\n "name": "YOLO v7",\n "spec": "[\n { \"id\": 0, \"name\": \"person\", \"type\": \"rectangle\" },\n { \"id\": 1, \"name\": \"bicycle\", \"type\": \"rectangle\" },\n { \"id\": 2, \"name\": \"car\", \"type\": \"rectangle\" },\n { \"id\": 3, \"name\": \"motorbike\", \"type\": \"rectangle\" },\n { \"id\": 4, \"name\": \"aeroplane\", \"type\": \"rectangle\" },\n { \"id\": 5, \"name\": \"bus\", \"type\": \"rectangle\" },\n { \"id\": 6, \"name\": \"train\", \"type\": \"rectangle\" },\n { \"id\": 7, \"name\": \"truck\", \"type\": \"rectangle\" },\n { \"id\": 8, \"name\": \"boat\", \"type\": \"rectangle\" },\n { \"id\": 9, \"name\": \"traffic light\", \"type\": \"rectangle\" },\n { \"id\": 10, \"name\": \"fire hydrant\", \"type\": \"rectangle\" },\n { \"id\": 11, \"name\": \"stop sign\", \"type\": \"rectangle\" },\n { \"id\": 12, \"name\": \"parking meter\", \"type\": \"rectangle\" },\n { \"id\": 13, \"name\": \"bench\", \"type\": \"rectangle\" },\n { \"id\": 14, \"name\": \"bird\", \"type\": \"rectangle\" },\n { \"id\": 15, \"name\": \"cat\", \"type\": \"rectangle\" },\n { \"id\": 16, \"name\": \"dog\", \"type\": \"rectangle\" },\n { \"id\": 17, \"name\": \"horse\", \"type\": \"rectangle\" },\n { \"id\": 18, \"name\": \"sheep\", \"type\": \"rectangle\" },\n { \"id\": 19, \"name\": \"cow\", \"type\": \"rectangle\" },\n { \"id\": 20, \"name\": \"elephant\", \"type\": \"rectangle\" },\n { \"id\": 21, \"name\": \"bear\", \"type\": \"rectangle\" },\n { \"id\": 22, \"name\": \"zebra\", \"type\": \"rectangle\" },\n { \"id\": 23, \"name\": \"giraffe\", \"type\": \"rectangle\" },\n { \"id\": 24, \"name\": \"backpack\", \"type\": \"rectangle\" },\n { \"id\": 25, \"name\": \"umbrella\", \"type\": \"rectangle\" },\n { \"id\": 26, \"name\": \"handbag\", \"type\": \"rectangle\" },\n { \"id\": 27, \"name\": \"tie\", \"type\": \"rectangle\" },\n { \"id\": 28, \"name\": \"suitcase\", \"type\": \"rectangle\" },\n { \"id\": 29, \"name\": \"frisbee\", \"type\": \"rectangle\" },\n { \"id\": 30, \"name\": \"skis\", \"type\": \"rectangle\" },\n { \"id\": 31, \"name\": \"snowboard\", \"type\": \"rectangle\" },\n { \"id\": 32, \"name\": \"sports ball\", \"type\": \"rectangle\" },\n { \"id\": 33, \"name\": \"kite\", \"type\": \"rectangle\" },\n { \"id\": 34, \"name\": \"baseball bat\", \"type\": \"rectangle\" },\n { \"id\": 35, \"name\": \"baseball glove\", \"type\": \"rectangle\" },\n { \"id\": 36, \"name\": \"skateboard\", \"type\": \"rectangle\" },\n { \"id\": 37, \"name\": \"surfboard\", \"type\": \"rectangle\" },\n { \"id\": 38, \"name\": \"tennis racket\", \"type\": \"rectangle\" },\n { \"id\": 39, \"name\": \"bottle\", \"type\": \"rectangle\" },\n { \"id\": 40, \"name\": \"wine glass\", \"type\": \"rectangle\" },\n { \"id\": 41, \"name\": \"cup\", \"type\": \"rectangle\" },\n { \"id\": 42, \"name\": \"fork\", \"type\": \"rectangle\" },\n { \"id\": 43, \"name\": \"knife\", \"type\": \"rectangle\" },\n { \"id\": 44, \"name\": \"spoon\", \"type\": \"rectangle\" },\n { \"id\": 45, \"name\": \"bowl\", \"type\": \"rectangle\" },\n { \"id\": 46, \"name\": \"banana\", \"type\": \"rectangle\" },\n { \"id\": 47, \"name\": \"apple\", \"type\": \"rectangle\" },\n { \"id\": 48, \"name\": \"sandwich\", \"type\": \"rectangle\" },\n { \"id\": 49, \"name\": \"orange\", \"type\": \"rectangle\" },\n { \"id\": 50, \"name\": \"broccoli\", \"type\": \"rectangle\" },\n { \"id\": 51, \"name\": \"carrot\", \"type\": \"rectangle\" },\n { \"id\": 52, \"name\": \"hot dog\", \"type\": \"rectangle\" },\n { \"id\": 53, \"name\": \"pizza\", \"type\": \"rectangle\" },\n { \"id\": 54, \"name\": \"donut\", \"type\": \"rectangle\" },\n { \"id\": 55, \"name\": \"cake\", \"type\": \"rectangle\" },\n { \"id\": 56, \"name\": \"chair\", \"type\": \"rectangle\" },\n { \"id\": 57, \"name\": \"sofa\", \"type\": \"rectangle\" },\n { \"id\": 58, \"name\": \"pottedplant\", \"type\": \"rectangle\" },\n { \"id\": 59, \"name\": \"bed\", \"type\": \"rectangle\" },\n { \"id\": 60, \"name\": \"diningtable\", \"type\": \"rectangle\" },\n { \"id\": 61, \"name\": \"toilet\", \"type\": \"rectangle\" },\n { \"id\": 62, \"name\": \"tvmonitor\", \"type\": \"rectangle\" },\n { \"id\": 63, \"name\": \"laptop\", \"type\": \"rectangle\" },\n { \"id\": 64, \"name\": \"mouse\", \"type\": \"rectangle\" },\n { \"id\": 65, \"name\": \"remote\", \"type\": \"rectangle\" },\n { \"id\": 66, \"name\": \"keyboard\", \"type\": \"rectangle\" },\n { \"id\": 67, \"name\": \"cell phone\", \"type\": \"rectangle\" },\n { \"id\": 68, \"name\": \"microwave\", \"type\": \"rectangle\" },\n { \"id\": 69, \"name\": \"oven\", \"type\": \"rectangle\" },\n { \"id\": 70, \"name\": \"toaster\", \"type\": \"rectangle\" },\n { \"id\": 71, \"name\": \"sink\", \"type\": \"rectangle\" },\n { \"id\": 72, \"name\": \"refrigerator\", \"type\": \"rectangle\" },\n { \"id\": 73, \"name\": \"book\", \"type\": \"rectangle\" },\n { \"id\": 74, \"name\": \"clock\", \"type\": \"rectangle\" },\n { \"id\": 75, \"name\": \"vase\", \"type\": \"rectangle\" },\n { \"id\": 76, \"name\": \"scissors\", \"type\": \"rectangle\" },\n { \"id\": 77, \"name\": \"teddy bear\", \"type\": \"rectangle\" },\n { \"id\": 78, \"name\": \"hair drier\", \"type\": \"rectangle\" },\n { \"id\": 79, \"name\": \"toothbrush\", \"type\": \"rectangle\" }\n]\n",\n "type": "detector"\n }\n },\n "spec": {\n "description": "YOLO v7 via onnx-runtime",\n "handler": "main:handler",\n "runtime": "python:3.8",\n "resources": {\n "limits": {\n "nvidia.com/gpu": "1"\n },\n "requests": {\n "cpu": "25m",\n "memory": "1Mi"\n }\n },\n "image": "cvat.onnx.wongkinyiu.yolov7:latest-gpu",\n "targetCPU": 75,\n "triggers": {\n "myHttpTrigger": {\n "class": "",\n "kind": "http",\n "name": "myHttpTrigger",\n "numWorkers": 4,\n "workerAvailabilityTimeoutMilliseconds": 20000,\n "attributes": {\n "maxRequestBodySize": 33554432\n },\n "maxWorkers": 4\n }\n },\n "build": {\n "functionConfigPath": "serverless/onnx/wongkinyiu/yolov7/nuclio/function-gpu.yaml",\n "image": "cvat.onnx.wongkinyiu.yolov7:latest-gpu",\n "baseImage": "nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04",\n "directives": {\n "preCopy": [\n {\n "kind": "USER",\n "value": "root"\n },\n {\n "kind": "RUN",\n "value": "apt update \u0026\u0026 apt install --no-install-recommends -y wget python3-pip"\n },\n {\n "kind": "WORKDIR",\n "value": "/opt/nuclio"\n },\n {\n "kind": "RUN",\n "value": "pip install onnxruntime-gpu=='1.16.*' opencv-python-headless pillow pyyaml"\n },\n {\n "kind": "WORKDIR",\n "value": "/opt/nuclio"\n },\n {\n "kind": "RUN",\n "value": "wget https://github.com/WongKinYiu/yolov7/releases/download/v0.1/yolov7-nms-640.onnx"\n },\n {\n "kind": "RUN",\n "value": "ln -s /usr/bin/python3 /usr/bin/python"\n }\n ]\n },\n "codeEntryType": "image",\n "timestamp": 1727943691\n },\n "platform": {\n "attributes": {\n "mountMode": "volume",\n "restartPolicy": {\n "maximumRetryCount": 3,\n "name": "always"\n }\n }\n },\n "readinessTimeoutSeconds": 120,\n "securityContext": {},\n "disableDefaultHTTPTrigger": false,\n "eventTimeout": "30s"\n },\n "PlatformConfig": null\n}"}
2024-10-03 17:25:11 24.10.03 08:25:11.376 (I) cessor.healthcheck.server Listening {"listenAddress": ":8082"}
2024-10-03 17:25:11 24.10.03 08:25:11.377 (D) processor.http Creating worker pool {"num": 4}
2024-10-03 17:25:11 24.10.03 08:25:11.377 (D) sor.http.w3.python.logger Creating listener socket {"path": "/tmp/nuclio-rpc-crv59ptm1t85fcfkogog.sock"}
2024-10-03 17:25:11 24.10.03 08:25:11.377 (D) sor.http.w1.python.logger Creating listener socket {"path": "/tmp/nuclio-rpc-crv59ptm1t85fcfkogpg.sock"}
2024-10-03 17:25:11 24.10.03 08:25:11.377 (D) sor.http.w2.python.logger Creating listener socket {"path": "/tmp/nuclio-rpc-crv59ptm1t85fcfkogp0.sock"}
2024-10-03 17:25:11 24.10.03 08:25:11.377 (D) sor.http.w0.python.logger Creating listener socket {"path": "/tmp/nuclio-rpc-crv59ptm1t85fcfkogq0.sock"}
2024-10-03 17:25:11 24.10.03 08:25:11.378 (D) sor.http.w1.python.logger Creating listener socket {"path": "/tmp/nuclio-rpc-crv59ptm1t85fcfkogqg.sock"}
2024-10-03 17:25:11 24.10.03 08:25:11.379 (D) sor.http.w3.python.logger Creating listener socket {"path": "/tmp/nuclio-rpc-crv59ptm1t85fcfkogr0.sock"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w2.python.logger Creating listener socket {"path": "/tmp/nuclio-rpc-crv59ptm1t85fcfkogrg.sock"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w2.python.logger Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w1.python.logger Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w2.python.logger Using Python handler {"handler": "main:handler"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w3.python.logger Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w3.python.logger Using Python handler {"handler": "main:handler"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w2.python.logger Using Python executable {"path": "/usr/bin/python3"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w0.python.logger Creating listener socket {"path": "/tmp/nuclio-rpc-crv59ptm1t85fcfkogs0.sock"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w1.python.logger Using Python handler {"handler": "main:handler"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w3.python.logger Using Python executable {"path": "/usr/bin/python3"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w3.python.logger Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w3.python.logger Running wrapper {"command": "/usr/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --event-socket-path /tmp/nuclio-rpc-crv59ptm1t85fcfkogog.sock --control-socket-path /tmp/nuclio-rpc-crv59ptm1t85fcfkogr0.sock --platform-kind local --namespace nuclio --worker-id 3 --trigger-kind http --trigger-name myHttpTrigger --decode-event-strings"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w0.python.logger Using Python wrapper script path {"path": "/opt/nuclio/_nuclio_wrapper.py"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w0.python.logger Using Python handler {"handler": "main:handler"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w1.python.logger Using Python executable {"path": "/usr/bin/python3"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w0.python.logger Using Python executable {"path": "/usr/bin/python3"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w0.python.logger Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w0.python.logger Running wrapper {"command": "/usr/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --event-socket-path /tmp/nuclio-rpc-crv59ptm1t85fcfkogq0.sock --control-socket-path /tmp/nuclio-rpc-crv59ptm1t85fcfkogs0.sock --platform-kind local --namespace nuclio --worker-id 0 --trigger-kind http --trigger-name myHttpTrigger --decode-event-strings"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w1.python.logger Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w2.python.logger Setting PYTHONPATH {"value": "PYTHONPATH=/opt/nuclio"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w2.python.logger Running wrapper {"command": "/usr/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --event-socket-path /tmp/nuclio-rpc-crv59ptm1t85fcfkogp0.sock --control-socket-path /tmp/nuclio-rpc-crv59ptm1t85fcfkogrg.sock --platform-kind local --namespace nuclio --worker-id 2 --trigger-kind http --trigger-name myHttpTrigger --decode-event-strings"}
2024-10-03 17:25:11 24.10.03 08:25:11.380 (D) sor.http.w1.python.logger Running wrapper {"command": "/usr/bin/python3 -u /opt/nuclio/_nuclio_wrapper.py --handler main:handler --event-socket-path /tmp/nuclio-rpc-crv59ptm1t85fcfkogpg.sock --control-socket-path /tmp/nuclio-rpc-crv59ptm1t85fcfkogqg.sock --platform-kind local --namespace nuclio --worker-id 1 --trigger-kind http --trigger-name myHttpTrigger --decode-event-strings"}
2024-10-03 17:25:10 panic: Wrapper process for worker 0 exited unexpectedly with: signal: segmentation fault
2024-10-03 17:25:10
2024-10-03 17:25:10 goroutine 65 [running]:
2024-10-03 17:25:10 github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).watchWrapperProcess(0xc000b980e0)
2024-10-03 17:25:10 /home/runner/work/nuclio/nuclio/pkg/processor/runtime/rpc/abstract.go:663 +0x434
2024-10-03 17:25:10 created by github.com/nuclio/nuclio/pkg/processor/runtime/rpc.(*AbstractRuntime).startWrapper in goroutine 13
2024-10-03 17:25:10 /home/runner/work/nuclio/nuclio/pkg/processor/runtime/rpc/abstract.go:318 +0x1f1
2024-10-03 17:25:11
2024-10-03 17:25:11 A module that was compiled using NumPy 1.x cannot be run in
2024-10-03 17:25:11 NumPy 2.0.1 as it may crash. To support both 1.x and 2.x
2024-10-03 17:25:11 versions of NumPy, modules must be compiled with NumPy 2.0.
2024-10-03 17:25:11 Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
2024-10-03 17:25:11
2024-10-03 17:25:11 If you are a user of the module, the easiest solution will be to
2024-10-03 17:25:11 downgrade to 'numpy<2' or try to upgrade the affected module.
2024-10-03 17:25:11 We expect that some modules will need time to support NumPy 2.
2024-10-03 17:25:11
2024-10-03 17:25:11 Traceback (most recent call last): File "/opt/nuclio/_nuclio_wrapper.py", line 567, in
2024-10-03 17:25:11 run_wrapper()
2024-10-03 17:25:11 File "/opt/nuclio/_nuclio_wrapper.py", line 534, in run_wrapper
2024-10-03 17:25:11 wrapper_instance = Wrapper(root_logger,
2024-10-03 17:25:11 File "/opt/nuclio/_nuclio_wrapper.py", line 88, in init
2024-10-03 17:25:11 self._entrypoint = self._load_entrypoint_from_handler(handler)
2024-10-03 17:25:11 File "/opt/nuclio/_nuclio_wrapper.py", line 331, in _load_entrypoint_from_handler
2024-10-03 17:25:11 module = import(module_name)
2024-10-03 17:25:11 File "/opt/nuclio/main.py", line 6, in
2024-10-03 17:25:11 from model_handler import ModelHandler
2024-10-03 17:25:11 File "/opt/nuclio/model_handler.py", line 7, in
2024-10-03 17:25:11 import onnxruntime as ort
2024-10-03 17:25:11 File "/usr/local/lib/python3.10/dist-packages/onnxruntime/init.py", line 23, in
2024-10-03 17:25:11 from onnxruntime.capi._pybind_state import ExecutionMode # noqa: F401
2024-10-03 17:25:11 File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/_pybind_state.py", line 32, in
2024-10-03 17:25:11 from .onnxruntime_pybind11_state import * # noqa
2024-10-03 17:25:11 AttributeError: _ARRAY_API not found
2024-10-03 17:25:11 ImportError: numpy.core.multiarray failed to import
2024-10-03 17:25:11
2024-10-03 17:25:11 The above exception was the direct cause of the following exception:
2024-10-03 17:25:11
2024-10-03 17:25:11 SystemError: returned a result with an exception set
2024-10-03 17:25:11
2024-10-03 17:25:11 A module that was compiled using NumPy 1.x cannot be run in
2024-10-03 17:25:11 NumPy 2.0.1 as it may crash. To support both 1.x and 2.x
2024-10-03 17:25:11 versions of NumPy, modules must be compiled with NumPy 2.0.
2024-10-03 17:25:11 Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
2024-10-03 17:25:11
2024-10-03 17:25:11 If you are a user of the module, the easiest solution will be to
2024-10-03 17:25:11 downgrade to 'numpy<2' or try to upgrade the affected module.
2024-10-03 17:25:11 We expect that some modules will need time to support NumPy 2.
2024-10-03 17:25:11
2024-10-03 17:25:11 Traceback (most recent call last): File "/opt/nuclio/_nuclio_wrapper.py", line 567, in
2024-10-03 17:25:11 run_wrapper()
2024-10-03 17:25:11 File "/opt/nuclio/_nuclio_wrapper.py", line 534, in run_wrapper
2024-10-03 17:25:11 wrapper_instance = Wrapper(root_logger,
2024-10-03 17:25:11 File "/opt/nuclio/_nuclio_wrapper.py", line 88, in init
2024-10-03 17:25:11 self._entrypoint = self._load_entrypoint_from_handler(handler)
2024-10-03 17:25:11 File "/opt/nuclio/_nuclio_wrapper.py", line 331, in _load_entrypoint_from_handler
2024-10-03 17:25:11 module = import(module_name)
2024-10-03 17:25:11 File "/opt/nuclio/main.py", line 6, in
2024-10-03 17:25:11 from model_handler import ModelHandler
2024-10-03 17:25:11 File "/opt/nuclio/model_handler.py", line 7, in
2024-10-03 17:25:11 import onnxruntime as ort
2024-10-03 17:25:11 File "/usr/local/lib/python3.10/dist-packages/onnxruntime/init.py", line 23, in
2024-10-03 17:25:11 from onnxruntime.capi._pybind_state import ExecutionMode # noqa: F401
2024-10-03 17:25:11 File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/_pybind_state.py", line 32, in
2024-10-03 17:25:11 from .onnxruntime_pybind11_state import * # noqa
2024-10-03 17:25:11 AttributeError: _ARRAY_API not found
2024-10-03 17:25:11 ImportError: numpy.core.multiarray failed to import
2024-10-03 17:25:11
2024-10-03 17:25:11 The above exception was the direct cause of the following exception:
2024-10-03 17:25:11
2024-10-03 17:25:11 SystemError: returned a result with an exception set
2024-10-03 17:25:11
2024-10-03 17:25:11 A module that was compiled using NumPy 1.x cannot be run in
2024-10-03 17:25:11 NumPy 2.0.1 as it may crash. To support both 1.x and 2.x
2024-10-03 17:25:11 versions of NumPy, modules must be compiled with NumPy 2.0.
2024-10-03 17:25:11 Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
2024-10-03 17:25:11
2024-10-03 17:25:11 If you are a user of the module, the easiest solution will be to
2024-10-03 17:25:11 downgrade to 'numpy<2' or try to upgrade the affected module.
2024-10-03 17:25:11 We expect that some modules will need time to support NumPy 2.
2024-10-03 17:25:11
2024-10-03 17:25:11 Traceback (most recent call last): File "/opt/nuclio/_nuclio_wrapper.py", line 567, in
2024-10-03 17:25:11 run_wrapper()
2024-10-03 17:25:11 File "/opt/nuclio/_nuclio_wrapper.py", line 534, in run_wrapper
2024-10-03 17:25:11 wrapper_instance = Wrapper(root_logger,
2024-10-03 17:25:11 File "/opt/nuclio/_nuclio_wrapper.py", line 88, in init
2024-10-03 17:25:11 self._entrypoint = self._load_entrypoint_from_handler(handler)
2024-10-03 17:25:11 File "/opt/nuclio/_nuclio_wrapper.py", line 331, in _load_entrypoint_from_handler
2024-10-03 17:25:11 module = import(module_name)
2024-10-03 17:25:11 File "/opt/nuclio/main.py", line 6, in
2024-10-03 17:25:11 from model_handler import ModelHandler
2024-10-03 17:25:11 File "/opt/nuclio/model_handler.py", line 7, in
2024-10-03 17:25:11 import onnxruntime as ort
2024-10-03 17:25:11 File "/usr/local/lib/python3.10/dist-packages/onnxruntime/init.py", line 23, in
2024-10-03 17:25:11 from onnxruntime.capi._pybind_state import ExecutionMode # noqa: F401
2024-10-03 17:25:11 File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/_pybind_state.py", line 32, in
2024-10-03 17:25:11 from .onnxruntime_pybind11_state import * # noqa
2024-10-03 17:25:11 AttributeError: _ARRAY_API not found
2024-10-03 17:25:11 ImportError: numpy.core.multiarray failed to import
2024-10-03 17:25:11
2024-10-03 17:25:11 The above exception was the direct cause of the following exception:
2024-10-03 17:25:11
2024-10-03 17:25:11 SystemError: returned a result with an exception set
2024-10-03 17:25:11
2024-10-03 17:25:11 A module that was compiled using NumPy 1.x cannot be run in
2024-10-03 17:25:11 NumPy 2.0.1 as it may crash. To support both 1.x and 2.x
2024-10-03 17:25:11 versions of NumPy, modules must be compiled with NumPy 2.0.
2024-10-03 17:25:11 Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
2024-10-03 17:25:11
2024-10-03 17:25:11 If you are a user of the module, the easiest solution will be to
2024-10-03 17:25:11 downgrade to 'numpy<2' or try to upgrade the affected module.
2024-10-03 17:25:11 We expect that some modules will need time to support NumPy 2.
2024-10-03 17:25:11
2024-10-03 17:25:11 Traceback (most recent call last): File "/opt/nuclio/_nuclio_wrapper.py", line 567, in
2024-10-03 17:25:11 run_wrapper()
2024-10-03 17:25:11 File "/opt/nuclio/_nuclio_wrapper.py", line 534, in run_wrapper
2024-10-03 17:25:11 wrapper_instance = Wrapper(root_logger,
2024-10-03 17:25:11 File "/opt/nuclio/_nuclio_wrapper.py", line 88, in init
2024-10-03 17:25:11 self._entrypoint = self._load_entrypoint_from_handler(handler)
2024-10-03 17:25:11 File "/opt/nuclio/_nuclio_wrapper.py", line 331, in _load_entrypoint_from_handler
2024-10-03 17:25:11 module = import(module_name)
2024-10-03 17:25:11 File "/opt/nuclio/main.py", line 6, in
2024-10-03 17:25:11 from model_handler import ModelHandler
2024-10-03 17:25:11 File "/opt/nuclio/model_handler.py", line 7, in
2024-10-03 17:25:11 import onnxruntime as ort
2024-10-03 17:25:11 File "/usr/local/lib/python3.10/dist-packages/onnxruntime/init.py", line 23, in
2024-10-03 17:25:11 from onnxruntime.capi._pybind_state import ExecutionMode # noqa: F401
2024-10-03 17:25:11 File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/_pybind_state.py", line 32, in
2024-10-03 17:25:11 from .onnxruntime_pybind11_state import * # noqa
2024-10-03 17:25:11 AttributeError: _ARRAY_API not found
2024-10-03 17:25:11 ImportError: numpy.core.multiarray failed to import
2024-10-03 17:25:11
2024-10-03 17:25:11 The above exception was the direct cause of the following exception:
2024-10-03 17:25:11
2024-10-03 17:25:11 SystemError: returned a result with an exception set
2024-10-03 17:25:11 24.10.03 08:25:11.754 (I) sor.http.w2.python.logger Wrapper connected {"wid": 2, "pid": 42}
2024-10-03 17:25:11 24.10.03 08:25:11.754 (D) sor.http.w2.python.logger Creating control connection {"wid": 2}
2024-10-03 17:25:11 24.10.03 08:25:11.754 (D) sor.http.w2.python.logger Control connection created {"wid": 2}
2024-10-03 17:25:11 24.10.03 08:25:11.754 (D) sor.http.w2.python.logger Waiting for start
2024-10-03 17:25:11 24.10.03 08:25:11.755 (I) sor.http.w2.python.logger Init context... 0% {"worker_id": "2"}
2024-10-03 17:25:11 24.10.03 08:25:11.756 (I) sor.http.w0.python.logger Wrapper connected {"wid": 0, "pid": 41}
2024-10-03 17:25:11 24.10.03 08:25:11.756 (D) sor.http.w0.python.logger Creating control connection {"wid": 0}
2024-10-03 17:25:11 24.10.03 08:25:11.756 (D) sor.http.w0.python.logger Control connection created {"wid": 0}
2024-10-03 17:25:11 24.10.03 08:25:11.756 (D) sor.http.w0.python.logger Waiting for start
2024-10-03 17:25:11 24.10.03 08:25:11.756 (I) sor.http.w0.python.logger Init context... 0% {"worker_id": "0"}
2024-10-03 17:25:11 24.10.03 08:25:11.767 (I) sor.http.w1.python.logger Wrapper connected {"wid": 1, "pid": 43}
2024-10-03 17:25:11 24.10.03 08:25:11.767 (D) sor.http.w1.python.logger Creating control connection {"wid": 1}
2024-10-03 17:25:11 24.10.03 08:25:11.767 (D) sor.http.w1.python.logger Control connection created {"wid": 1}
2024-10-03 17:25:11 24.10.03 08:25:11.767 (D) sor.http.w1.python.logger Waiting for start
2024-10-03 17:25:11 24.10.03 08:25:11.767 (I) sor.http.w1.python.logger Init context... 0% {"worker_id": "1"}
2024-10-03 17:25:11 24.10.03 08:25:11.774 (I) sor.http.w3.python.logger Wrapper connected {"wid": 3, "pid": 40}
2024-10-03 17:25:11 24.10.03 08:25:11.774 (D) sor.http.w3.python.logger Creating control connection {"wid": 3}
2024-10-03 17:25:11 24.10.03 08:25:11.774 (D) sor.http.w3.python.logger Control connection created {"wid": 3}
2024-10-03 17:25:11 24.10.03 08:25:11.774 (D) sor.http.w3.python.logger Waiting for start
2024-10-03 17:25:11 24.10.03 08:25:11.775 (I) sor.http.w3.python.logger Init context... 0% {"worker_id": "3"}
2024-10-03 17:25:12 24.10.03 08:25:12.742 (I) sor.http.w0.python.logger Init context...100% {"worker_id": "0"}
2024-10-03 17:25:12 24.10.03 08:25:12.742 (D) sor.http.w0.python.logger Started
2024-10-03 17:25:12 24.10.03 08:25:12.742 (D) sor.http.w0.python.logger Sending data on control socket {"data_length": 2, "worker_id": "0"}
2024-10-03 17:25:12 24.10.03 08:25:12.742 (D) sor.http.w0.python.logger Received control message {"messageKind": "wrapperInitialized"}
2024-10-03 17:25:12 24.10.03 08:25:12.761 (I) sor.http.w2.python.logger Init context...100% {"worker_id": "2"}
2024-10-03 17:25:12 24.10.03 08:25:12.761 (D) sor.http.w2.python.logger Started
2024-10-03 17:25:12 24.10.03 08:25:12.761 (D) sor.http.w2.python.logger Sending data on control socket {"data_length": 2, "worker_id": "2"}
2024-10-03 17:25:12 24.10.03 08:25:12.761 (D) sor.http.w2.python.logger Received control message {"messageKind": "wrapperInitialized"}
2024-10-03 17:25:12 24.10.03 08:25:12.769 (I) sor.http.w3.python.logger Init context...100% {"worker_id": "3"}
2024-10-03 17:25:12 24.10.03 08:25:12.770 (D) sor.http.w3.python.logger Started
2024-10-03 17:25:12 24.10.03 08:25:12.770 (D) sor.http.w3.python.logger Sending data on control socket {"data_length": 2, "worker_id": "3"}
2024-10-03 17:25:12 24.10.03 08:25:12.770 (D) sor.http.w3.python.logger Received control message {"messageKind": "wrapperInitialized"}
2024-10-03 17:25:12 24.10.03 08:25:12.773 (I) sor.http.w1.python.logger Init context...100% {"worker_id": "1"}
2024-10-03 17:25:12 24.10.03 08:25:12.773 (D) sor.http.w1.python.logger Started
2024-10-03 17:25:12 24.10.03 08:25:12.773 (I) processor Starting event timeout watcher {"timeout": "30s"}
2024-10-03 17:25:12 24.10.03 08:25:12.773 (D) sor.http.w1.python.logger Received control message {"messageKind": "wrapperInitialized"}
2024-10-03 17:25:12 24.10.03 08:25:12.773 (D) sor.http.w1.python.logger Sending data on control socket {"data_length": 2, "worker_id": "1"}
2024-10-03 17:25:12 24.10.03 08:25:12.774 (D) .webadmin.server.triggers Registered custom route {"routeName": "triggers", "stream": false, "pattern": "/{id}/stats", "method": "GET"}
2024-10-03 17:25:12 24.10.03 08:25:12.774 (D) processor.webadmin.server Registered resource {"name": "triggers"}
2024-10-03 17:25:12 24.10.03 08:25:12.774 (W) processor No metric sinks configured, metrics will not be published
2024-10-03 17:25:12 24.10.03 08:25:12.774 (D) processor Starting triggers {"triggersError": "json: unsupported value: encountered a cycle via *http.http"}
2024-10-03 17:25:12 24.10.03 08:25:12.776 (I) processor.http Starting {"listenAddress": ":8080", "readBufferSize": 16384, "maxRequestBodySize": 33554432, "reduceMemoryUsage": false, "cors": null}
2024-10-03 17:25:12 24.10.03 08:25:12.776 (I) processor.webadmin.server Listening {"listenAddress": ":8081"}
2024-10-03 17:25:12 24.10.03 08:25:12.776 (D) processor Processor started

I don't think the version of numpy is suitable, what do you think?

Environment

WSL2 Ubuntu 22.04
Docker desktop 4.33.1(docker engine 27.1.1)
GPU RTX4090
@prnr prnr added the bug Something isn't working label Oct 3, 2024
@bladekeys
Copy link

bladekeys commented Oct 23, 2024 •
edited
Loading

Fixed this error. Add numpy=='1.26.4' to function-gpu.yaml

- kind: RUN
  value: pip install onnxruntime-gpu=='1.16.*' opencv-python-headless pillow pyyaml numpy=='1.26.4'

To specify version "less than", you need to use quotation because of some errors in parsing.

- kind: RUN
  value: pip install onnxruntime-gpu=='1.16.*' opencv-python-headless pillow pyyaml "numpy<2.0"

@bsekachev bsekachev mentioned this issue Dec 12, 2024
Merged
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixed installation of YOLOv7 on GPU cvat-ai/cvat
2 participants
@prnr @bladekeys