Unable to invoke nuclio functions from cvat #4855
Comments
|
@RadekZenkl It's very strange that the nuclio function was deployed on port 8000, please specify your environmet (OS at least) and exact steps to reporduce the issue. This error occurs only with yolov3? What about other functions? |
|
Hi @azhavoro I reinstalled cvat from scratch and redeployed the nuclio functions. I also switched My environment is: Fedora 35, 5.18.18-100.fc35.x86_64 I start CVAT with the following commands: the deployed tested functions are: Upon invoking them from CVAT I get the following errors: Respectively:
docker ps: The Nuclio functions are deployed with the following command: I also tested this with Thank you for your help |
|
What is output of |
|
hi, the output from 127.0.0.1 localhost and 127.0.0.1 localhost |
|
The reason is that the |
|
Hey, I updated to v2.2.0. The docker internal is now part of
the error still prevails |
|
I will try to reproduce the issue on Fedora |
|
Unfortunately, the problem does not occur in a freshly installed Fedora 35 (VM). I think the issue may be related to firewall settings. What is output of |
|
Executing
|
|
@RadekZenkl have you solved this problem? I have the same problem with hrnet but yolo dextr are working properly I solved my problem, there the model tried to use the first gpu (i have 2 gpu) that was busy, I changed the gpu ID in function-gpu.yaml (https://docs.docker.com/compose/gpu-support/) |
|
Hi, no I still haven't solved the problem. |
|
Please try our latest release. I believe right now some issues with nuclio should be resolved. |
|
Hello, I have just faced this problem. When I try to invoke a nuclio function from cvat, I get Detection error occurred
<html> <head><title>504 Gateway Time-out</title></head> <body> <center><h1>504 Gateway Time-out</h1>nuctl get functions
NAMESPACE | NAME | PROJECT | STATE | REPLICAS | NODE PORT
nuclio | custom-func | cvat | ready | 1/1 | 32805 Nuclio function's status in dashboard is Update
After allowing the port, I was able to successfully invoke the Nuclio function without any timeout errors. |
Hi
Unfortunately, I am not able to make the semi automatic annotation work.
I followed the guide to set it up: https://cvat-ai.github.io/cvat/docs/administration/advanced/installation_automatic_annotation/
I can indeed invoke the function with nuctl but not from cvat.
When testing the yolov3 from cvat I get the following error:
Detection error occured Error: Request failed with status code 500. "500 Server Error: Internal Server Error for url: http://nuclio:8070/api/function_invocations".I am working with
commit 3bd7c7e (HEAD -> master, tag: v2.1.0, origin/master)
nuclio 1.5.16 as stated in
cvat/components/serverless/docker-compose.serverless.ymllooking at the nuclio logs with logs --follow nuclio --tail=1:
22.08.18 13:24:11.788 �[0;37mashboard.platform.invoker�[0m �[0;34m(I)�[0m Executing function {"method": "POST", "url": "http://172.17.0.1:8000", "headers": {"Accept":["/"],"Accept-Encoding":["gzip, deflate"],"Connection":["close"],"Content-Length":["612797"],"Content-Type":["application/json"],"User-Agent":["python-requests/2.26.0"],"X-Nuclio-Function-Name":["openvino-omz-public-yolo-v3-tf"],"X-Nuclio-Function-Namespace":["nuclio"],"X-Nuclio-Log-Level":[""],"X-Nuclio-Path":["/"],"X-Nuclio-Project-Name":["cvat"],"X-Nuclio-Target":["openvino-omz-public-yolo-v3-tf"]}}
22.08.18 13:24:11.788 �[0;37m.api/function_invocations�[0m �[0;33m(W)�[0m Failed to invoke function {"err": "Failed to send HTTP request", "errVerbose": "\nError - Post "http://172.17.0.1:8000": dial tcp 172.17.0.1:8000: connect: no route to host\n /nuclio/pkg/platform/abstract/invoker.go:119\n\nCall stack:\nFailed to send HTTP request\n /nuclio/pkg/platform/abstract/invoker.go:119\nFailed to send HTTP request", "errCauses": [{"error": "Post "http://172.17.0.1:8000": dial tcp 172.17.0.1:8000: connect: no route to host"}]}
22.08.18 13:24:11.788 �[0;37m dashboard.server�[0m �[0;32m(D)�[0m Handled request {"requestMethod": "POST", "requestPath": "/api/function_invocations", "requestHeaders": {"Accept":["/"],"Accept-Encoding":["gzip, deflate"],"Connection":["close"],"Content-Length":["612797"],"Content-Type":["application/json"],"User-Agent":["python-requests/2.26.0"],"X-Nuclio-Function-Name":["openvino-omz-public-yolo-v3-tf"],"X-Nuclio-Function-Namespace":["nuclio"],"X-Nuclio-Log-Level":[""],"X-Nuclio-Path":["/"],"X-Nuclio-Project-Name":["cvat"],"X-Nuclio-Target":["openvino-omz-public-yolo-v3-tf"]}, "requestBody": "{"image":
....
For me it looks like that cvat tries to correctly reach http://172.17.0.1:8000 but cvat cannot talk to nuclio.
I have very limited knowledge about networking, so any pointers to debugging this would be great. Thank you!
Details:
docker ps:CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
90da9c9eb929 cvat/openvino.omz.public.yolo-v3-tf:latest "processor" 10 minutes ago Up 10 minutes (healthy) 0.0.0.0:8000->8080/tcp, :::8000->8080/tcp nuclio-nuclio-openvino-omz-public-yolo-v3-tf
9914fa07ec33 cvat/openvino.dextr:latest "processor" 10 minutes ago Up 10 minutes (healthy) 0.0.0.0:49201->8080/tcp, :::49201->8080/tcp nuclio-nuclio-openvino-dextr
4e7eb17e209b alpine:3.11 "/bin/sh -c '/bin/sl…" 47 minutes ago Up 47 minutes nuclio-local-storage-reader
a6e2ee805a5c openvino/cvat_ui "/docker-entrypoint.…" 2 hours ago Up 2 hours 80/tcp cvat_ui
948469b65c8f openvino/cvat_server "/usr/bin/supervisord" 2 hours ago Up 2 hours 8080/tcp cvat
68475d07147d redis:4.0-alpine "docker-entrypoint.s…" 2 hours ago Up 2 hours 6379/tcp cvat_redis
b8eb3153ad24 postgres:10-alpine "docker-entrypoint.s…" 2 hours ago Up 2 hours 5432/tcp cvat_db
1014c7b27c0e quay.io/nuclio/dashboard:1.5.16-amd64 "/docker-entrypoint.…" 2 hours ago Up 2 hours (healthy) 80/tcp, 0.0.0.0:8070->8070/tcp, :::8070->8070/tcp nuclio
d7084596806e traefik:v2.4 "/entrypoint.sh --pr…" 2 hours ago Up 2 hours 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 80/tcp, 0.0.0.0:8090->8090/tcp, :::8090->8090/tcp traefik
578b14aa28d3 openpolicyagent/opa:0.34.2-rootless "/opa run --server -…" 2 hours ago Up 2 hours 0.0.0.0:8181->8181/tcp, :::8181->8181/tcp cvat_opa
71297bac6ff6 cvat/ultralytics-yolov5:latest "processor" 4 hours ago Up 4 hours (healthy) 0.0.0.0:49204->8080/tcp, :::49204->8080/tcp nuclio-nuclio-ultralytics-yolov5
nuctl get functions:NAMESPACE | NAME | PROJECT | STATE | NODE PORT | REPLICAS
nuclio | openvino-dextr | cvat | ready | 49201 | 1/1
nuclio | openvino-omz-public-yolo-v3-tf | cvat | ready | 8000 | 1/1
nuclio | ultralytics-yolov5 | cvat | ready | 49204 | 1/1
Issue by @RadekZenkl
The text was updated successfully, but these errors were encountered: