Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Helm Install results in a CVAT instance where I am unable to login #3725

Closed
2 tasks done
bhlarson opened this issue Sep 24, 2021 · 9 comments
Closed
2 tasks done

Helm Install results in a CVAT instance where I am unable to login #3725

bhlarson opened this issue Sep 24, 2021 · 9 comments
Assignees
@azhavoro
Labels
question Further information is requested

Comments

@bhlarson
Copy link

My actions before raising this issue

Following the helm-chart install procedure produces a CVAT instance that is impossible to log in to:
login failure screen shot

Expected Behaviour

I would expect the superuser login described in How to crate a superuser or "Create an account" to successfully log into the CVAT annotation tool.

Current Behaviour

Instead, The "Could not check autorization on the sever" message was returned each time. The error details on the browser console only returns "Not Allowed" which is not helpful in diagnosing the problem.

Possible Solution

Steps to Reproduce (for bugs)

Helm Install:

Helm Overrides:

$ cat helm-chart/values.override.yaml
postgresql:
  secret:
    password: cvat_passwd
    postgres_password: cvat_passwd
    replication_password: cvat_passwd

Helm Install from clone of CVAT repository

$ helm upgrade cvat --install ./helm-chart -f ./helm-chart/values.yaml -f ./helm-chart/values.override.yaml --namespace cv --create-namespace
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2488/credentials/client.config
Release "cvat" does not exist. Installing it now.
NAME: cvat
LAST DEPLOYED: Fri Sep 24 14:27:00 2021
NAMESPACE: cv
STATUS: deployed
REVISION: 1
TEST SUITE: None

CVAT runtime following helm install:

$ kubectl get all -n cv
NAME                                 READY   STATUS    RESTARTS   AGE
pod/cvat-frontend-54674747d6-n8ccf   1/1     Running   0          65s
pod/cvat-backend-6f7d85f558-4tqnk    1/1     Running   0          65s
pod/cvat-postgresql-0                1/1     Running   0          65s
pod/cvat-redis-master-0              1/1     Running   0          65s

NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/cvat-redis-headless        ClusterIP   None             <none>        6379/TCP   66s
service/cvat-postgresql-headless   ClusterIP   None             <none>        5432/TCP   66s
service/cvat-postgresql            ClusterIP   10.152.183.4     <none>        5432/TCP   65s
service/cvat-redis-master          ClusterIP   10.152.183.93    <none>        6379/TCP   65s
service/cvat-frontend-service      ClusterIP   10.152.183.232   <none>        80/TCP     65s
service/cvat-backend-service       ClusterIP   10.152.183.143   <none>        8080/TCP   65s

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cvat-frontend   1/1     1            1           65s
deployment.apps/cvat-backend    1/1     1            1           65s

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/cvat-frontend-54674747d6   1         1         1       65s
replicaset.apps/cvat-backend-6f7d85f558    1         1         1       65s

NAME                                 READY   AGE
statefulset.apps/cvat-postgresql     1/1     65s
statefulset.apps/cvat-redis-master   1/1     65s

Port forward backend:

$ kubectl port-forward service/cvat-backend-service 8080:8080 -n cv
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

Port forward frontend:

$ kubectl port-forward service/cvat-frontend-service 8880:80 -n cv
Forwarding from 127.0.0.1:8880 -> 80

Chrome version: Chrome/94.0.4606.61 Safari/537.36" "-"

Context

Without the helm install, we cannot evaluate CVAT in our Kubernetes production environment.

Your Environment

  • Git hash commit (git log -1):
$ git log -1
commit 8d03ed7959632d6f3ffd76020ad577fa409acfcd (HEAD, tag: v1.6.0, origin/master)
Merge: 56c9626a 476b2fcf
Author: Nikita Manovich <nikita.manovich@intel.com>
Date:   Fri Sep 17 14:22:09 2021 +0300

    Merge branch 'release-1.6.0'
  • Docker version docker version (e.g. Docker 17.0.05):
$ docker version
Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:54:08 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:16 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 nvidia:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
  • Are you using Docker Swarm or Kubernetes?
    Kubernetes
  • Operating System and version (e.g. Linux, Windows, MacOS):
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.3 LTS
Release:        20.04
Codename:       focal

Kubernetes and helm versions:

$ snap list microk8s
Name      Version  Rev   Tracking     Publisher   Notes
microk8s  v1.22.2  2488  1.22/stable  canonical✓  classic
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.2-3+a4bd0397e1cb5e", GitCommit:"a4bd0397e1cb5e638a6a6464f759114ee2250ea1", GitTreeState:"clean", BuildDate:"2021-09-16T00:05:14Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22+", GitVersion:"v1.22.2-3+a4bd0397e1cb5e", GitCommit:"a4bd0397e1cb5e638a6a6464f759114ee2250ea1", GitTreeState:"clean", BuildDate:"2021-09-16T00:00:16Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}
Helm:
$ helm version
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/snap/microk8s/2488/credentials/client.config
version.BuildInfo{Version:"v3.5.0", GitCommit:"32c22239423b3b4ba6706d450bd044baffdcf9e6", GitTreeState:"clean", GoVersion:"go1.15.6"}
  • Code example or link to GitHub repo or gist to reproduce problem:
  • Other diagnostic information / logs:
    Logs from `cvat` container

Backend log:

$ kubectl logs deployment.apps/cvat-backend -n cv --tail=20
debug1: process_message: socket 1 (fd=4) type 11

2021-09-24 21:36:09,072 DEBG 'rqworker_default_0' stderr output:
DEBUG - 2021-09-24 21:36:09,072 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-09-24 21:36:09,073 DEBG 'rqworker_default_1' stderr output:
DEBUG - 2021-09-24 21:36:09,072 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-09-24 21:36:09,474 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-09-24 21:36:09,474 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-09-24 21:42:54,091 DEBG 'rqworker_default_1' stderr output:
DEBUG - 2021-09-24 21:42:54,091 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-09-24 21:42:54,092 DEBG 'rqworker_default_0' stderr output:
DEBUG - 2021-09-24 21:42:54,091 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-09-24 21:42:54,492 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-09-24 21:42:54,491 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

Frontend Log:

$ kubectl logs deployment.apps/cvat-frontend -n cv --tail 25
127.0.0.1 - - [24/Sep/2021:21:31:34 +0000] "OPTIONS /api/v1/auth/password/change HTTP/1.1" 405 559 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:31:34 +0000] "OPTIONS /api/v1/auth/password/reset HTTP/1.1" 405 559 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:23 +0000] "GET /cvat-ui.f9340a4e583f3306fd17.min.js HTTP/1.1" 200 5955484 "http://localhost:8880/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:24 +0000] "OPTIONS /api/v1/auth/password/change HTTP/1.1" 405 559 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:24 +0000] "OPTIONS /api/v1/auth/password/reset HTTP/1.1" 405 559 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:31 +0000] "GET / HTTP/1.1" 200 1045 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:31 +0000] "GET /cvat-ui.f9340a4e583f3306fd17.min.js HTTP/1.1" 200 5955484 "http://localhost:8880/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:32 +0000] "GET /api/v1/users/self HTTP/1.1" 200 1045 "http://localhost:8880/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:32 +0000] "GET /download.worker.cb069004e9ace5f89e8e.js HTTP/1.1" 200 16721 "http://localhost:8880/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:32 +0000] "GET /favicon.ico HTTP/1.1" 200 102581 "http://localhost:8880/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:32 +0000] "GET /api/v1/restrictions/user-agreements HTTP/1.1" 200 1045 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:32 +0000] "OPTIONS /api/v1/auth/password/change HTTP/1.1" 405 559 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:32 +0000] "OPTIONS /api/v1/auth/password/reset HTTP/1.1" 405 559 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:42 +0000] "GET /cvat-ui.f9340a4e583f3306fd17.min.js.map HTTP/1.1" 200 19371636 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:34:42 +0000] "GET /download.worker.cb069004e9ace5f89e8e.js.map HTTP/1.1" 200 77935 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:38:46 +0000] "GET /auth/login HTTP/1.1" 200 1045 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:38:46 +0000] "GET /cvat-ui.f9340a4e583f3306fd17.min.js.map HTTP/1.1" 200 19371636 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:38:46 +0000] "GET /favicon.ico HTTP/1.1" 200 102581 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:38:46 +0000] "OPTIONS /api/v1/auth/password/reset HTTP/1.1" 405 559 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:38:46 +0000] "OPTIONS /api/v1/auth/password/change HTTP/1.1" 405 559 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:38:52 +0000] "GET /auth/login HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:38:52 +0000] "GET /cvat-ui.f9340a4e583f3306fd17.min.js.map HTTP/1.1" 200 19371636 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:38:52 +0000] "GET /favicon.ico HTTP/1.1" 200 102581 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:38:52 +0000] "OPTIONS /api/v1/auth/password/change HTTP/1.1" 405 559 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36" "-"
127.0.0.1 - - [24/Sep/2021:21:38:52 +0000] "OPTIONS /api/v1/auth/password/reset HTTP/1.1" 405 559 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)

Next steps

  • Added the above information to the https://gitter.im/opencv-cvat/public thread by Asal Jalilvand @asaljalilvand Aug 25 14:29
  • I believe there is failed communications between server pods because the same software version works well in Docker Compose.
  • Compare docker compose yaml and helm install to identify the missing communication
@Hadleigh-Schwartz
Copy link

I'm having the exact same issue. Software works well in Docker Compose, but fails when I install it on a Google kubernetes cluster with the helm install.

@nmanovic nmanovic added the bug Something isn't working label Sep 30, 2021
@shortcipher3
Copy link
Contributor

I've also been looking at this, but with an external postgres sql database. I installed with an override file:

helm upgrade cvat --install . -f values.yaml -f values.override.yaml --namespace cvat-namespace

Containing enabled: false, user: postgres, password: mypassword, host=my_ip.

After install I can't login or create a super user using:

kubectl exec -it --namespace $HELM_RELEASE_NAMESPACE $BACKEND_POD_NAME -c cvat-backend-app-container -- python manage.py createsuperuser

I get errors around the database usage.

To understand I logged in and took a look around:

$ kubectl exec -it --namespace $HELM_RELEASE_NAMESPACE $BACKEND_POD_NAME -c cvat-backend-app-container -- bash
$ env | grep -i postgres
CVAT_POSTGRES_DBNAME=cvat
CVAT_POSTGRESQL_SERVICE_HOST=10.3.241.133
CVAT_POSTGRESQL_PORT_5432_TCP_PROTO=tcp
CVAT_POSTGRES_PASSWORD=cvat_postgresql
CVAT_POSTGRESQL_PORT_5432_TCP_PORT=5432
CVAT_POSTGRESQL_SERVICE_PORT=5432
CVAT_POSTGRES_USER=cvat
CVAT_POSTGRESQL_PORT=tcp://10.3.241.133:5432
CVAT_POSTGRES_HOST=cvat-postgresql
CVAT_POSTGRESQL_SERVICE_PORT_TCP_POSTGRESQL=5432
CVAT_POSTGRESQL_PORT_5432_TCP_ADDR=10.3.241.133
CVAT_POSTGRESQL_PORT_5432_TCP=tcp://10.3.241.133:5432

The password is not the one I set, ip address isn't the one I set, etc. So I changed the environment variables from the terminal and was able to successfully create a user:

$ export CVAT_POSTGRES_USER=postgres
$ export CVAT_POSTGRES_HOST=my_ip
$ export CVAT_POSTGRESQL_SERVICE_HOST=my_ip
$ export CVAT_POSTGRESQL_PORT=tcp://my_ip5432
$ export CVAT_POSTGRESQL_PORT_5432_TCP_ADDR=my_ip
$ export CVAT_POSTGRESQL_PORT_5432_TCP=tcp://my_ip:5432
$ export CVAT_POSTGRES_PASSWORD=mypassword
$ python3 manage.py createsuperuser

I have been tinkering with the values.yml and values.override.yml files and haven't been able to see any change in my environment variables.

@shortcipher3
Copy link
Contributor

I found a bug related to using an external database and made a fix here: #3779

I am able to run createsuperuser successfully, however I'm still not able to log in with the account that I have created.

I checked the frontend app and noticed that it also has some environment variables related to the postgres database that are not using my settings in the values.override.yaml. I suspect that is the cause of my login problem.

$ kubectl exec -it --namespace $HELM_RELEASE_NAMESPACE $FRONTEND_POD_NAME -c cvat-frontend-app-container -- sh
# env | grep -i postgres

@TianbaoLi
Copy link

I was experiencing the same issue last week. Similarly, I deployed CVAT on Kubernetes cluster with my modified deployment files. The main reason of this issue is about request routing on k8s cluster. Ingress on k8s is more complicated than the original docker-compose network.
From your logs in deployment.apps/cvat-frontend, you can see that a lot of request are routed to frontend. However:

  • only these should be routed to frontend pod: /cvat-ui.f9340a4e583f3306fd17.min.js, /favicon.ico
  • these should be routed to backend pod: /api/v1/auth/, /api/v1/users/, /auth/login, ...

I used ingress to configure routing. I see ingress is mentioned in helm chart but routing rules are not configured correctly.
Hope these can help you debug. Deploying CVAT on k8s without official support is painful.

@bhlarson
Copy link
Author

From the comments of @shortcipher3 and @TianbaoLi, I was curios if these errors were due to sub-packages that have changed from release v1.4.0 to v1.6.0. Performing the helm install with CVAT 1.4.0 resulted in the same behavior and errors but a bit different log information:
blarson@ipc:/data/git/cvatkc logs deployment.apps/cvat-frontend -n cv
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2021/10/11 18:44:19 [warn] 1#1: duplicate extension "wasm", content type: "application/wasm", previous content type: "application/wasm" in /etc/nginx/mime.types:98
nginx: [warn] duplicate extension "wasm", content type: "application/wasm", previous content type: "application/wasm" in /etc/nginx/mime.types:98
2021/10/11 18:44:19 [notice] 1#1: using the "epoll" event method
2021/10/11 18:44:19 [notice] 1#1: nginx/1.21.3
2021/10/11 18:44:19 [notice] 1#1: built by gcc 10.3.1 20210424 (Alpine 10.3.1_git20210424)
2021/10/11 18:44:19 [notice] 1#1: OS: Linux 5.10.0-1044-oem
2021/10/11 18:44:19 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 65536:65536
2021/10/11 18:44:19 [notice] 1#1: start worker processes
2021/10/11 18:44:19 [notice] 1#1: start worker process 32
2021/10/11 18:44:19 [notice] 1#1: start worker process 33
2021/10/11 18:44:19 [notice] 1#1: start worker process 34
2021/10/11 18:44:19 [notice] 1#1: start worker process 35
2021/10/11 18:44:19 [notice] 1#1: start worker process 36
2021/10/11 18:44:19 [notice] 1#1: start worker process 37
2021/10/11 18:44:19 [notice] 1#1: start worker process 38
2021/10/11 18:44:19 [notice] 1#1: start worker process 39
127.0.0.1 - - [11/Oct/2021:18:45:25 +0000] "GET /cvat-ui.f9340a4e583f3306fd17.min.js.map HTTP/1.1" 200 19371636 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36" "-"
127.0.0.1 - - [11/Oct/2021:18:45:26 +0000] "OPTIONS /api/v1/auth/password/change HTTP/1.1" 405 559 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36" "-"
127.0.0.1 - - [11/Oct/2021:18:45:26 +0000] "OPTIONS /api/v1/auth/password/reset HTTP/1.1" 405 559 "http://localhost:8880/auth/login" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.81 Safari/537.36" "-"
blarson@ipc:/data/git/cvat$ kc logs deployment.apps/cvat-backend -n cv
2021-10-11 18:44:26,024 INFO RPC interface 'supervisor' initialized
2021-10-11 18:44:26,024 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2021-10-11 18:44:26,024 INFO supervisord started with pid 1
2021-10-11 18:44:27,028 INFO spawned: 'ssh-agent' with pid 8
2021-10-11 18:44:27,030 INFO spawned: 'clamav_update' with pid 9
2021-10-11 18:44:27,033 INFO spawned: 'git_status_updater' with pid 10
2021-10-11 18:44:27,036 INFO spawned: 'rqscheduler' with pid 11
2021-10-11 18:44:27,038 INFO spawned: 'rqworker_default_0' with pid 12
2021-10-11 18:44:27,046 INFO spawned: 'rqworker_default_1' with pid 13
2021-10-11 18:44:27,048 INFO spawned: 'rqworker_low' with pid 20
2021-10-11 18:44:27,050 INFO spawned: 'runserver' with pid 21
2021-10-11 18:44:27,050 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140527361225872 for <Subprocess at 140527361225200 with name clamav_update in state STARTING> (stdout)>
2021-10-11 18:44:27,050 DEBG fd 14 closed, stopped monitoring <POutputDispatcher at 140527360983104 for <Subprocess at 140527361225200 with name clamav_update in state STARTING> (stderr)>
2021-10-11 18:44:27,050 INFO exited: clamav_update (exit status 0; not expected)
2021-10-11 18:44:27,050 DEBG received SIGCHLD indicating a child quit
2021-10-11 18:44:27,051 DEBG 'rqscheduler' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout

2021-10-11 18:44:27,051 DEBG 'ssh-agent' stdout output:
SSH_AUTH_SOCK=/tmp/ssh-agent.sock; export SSH_AUTH_SOCK;
echo Agent pid 8;

2021-10-11 18:44:27,051 DEBG 'ssh-agent' stderr output:
debug2: fd 3 setting O_NONBLOCK

2021-10-11 18:44:27,055 DEBG 'rqworker_default_0' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout

2021-10-11 18:44:27,058 DEBG 'rqworker_low' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout

2021-10-11 18:44:27,062 DEBG 'git_status_updater' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout

2021-10-11 18:44:27,063 DEBG 'rqworker_default_1' stderr output:
wait-for-it.sh: waiting for cvat-redis-master:6379 without a timeout

2021-10-11 18:44:27,064 DEBG 'runserver' stderr output:
wait-for-it.sh: waiting for cvat-postgresql:5432 without a timeout

2021-10-11 18:44:28,065 INFO success: ssh-agent entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-10-11 18:44:28,068 INFO spawned: 'clamav_update' with pid 47
2021-10-11 18:44:28,068 INFO success: git_status_updater entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-10-11 18:44:28,068 INFO success: rqscheduler entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-10-11 18:44:28,068 INFO success: rqworker_default_0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-10-11 18:44:28,069 INFO success: rqworker_default_1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-10-11 18:44:28,069 INFO success: rqworker_low entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-10-11 18:44:28,069 INFO success: runserver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-10-11 18:44:28,078 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140527361277376 for <Subprocess at 140527361225200 with name clamav_update in state STARTING> (stdout)>
2021-10-11 18:44:28,078 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 140527360983104 for <Subprocess at 140527361225200 with name clamav_update in state STARTING> (stderr)>
2021-10-11 18:44:28,078 INFO exited: clamav_update (exit status 0; not expected)
2021-10-11 18:44:28,078 DEBG received SIGCHLD indicating a child quit
2021-10-11 18:44:30,082 INFO spawned: 'clamav_update' with pid 48
2021-10-11 18:44:30,092 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140527361225872 for <Subprocess at 140527361225200 with name clamav_update in state STARTING> (stdout)>
2021-10-11 18:44:30,092 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 140527360983296 for <Subprocess at 140527361225200 with name clamav_update in state STARTING> (stderr)>
2021-10-11 18:44:30,092 INFO exited: clamav_update (exit status 0; not expected)
2021-10-11 18:44:30,092 DEBG received SIGCHLD indicating a child quit
2021-10-11 18:44:33,097 INFO spawned: 'clamav_update' with pid 49
2021-10-11 18:44:33,107 DEBG fd 10 closed, stopped monitoring <POutputDispatcher at 140527361277376 for <Subprocess at 140527361225200 with name clamav_update in state STARTING> (stdout)>
2021-10-11 18:44:33,107 DEBG fd 16 closed, stopped monitoring <POutputDispatcher at 140527360983392 for <Subprocess at 140527361225200 with name clamav_update in state STARTING> (stderr)>
2021-10-11 18:44:33,107 INFO exited: clamav_update (exit status 0; not expected)
2021-10-11 18:44:33,107 DEBG received SIGCHLD indicating a child quit
2021-10-11 18:44:34,108 INFO gave up: clamav_update entered FATAL state, too many start retries too quickly
2021-10-11 18:46:38,042 DEBG 'rqscheduler' stderr output:
wait-for-it.sh: cvat-redis-master:6379 is available after 131 seconds

2021-10-11 18:46:38,042 DEBG 'rqworker_default_0' stderr output:
wait-for-it.sh: cvat-redis-master:6379 is available after 131 seconds

2021-10-11 18:46:38,044 DEBG 'runserver' stderr output:
wait-for-it.sh: cvat-postgresql:5432 is available after 131 seconds

2021-10-11 18:46:38,045 DEBG 'rqscheduler' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-10-11 18:46:38,045 DEBG 'rqworker_default_0' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-10-11 18:46:38,046 DEBG 'git_status_updater' stderr output:
wait-for-it.sh: cvat-redis-master:6379 is available after 131 seconds

2021-10-11 18:46:38,046 DEBG 'rqworker_low' stderr output:
wait-for-it.sh: cvat-redis-master:6379 is available after 131 seconds

2021-10-11 18:46:38,047 DEBG 'rqworker_default_1' stderr output:
wait-for-it.sh: cvat-redis-master:6379 is available after 131 seconds

2021-10-11 18:46:38,048 DEBG 'rqworker_default_1' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-10-11 18:46:38,049 DEBG 'rqworker_low' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-10-11 18:46:38,049 DEBG 'runserver' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-10-11 18:46:38,050 DEBG 'git_status_updater' stderr output:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

2021-10-11 18:46:38,160 DEBG 'rqscheduler' stderr output:
18:46:38 Registering birth

2021-10-11 18:46:38,170 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-10-11 18:46:38,173 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-10-11 18:46:38,379 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-10-11 18:46:38,381 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-10-11 18:46:38,383 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:j7+I/H0YBq09J+JaZ5ag2jTGKvbHRf7+tnc0oQeELjc "django@cvat-backend-6f7d85f558-z9dsz" (life: 0) (confirm: 0) (provider: none)

2021-10-11 18:46:38,383 DEBG 'rqworker_default_1' stderr output:
Identity added: /home/django/.ssh/id_rsa (django@cvat-backend-6f7d85f558-z9dsz)

2021-10-11 18:46:38,388 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-10-11 18:46:38,389 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-10-11 18:46:38,394 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:j7+I/H0YBq09J+JaZ5ag2jTGKvbHRf7+tnc0oQeELjc "django@cvat-backend-6f7d85f558-z9dsz" (life: 0) (confirm: 0) (provider: none)

2021-10-11 18:46:38,400 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-10-11 18:46:38,407 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-10-11 18:46:38,411 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-10-11 18:46:38,416 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:j7+I/H0YBq09J+JaZ5ag2jTGKvbHRf7+tnc0oQeELjc "django@cvat-backend-6f7d85f558-z9dsz" (life: 0) (confirm: 0) (provider: none)

2021-10-11 18:46:38,422 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-10-11 18:46:38,429 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-10-11 18:46:38,433 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-10-11 18:46:38,435 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:j7+I/H0YBq09J+JaZ5ag2jTGKvbHRf7+tnc0oQeELjc "django@cvat-backend-6f7d85f558-z9dsz" (life: 0) (confirm: 0) (provider: none)

2021-10-11 18:46:38,439 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-10-11 18:46:38,442 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-10-11 18:46:38,444 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-10-11 18:46:38,449 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:j7+I/H0YBq09J+JaZ5ag2jTGKvbHRf7+tnc0oQeELjc "django@cvat-backend-6f7d85f558-z9dsz" (life: 0) (confirm: 0) (provider: none)

2021-10-11 18:46:38,456 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-10-11 18:46:45,380 DEBG 'rqworker_default_1' stderr output:
System check identified some issues:

WARNINGS:
?: (urls.W005) URL namespace 'v1' isn't unique. You may not be able to reverse all URLs in this namespace
DEBUG - 2021-10-11 18:46:45,379 - worker - Registering birth of worker b962ffd8401041fa881330a884e2b25b

2021-10-11 18:46:45,382 DEBG 'rqworker_default_1' stderr output:
INFO - 2021-10-11 18:46:45,382 - worker - Worker rq:worker:b962ffd8401041fa881330a884e2b25b: started, version 1.5.1

2021-10-11 18:46:45,383 DEBG 'rqworker_default_1' stderr output:
INFO - 2021-10-11 18:46:45,383 - worker - *** Listening on default...

2021-10-11 18:46:45,383 DEBG 'rqworker_default_1' stderr output:
DEBUG - 2021-10-11 18:46:45,383 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-10-11 18:46:45,384 DEBG 'rqworker_default_1' stderr output:
DEBUG - 2021-10-11 18:46:45,384 - worker - *** Listening on default...

2021-10-11 18:46:45,385 DEBG 'rqworker_default_1' stderr output:
DEBUG - 2021-10-11 18:46:45,385 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-10-11 18:46:45,438 DEBG 'rqworker_default_0' stderr output:
System check identified some issues:

WARNINGS:
?: (urls.W005) URL namespace 'v1' isn't unique. You may not be able to reverse all URLs in this namespace
DEBUG - 2021-10-11 18:46:45,438 - worker - Registering birth of worker 7e5cd726081142b2a5a32083901cc4ad

2021-10-11 18:46:45,457 DEBG 'rqworker_default_0' stderr output:
INFO - 2021-10-11 18:46:45,457 - worker - Worker rq:worker:7e5cd726081142b2a5a32083901cc4ad: started, version 1.5.1

2021-10-11 18:46:45,458 DEBG 'rqworker_default_0' stderr output:
INFO - 2021-10-11 18:46:45,458 - worker - *** Listening on default...

2021-10-11 18:46:45,459 DEBG 'rqworker_default_0' stderr output:
DEBUG - 2021-10-11 18:46:45,459 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-10-11 18:46:45,460 DEBG 'rqworker_default_0' stderr output:
DEBUG - 2021-10-11 18:46:45,460 - worker - *** Listening on default...

2021-10-11 18:46:45,461 DEBG 'rqworker_default_0' stderr output:
DEBUG - 2021-10-11 18:46:45,460 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-10-11 18:46:45,510 DEBG 'rqworker_low' stderr output:
System check identified some issues:

WARNINGS:
?: (urls.W005) URL namespace 'v1' isn't unique. You may not be able to reverse all URLs in this namespace
DEBUG - 2021-10-11 18:46:45,509 - worker - Registering birth of worker 9a67f6e0b50f4626861f8b737781abb3

2021-10-11 18:46:45,512 DEBG 'rqworker_low' stderr output:
INFO - 2021-10-11 18:46:45,512 - worker - Worker rq:worker:9a67f6e0b50f4626861f8b737781abb3: started, version 1.5.1

2021-10-11 18:46:45,513 DEBG 'rqworker_low' stderr output:
INFO - 2021-10-11 18:46:45,513 - worker - *** Listening on low...

2021-10-11 18:46:45,514 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-10-11 18:46:45,513 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-10-11 18:46:45,515 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-10-11 18:46:45,515 - worker - *** Listening on low...

2021-10-11 18:46:45,516 DEBG 'rqworker_low' stderr output:
DEBUG - 2021-10-11 18:46:45,516 - worker - Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2021-10-11 18:46:46,137 DEBG 'runserver' stdout output:
Operations to perform:
Apply all migrations: account, admin, auth, authtoken, contenttypes, dataset_repo, engine, sessions, sites, socialaccount
Running migrations:
No migrations to apply.

2021-10-11 18:46:46,138 DEBG 'runserver' stderr output:
System check identified some issues:

WARNINGS:
?: (urls.W005) URL namespace 'v1' isn't unique. You may not be able to reverse all URLs in this namespace

2021-10-11 18:46:46,888 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-10-11 18:46:46,892 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-10-11 18:46:46,893 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:j7+I/H0YBq09J+JaZ5ag2jTGKvbHRf7+tnc0oQeELjc "django@cvat-backend-6f7d85f558-z9dsz" (life: 0) (confirm: 0) (provider: none)

2021-10-11 18:46:46,898 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-10-11 18:46:47,503 DEBG 'runserver' stdout output:

0 static files copied to '/home/django/static', 182 unmodified.

2021-10-11 18:46:47,798 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-10-11 18:46:47,800 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-10-11 18:46:47,804 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:j7+I/H0YBq09J+JaZ5ag2jTGKvbHRf7+tnc0oQeELjc "django@cvat-backend-6f7d85f558-z9dsz" (life: 0) (confirm: 0) (provider: none)

2021-10-11 18:46:47,809 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

2021-10-11 18:46:52,472 DEBG 'runserver' stderr output:
System check identified some issues:

WARNINGS:
?: (urls.W005) URL namespace 'v1' isn't unique. You may not be able to reverse all URLs in this namespace

2021-10-11 18:46:52,473 DEBG 'runserver' stdout output:
Successfully ran command.

2021-10-11 18:46:52,475 DEBG 'runserver' stdout output:
Server URL : http://localhost:8080/
Server Root : /tmp/cvat-server
Server Conf : /tmp/cvat-server/httpd.conf
Error Log File : /dev/stderr (INFO)
Request Capacity : 5 (1 process * 5 threads)
Request Timeout : 60 (seconds)
Startup Timeout : 15 (seconds)
Queue Backlog : 100 (connections)
Queue Timeout : 45 (seconds)
Server Capacity : 20 (event/worker), 20 (prefork)
Server Backlog : 500 (connections)
Locale Setting : C.UTF-8

2021-10-11 18:46:52,516 DEBG 'runserver' stderr output:
[Mon Oct 11 18:46:52.516420 2021] [mpm_event:notice] [pid 21:tid 140495742585920] AH00489: Apache/2.4.41 (Ubuntu) mod_wsgi/4.7.1 Python/3.8 configured -- resuming normal operations
[Mon Oct 11 18:46:52.516482 2021] [mpm_event:info] [pid 21:tid 140495742585920] AH00490: Server built: 2021-07-05T07:16:56
[Mon Oct 11 18:46:52.516487 2021] [core:notice] [pid 21:tid 140495742585920] AH00094: Command line: 'apache2 (mod_wsgi-express) -f /tmp/cvat-server/httpd.conf -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'
[Mon Oct 11 18:46:52.516407 2021] [wsgi:info] [pid 345:tid 140495742585920] mod_wsgi (pid=345): Starting process 'localhost:8080' with threads=5.

2021-10-11 18:46:52,517 DEBG 'runserver' stderr output:
[Mon Oct 11 18:46:52.516965 2021] [wsgi:info] [pid 345:tid 140495742585920] mod_wsgi (pid=345): Python home /opt/venv.
[Mon Oct 11 18:46:52.517030 2021] [wsgi:info] [pid 345:tid 140495742585920] mod_wsgi (pid=345): Initializing Python.

2021-10-11 18:46:52,545 DEBG 'runserver' stderr output:
[Mon Oct 11 18:46:52.545227 2021] [wsgi:info] [pid 345:tid 140495742585920] mod_wsgi (pid=345): Attach interpreter ''.

2021-10-11 18:46:52,550 DEBG 'runserver' stderr output:
[Mon Oct 11 18:46:52.550574 2021] [wsgi:info] [pid 345:tid 140495742585920] mod_wsgi (pid=345): Imported 'mod_wsgi'.

2021-10-11 18:46:52,550 DEBG 'runserver' stderr output:
[Mon Oct 11 18:46:52.550944 2021] [wsgi:info] [pid 345:tid 140495742585920] mod_wsgi (pid=345, process='localhost:8080', application=''): Loading Python script file '/tmp/cvat-server/handler.wsgi'.

2021-10-11 18:46:52,747 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK

2021-10-11 18:46:52,749 DEBG 'ssh-agent' stderr output:
debug1: process_message: socket 1 (fd=4) type 17

2021-10-11 18:46:52,755 DEBG 'ssh-agent' stderr output:
debug1: process_add_identity: add ssh-rsa SHA256:j7+I/H0YBq09J+JaZ5ag2jTGKvbHRf7+tnc0oQeELjc "django@cvat-backend-6f7d85f558-z9dsz" (life: 0) (confirm: 0) (provider: none)

2021-10-11 18:46:52,760 DEBG 'ssh-agent' stderr output:
debug2: fd 4 setting O_NONBLOCK
debug1: process_message: socket 1 (fd=4) type 11

blarson@ipc:/data/git/cvat$

@TianbaoLi
Copy link

FYI, I'm using CVAT 1.6.
Upstream CVAT version: 'release-1.6.0' 8d03ed7

@nmanovic
Copy link
Contributor

@TianbaoLi , thanks for comments and your suggestions.

@bhlarson , we are going to support helm charts officially in the nearest future. For now I will close the issue.

@nmanovic nmanovic added question Further information is requested and removed bug Something isn't working labels Nov 20, 2021
@ghost
Copy link

ghost commented Dec 16, 2021 •
edited by ghost
Loading

I was experiencing the same issue last week. Similarly, I deployed CVAT on Kubernetes cluster with my modified deployment files. The main reason of this issue is about request routing on k8s cluster. Ingress on k8s is more complicated than the original docker-compose network. From your logs in deployment.apps/cvat-frontend, you can see that a lot of request are routed to frontend. However:

  • only these should be routed to frontend pod: /cvat-ui.f9340a4e583f3306fd17.min.js, /favicon.ico
  • these should be routed to backend pod: /api/v1/auth/, /api/v1/users/, /auth/login, ...

I used ingress to configure routing. I see ingress is mentioned in helm chart but routing rules are not configured correctly. Hope these can help you debug. Deploying CVAT on k8s without official support is painful.

Thanks very much. I use this configuration in k8s and work

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: cvat-ingress
  namespace: mynamespace
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "120"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "120"
    cert-manager.io/cluster-issuer: example.issuer.name
spec:
  rules:
    - host: myhost
      http:
        paths:
          - path: /
            backend:
              serviceName: cvat-ui
              servicePort: 80
          - path: /api/.*|git/.*|tensorflow/.*|auto_annotation/.*|analytics/.*|static/.*|admin|admin/.*|documentation/.*|dextr/.*|reid/.*
            backend:
              serviceName: cvat-server
              servicePort: 8080

@zerols
Copy link

zerols commented Jul 7, 2022

if don't user ingress, only nodeport how should change routed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@azhavoro azhavoro

Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@bhlarson @shortcipher3 @TianbaoLi @zerols @nmanovic @azhavoro @Hadleigh-Schwartz