Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Detection Error occurred #4978

Closed
2 tasks done
westlinkin opened this issue Sep 20, 2022 · 2 comments
Closed
2 tasks done

Detection Error occurred #4978

westlinkin opened this issue Sep 20, 2022 · 2 comments
Labels
bug Something isn't working nuclio

Comments

@westlinkin
Copy link

My actions before raising this issue

image

Error:

image

Your Environment

  • Git hash commit (git log -1): v2.2.0
  • Docker version docker version (e.g. Docker 17.0.05): Docker version 20.10.17, build 100c701
  • Are you using Docker Swarm or Kubernetes? No
  • Operating System and version (e.g. Linux, Windows, MacOS): MacOS
  • Other diagnostic information / logs: Logs from `cvat` container 
cvat_kibana | {"type":"log","@timestamp":"2022-09-20T12:18:40Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
cvat_kibana | {"type":"log","@timestamp":"2022-09-20T12:18:40Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
cvat_server | 2022-09-20 12:18:41,332 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.332470 2022] [wsgi:error] [pid 115:tid 275233875712] ERROR:LogProcessingWorker:An error occurred while sending events: HTTPConnectionPool(host='logstash', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4074bf2610>: Failed to establish a new connection: [Errno 111] Connection refused'))
cvat_server | 
cvat_server | 2022-09-20 12:18:41,333 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.333535 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | 
cvat_server | 2022-09-20 12:18:41,334 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.334512 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 118, in _fetch_events
cvat_server | 
cvat_server | 2022-09-20 12:18:41,335 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.335275 2022] [wsgi:error] [pid 115:tid 275233875712]     self._fetch_event()
cvat_server | 
cvat_server | 2022-09-20 12:18:41,336 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.336158 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 140, in _fetch_event
cvat_server | 
cvat_server | 2022-09-20 12:18:41,337 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.336863 2022] [wsgi:error] [pid 115:tid 275233875712]     self._event = self._queue.get(block=False)
cvat_server | 
cvat_server | 2022-09-20 12:18:41,337 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.337550 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/queue.py", line 167, in get
cvat_server | 
cvat_server | 2022-09-20 12:18:41,338 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.338323 2022] [wsgi:error] [pid 115:tid 275233875712]     raise Empty
cvat_server | 
cvat_server | 2022-09-20 12:18:41,339 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.338978 2022] [wsgi:error] [pid 115:tid 275233875712] _queue.Empty
cvat_server | 
cvat_server | 2022-09-20 12:18:41,339 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.339296 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | 
cvat_server | 2022-09-20 12:18:41,340 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.339952 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | 
cvat_server | 2022-09-20 12:18:41,340 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.340394 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:41.340403 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:41.340410 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
cvat_server | [Tue Sep 20 12:18:41.340417 2022] [wsgi:error] [pid 115:tid 275233875712]     conn = connection.create_connection(
cvat_server | [Tue Sep 20 12:18:41.340424 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
cvat_server | 
cvat_server | 2022-09-20 12:18:41,341 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.340432 2022] [wsgi:error] [pid 115:tid 275233875712]     raise err
cvat_server | [Tue Sep 20 12:18:41.340976 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
cvat_server | [Tue Sep 20 12:18:41.340989 2022] [wsgi:error] [pid 115:tid 275233875712]     sock.connect(sa)
cvat_server | [Tue Sep 20 12:18:41.340995 2022] [wsgi:error] [pid 115:tid 275233875712] ConnectionRefusedError: [Errno 111] Connection refused
cvat_server | [Tue Sep 20 12:18:41.340999 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:41.341004 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:41.341009 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:41.341013 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:41.341019 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
cvat_server | [Tue Sep 20 12:18:41.341025 2022] [wsgi:error] [pid 115:tid 275233875712]     httplib_response = self._make_request(
cvat_server | [Tue Sep 20 12:18:41.341030 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
cvat_server | [Tue Sep 20 12:18:41.341036 2022] [wsgi:error] [pid 115:tid 275233875712]     conn.request(method, url, **httplib_request_kw)
cvat_server | [Tue Sep 20 12:18:41.341042 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
cvat_server | [Tue Sep 20 12:18:41.341048 2022] [wsgi:error] [pid 115:tid 275233875712]     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
cvat_server | [Tue Sep 20 12:18:41.341053 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1256, in request
cvat_server | [Tue Sep 20 12:18:41.341059 2022] [wsgi:error] [pid 115:tid 275233875712]     self._send_request(method, url, body, headers, encode_chunked)
cvat_server | [Tue Sep 20 12:18:41.341064 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
cvat_server | [Tue Sep 20 12:18:41.341070 2022] [wsgi:error] [pid 115:tid 275233875712]     self.endheaders(body, encode_chunked=encode_chunked)
cvat_server | [Tue Sep 20 12:18:41.341075 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
cvat_server | [Tue Sep 20 12:18:41.341080 2022] [wsgi:error] [pid 115:tid 275233875712]     self._send_output(message_body, encode_chunked=encode_chunked)
cvat_server | [Tue Sep 20 12:18:41.341085 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
cvat_server | [Tue Sep 20 12:18:41.341090 2022] [wsgi:error] [pid 115:tid 275233875712]     self.send(msg)
cvat_server | [Tue Sep 20 12:18:41.341095 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 951, in send
cvat_server | [Tue Sep 20 12:18:41.341100 2022] [wsgi:error] [pid 115:tid 275233875712]     self.connect()
cvat_server | [Tue Sep 20 12:18:41.341106 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
cvat_server | [Tue Sep 20 12:18:41.341111 2022] [wsgi:error] [pid 115:tid 275233875712]     conn = self._new_conn()
cvat_server | [Tue Sep 20 12:18:41.341116 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
cvat_server | [Tue Sep 20 12:18:41.341122 2022] [wsgi:error] [pid 115:tid 275233875712]     raise NewConnectionError(
cvat_server | [Tue Sep 20 12:18:41.341128 2022] [wsgi:error] [pid 115:tid 275233875712] urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x4074bf2610>: Failed to establish a new connection: [Errno 111] Connection refused
cvat_server | [Tue Sep 20 12:18:41.341134 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:41.341138 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:41.341144 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:41.341148 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:41.341154 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
cvat_server | [Tue Sep 20 12:18:41.341159 2022] [wsgi:error] [pid 115:tid 275233875712]     resp = conn.urlopen(
cvat_server | [Tue Sep 20 12:18:41.341164 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
cvat_server | [Tue Sep 20 12:18:41.341169 2022] [wsgi:error] [pid 115:tid 275233875712]     retries = retries.increment(
cvat_server | 
cvat_server | 2022-09-20 12:18:41,342 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:41.341174 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
cvat_server | [Tue Sep 20 12:18:41.341703 2022] [wsgi:error] [pid 115:tid 275233875712]     raise MaxRetryError(_pool, url, error or ResponseError(cause))
cvat_server | [Tue Sep 20 12:18:41.341712 2022] [wsgi:error] [pid 115:tid 275233875712] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='logstash', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4074bf2610>: Failed to establish a new connection: [Errno 111] Connection refused'))
cvat_server | [Tue Sep 20 12:18:41.341719 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:41.341723 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:41.341728 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:41.341733 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:41.341738 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 214, in _flush_queued_events
cvat_server | [Tue Sep 20 12:18:41.341744 2022] [wsgi:error] [pid 115:tid 275233875712]     self._send_events(events)
cvat_server | [Tue Sep 20 12:18:41.341749 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 268, in _send_events
cvat_server | [Tue Sep 20 12:18:41.341756 2022] [wsgi:error] [pid 115:tid 275233875712]     self._transport.send(events, use_logging=use_logging)
cvat_server | [Tue Sep 20 12:18:41.341762 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/transport.py", line 364, in send
cvat_server | [Tue Sep 20 12:18:41.341767 2022] [wsgi:error] [pid 115:tid 275233875712]     response = self.__session.post(
cvat_server | [Tue Sep 20 12:18:41.341772 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
cvat_server | [Tue Sep 20 12:18:41.341777 2022] [wsgi:error] [pid 115:tid 275233875712]     return self.request('POST', url, data=data, json=json, **kwargs)
cvat_server | [Tue Sep 20 12:18:41.341783 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
cvat_server | [Tue Sep 20 12:18:41.341788 2022] [wsgi:error] [pid 115:tid 275233875712]     resp = self.send(prep, **send_kwargs)
cvat_server | [Tue Sep 20 12:18:41.341793 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
cvat_server | [Tue Sep 20 12:18:41.341798 2022] [wsgi:error] [pid 115:tid 275233875712]     r = adapter.send(request, **kwargs)
cvat_server | [Tue Sep 20 12:18:41.341804 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
cvat_server | [Tue Sep 20 12:18:41.341809 2022] [wsgi:error] [pid 115:tid 275233875712]     raise ConnectionError(e, request=request)
cvat_server | [Tue Sep 20 12:18:41.341817 2022] [wsgi:error] [pid 115:tid 275233875712] requests.exceptions.ConnectionError: HTTPConnectionPool(host='logstash', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4074bf2610>: Failed to establish a new connection: [Errno 111] Connection refused'))
cvat_server | 
cvat_opa | {"client_addr":"172.18.0.8:59834","level":"info","msg":"Received request.","req_id":89,"req_method":"POST","req_path":"/v1/data/lambda/allow","time":"2022-09-20T12:18:42Z"}
cvat_opa | {"decision_id":"9fc0f06a-d642-4364-bf22-c1acba1a570e","input":{"auth":{"organization":null,"user":{"id":1,"privilege":"admin"}},"resource":null,"scope":"call:online"},"labels":{"id":"2e755e82-3ff9-47dc-a886-f62cde6ce5de","version":"0.34.2"},"level":"info","metrics":{"counter_server_query_cache_hit":1,"timer_rego_input_parse_ns":515167,"timer_rego_query_eval_ns":2074709,"timer_server_handler_ns":2816416},"msg":"Decision Log","path":"lambda/allow","requested_by":"172.18.0.8:59834","result":true,"time":"2022-09-20T12:18:42Z","timestamp":"2022-09-20T12:18:42.170518968Z","type":"openpolicyagent.org/decision_logs"}
cvat_opa | {"client_addr":"172.18.0.8:59834","level":"info","msg":"Sent response.","req_id":89,"req_method":"POST","req_path":"/v1/data/lambda/allow","resp_bytes":68,"resp_duration":4.451334,"resp_status":200,"time":"2022-09-20T12:18:42Z"}
cvat_opa | {"client_addr":"172.18.0.8:59836","level":"info","msg":"Received request.","req_id":90,"req_method":"POST","req_path":"/v1/data/tasks/allow","time":"2022-09-20T12:18:42Z"}
cvat_opa | {"decision_id":"8d8a78ee-a8d3-4003-806f-1a15751f8867","input":{"auth":{"organization":null,"user":{"id":1,"privilege":"admin"}},"resource":{"assignee":{"id":null},"id":1,"organization":{"id":null},"owner":{"id":1},"project":{"assignee":{"id":null},"organization":{"id":null},"owner":{"id":1}}},"scope":"view:data"},"labels":{"id":"2e755e82-3ff9-47dc-a886-f62cde6ce5de","version":"0.34.2"},"level":"info","metrics":{"counter_server_query_cache_hit":1,"timer_rego_input_parse_ns":532250,"timer_rego_query_eval_ns":4921625,"timer_server_handler_ns":6015667},"msg":"Decision Log","path":"tasks/allow","requested_by":"172.18.0.8:59836","result":true,"time":"2022-09-20T12:18:42Z","timestamp":"2022-09-20T12:18:42.191990301Z","type":"openpolicyagent.org/decision_logs"}
cvat_opa | {"client_addr":"172.18.0.8:59836","level":"info","msg":"Sent response.","req_id":90,"req_method":"POST","req_path":"/v1/data/tasks/allow","resp_bytes":68,"resp_duration":8.015084,"resp_status":200,"time":"2022-09-20T12:18:42Z"}
nuclio | 22.09.20 12:18:42.215 dashboard.platform.docker (D) Executing in container {"containerID": "nuclio-local-storage-reader", "execOptions": {"Command":"/bin/sh -c \"/bin/cat /etc/nuclio/store/functions/nuclio/ultralytics-yolov5.json\"","Stdout":"","Stderr":"","Env":null}}
nuclio | 22.09.20 12:18:42.216 rd.platform.docker.runner (D) Executing {"command": "docker exec  nuclio-local-storage-reader /bin/sh -c \"/bin/cat /etc/nuclio/store/functions/nuclio/ultralytics-yolov5.json\""}
cvat_server | 2022-09-20 12:18:43,059 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.059471 2022] [wsgi:error] [pid 115:tid 275437655808] [remote 172.18.0.6:40166] [2022-09-20 12:18:43,059] ERROR django.request: Service Unavailable: /api/lambda/functions/ultralytics-yolov5
cvat_server | 
cvat_server | 2022-09-20 12:18:43,060 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.060074 2022] [wsgi:error] [pid 115:tid 275437655808] [remote 172.18.0.6:40166] ERROR:django.request:Service Unavailable: /api/lambda/functions/ultralytics-yolov5
cvat_server | 
cvat_kibana | {"type":"log","@timestamp":"2022-09-20T12:18:43Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
cvat_kibana | {"type":"log","@timestamp":"2022-09-20T12:18:43Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
cvat_server | 2022-09-20 12:18:43,363 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.362867 2022] [wsgi:error] [pid 115:tid 275233875712] ERROR:LogProcessingWorker:An error occurred while sending events: HTTPConnectionPool(host='logstash', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4084f43310>: Failed to establish a new connection: [Errno 111] Connection refused'))
cvat_server | 
cvat_server | 2022-09-20 12:18:43,364 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.363990 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | 
cvat_server | 2022-09-20 12:18:43,364 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.364336 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 118, in _fetch_events
cvat_server | 
cvat_server | 2022-09-20 12:18:43,364 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.364775 2022] [wsgi:error] [pid 115:tid 275233875712]     self._fetch_event()
cvat_server | 
cvat_server | 2022-09-20 12:18:43,365 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.365126 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 140, in _fetch_event
cvat_server | 
cvat_server | 2022-09-20 12:18:43,365 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.365429 2022] [wsgi:error] [pid 115:tid 275233875712]     self._event = self._queue.get(block=False)
cvat_server | 
cvat_server | 2022-09-20 12:18:43,366 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.366148 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/queue.py", line 167, in get
cvat_server | 
cvat_server | 2022-09-20 12:18:43,366 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.366819 2022] [wsgi:error] [pid 115:tid 275233875712]     raise Empty
cvat_server | 
cvat_server | 2022-09-20 12:18:43,367 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.367455 2022] [wsgi:error] [pid 115:tid 275233875712] _queue.Empty
cvat_server | 
cvat_server | 2022-09-20 12:18:43,368 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.368047 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | 
cvat_server | 2022-09-20 12:18:43,368 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.368818 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | 
cvat_server | 2022-09-20 12:18:43,369 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.369168 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | 
cvat_server | 2022-09-20 12:18:43,369 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.369751 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | 
cvat_server | 2022-09-20 12:18:43,370 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.370375 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
cvat_server | [Tue Sep 20 12:18:43.370401 2022] [wsgi:error] [pid 115:tid 275233875712]     conn = connection.create_connection(
cvat_server | [Tue Sep 20 12:18:43.370408 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
cvat_server | [Tue Sep 20 12:18:43.370414 2022] [wsgi:error] [pid 115:tid 275233875712]     raise err
cvat_server | [Tue Sep 20 12:18:43.370420 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
cvat_server | [Tue Sep 20 12:18:43.370426 2022] [wsgi:error] [pid 115:tid 275233875712]     sock.connect(sa)
cvat_server | [Tue Sep 20 12:18:43.370431 2022] [wsgi:error] [pid 115:tid 275233875712] ConnectionRefusedError: [Errno 111] Connection refused
cvat_server | [Tue Sep 20 12:18:43.370436 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:43.370440 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:43.370446 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:43.370450 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:43.370457 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
cvat_server | [Tue Sep 20 12:18:43.370462 2022] [wsgi:error] [pid 115:tid 275233875712]     httplib_response = self._make_request(
cvat_server | [Tue Sep 20 12:18:43.370467 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
cvat_server | [Tue Sep 20 12:18:43.370473 2022] [wsgi:error] [pid 115:tid 275233875712]     conn.request(method, url, **httplib_request_kw)
cvat_server | [Tue Sep 20 12:18:43.370479 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
cvat_server | [Tue Sep 20 12:18:43.370485 2022] [wsgi:error] [pid 115:tid 275233875712]     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
cvat_server | [Tue Sep 20 12:18:43.370492 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1256, in request
cvat_server | [Tue Sep 20 12:18:43.370497 2022] [wsgi:error] [pid 115:tid 275233875712]     self._send_request(method, url, body, headers, encode_chunked)
cvat_server | [Tue Sep 20 12:18:43.370502 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
cvat_server | [Tue Sep 20 12:18:43.370508 2022] [wsgi:error] [pid 115:tid 275233875712]     self.endheaders(body, encode_chunked=encode_chunked)
cvat_server | [Tue Sep 20 12:18:43.370513 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
cvat_server | [Tue Sep 20 12:18:43.370519 2022] [wsgi:error] [pid 115:tid 275233875712]     self._send_output(message_body, encode_chunked=encode_chunked)
cvat_server | [Tue Sep 20 12:18:43.370524 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
cvat_server | [Tue Sep 20 12:18:43.370529 2022] [wsgi:error] [pid 115:tid 275233875712]     self.send(msg)
cvat_server | [Tue Sep 20 12:18:43.370534 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 951, in send
cvat_server | [Tue Sep 20 12:18:43.370539 2022] [wsgi:error] [pid 115:tid 275233875712]     self.connect()
cvat_server | [Tue Sep 20 12:18:43.370545 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
cvat_server | [Tue Sep 20 12:18:43.370550 2022] [wsgi:error] [pid 115:tid 275233875712]     conn = self._new_conn()
cvat_server | [Tue Sep 20 12:18:43.370658 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
cvat_server | [Tue Sep 20 12:18:43.370666 2022] [wsgi:error] [pid 115:tid 275233875712]     raise NewConnectionError(
cvat_server | 
cvat_server | 2022-09-20 12:18:43,372 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:43.370675 2022] [wsgi:error] [pid 115:tid 275233875712] urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x4084f43310>: Failed to establish a new connection: [Errno 111] Connection refused
cvat_server | [Tue Sep 20 12:18:43.371262 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:43.371268 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:43.371273 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:43.371277 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:43.371284 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
cvat_server | [Tue Sep 20 12:18:43.371289 2022] [wsgi:error] [pid 115:tid 275233875712]     resp = conn.urlopen(
cvat_server | [Tue Sep 20 12:18:43.371294 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
cvat_server | [Tue Sep 20 12:18:43.371300 2022] [wsgi:error] [pid 115:tid 275233875712]     retries = retries.increment(
cvat_server | [Tue Sep 20 12:18:43.371304 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
cvat_server | [Tue Sep 20 12:18:43.371310 2022] [wsgi:error] [pid 115:tid 275233875712]     raise MaxRetryError(_pool, url, error or ResponseError(cause))
cvat_server | [Tue Sep 20 12:18:43.371316 2022] [wsgi:error] [pid 115:tid 275233875712] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='logstash', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4084f43310>: Failed to establish a new connection: [Errno 111] Connection refused'))
cvat_server | [Tue Sep 20 12:18:43.371323 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:43.371328 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:43.371333 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:43.371337 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:43.371342 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 214, in _flush_queued_events
cvat_server | [Tue Sep 20 12:18:43.371348 2022] [wsgi:error] [pid 115:tid 275233875712]     self._send_events(events)
cvat_server | [Tue Sep 20 12:18:43.371353 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 268, in _send_events
cvat_server | [Tue Sep 20 12:18:43.371359 2022] [wsgi:error] [pid 115:tid 275233875712]     self._transport.send(events, use_logging=use_logging)
cvat_server | [Tue Sep 20 12:18:43.371364 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/transport.py", line 364, in send
cvat_server | [Tue Sep 20 12:18:43.371370 2022] [wsgi:error] [pid 115:tid 275233875712]     response = self.__session.post(
cvat_server | [Tue Sep 20 12:18:43.371375 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
cvat_server | [Tue Sep 20 12:18:43.371380 2022] [wsgi:error] [pid 115:tid 275233875712]     return self.request('POST', url, data=data, json=json, **kwargs)
cvat_server | [Tue Sep 20 12:18:43.371386 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
cvat_server | [Tue Sep 20 12:18:43.371391 2022] [wsgi:error] [pid 115:tid 275233875712]     resp = self.send(prep, **send_kwargs)
cvat_server | [Tue Sep 20 12:18:43.371396 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
cvat_server | [Tue Sep 20 12:18:43.371402 2022] [wsgi:error] [pid 115:tid 275233875712]     r = adapter.send(request, **kwargs)
cvat_server | [Tue Sep 20 12:18:43.371408 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
cvat_server | [Tue Sep 20 12:18:43.371414 2022] [wsgi:error] [pid 115:tid 275233875712]     raise ConnectionError(e, request=request)
cvat_server | [Tue Sep 20 12:18:43.371421 2022] [wsgi:error] [pid 115:tid 275233875712] requests.exceptions.ConnectionError: HTTPConnectionPool(host='logstash', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4084f43310>: Failed to establish a new connection: [Errno 111] Connection refused'))
cvat_server | 
nuclio | 22.09.20 12:18:42.898 rd.platform.docker.runner (D) Command executed successfully {"output": "eyJtZXRhZGF0YSI6eyJuYW1lIjoidWx0cmFseXRpY3MteW9sb3Y1IiwibmFtZXNwYWNlIjoibnVjbGlvIiwibGFiZWxzIjp7Im51Y2xpby5pby9wcm9qZWN0LW5hbWUiOiJjdmF0In0sImFubm90YXRpb25zIjp7ImZyYW1ld29yayI6InB5dG9yY2giLCJuYW1lIjoiWU9MTyB2NSIsInNwZWMiOiJbXG4gIHsgXCJpZFwiOiAwLCBcIm5hbWVcIjogXCJwZXJzb25cIiB9LFxuICB7IFwiaWRcIjogMSwgXCJuYW1lXCI6IFwiYmljeWNsZVwiIH0sXG4gIHsgXCJpZFwiOiAyLCBcIm5hbWVcIjogXCJjYXJcIiB9LFxuICB7IFwiaWRcIjogMywgXCJuYW1lXCI6IFwibW90b3JiaWtlXCIgfSxcbiAgeyBcImlkXCI6IDQsIFwibmFtZVwiOiBcImFlcm9wbGFuZVwiIH0sXG4gIHsgXCJpZFwiOiA1LCBcIm5hbWVcIjogXCJidXNcIiB9LFxuICB7IFwiaWRcIjogNiwgXCJuYW1lXCI6IFwidHJhaW5cIiB9LFxuICB7IFwiaWRcIjogNywgXCJuYW1lXCI6IFwidHJ1Y2tcIiB9LFxuICB7IFwiaWRcIjogOCwgXCJuYW1lXCI6IFwiYm9hdFwiIH0sXG4gIHsgXCJpZFwiOiA5LCBcIm5hbWVcIjogXCJ0cmFmZmljIGxpZ2h0XCIgfSxcbiAgeyBcImlkXCI6IDEwLCBcIm5hbWVcIjogXCJmaXJlIGh5ZHJhbnRcIiB9LFxuICB7IFwiaWRcIjogMTEsIFwibmFtZVwiOiBcInN0b3Agc2lnblwiIH0sXG4gIHsgXCJpZFwiOiAxMiwgXCJuYW1lXCI6IFwicGFya2luZyBtZXRlclwiIH0sXG4gIHsgXCJpZFwiOiAxMywgXCJuYW1lXCI6IFwiYmVuY2hcIiB9LFxuICB7IFwiaWRcIjogMTQsIFwibmFtZVwiOiBcImJpcmRcIiB9LFxuICB7IFwiaWRcIjogMTUsIFwibmFtZVwiOiBcImNhdFwiIH0sXG4gIHsgXCJpZFwiOiAxNiwgXCJuYW1lXCI6IFwiZG9nXCIgfSxcbiAgeyBcImlkXCI6IDE3LCBcIm5hbWVcIjogXCJob3JzZVwiIH0sXG4gIHsgXCJpZFwiOiAxOCwgXCJuYW1lXCI6IFwic2hlZXBcIiB9LFxuICB7IFwiaWRcIjogMTksIFwibmFtZVwiOiBcImNvd1wiIH0sXG4gIHsgXCJpZFwiOiAyMCwgXCJuYW1lXCI6IFwiZWxlcGhhbnRcIiB9LFxuICB7IFwiaWRcIjogMjEsIFwibmFtZVwiOiBcImJlYXJcIiB9LFxuICB7IFwiaWRcIjogMjIsIFwibmFtZVwiOiBcInplYnJhXCIgfSxcbiAgeyBcImlkXCI6IDIzLCBcIm5hbWVcIjogXCJnaXJhZmZlXCIgfSxcbiAgeyBcImlkXCI6IDI0LCBcIm5hbWVcIjogXCJiYWNrcGFja1wiIH0sXG4gIHsgXCJpZFwiOiAyNSwgXCJuYW1lXCI6IFwidW1icmVsbGFcIiB9LFxuICB7IFwiaWRcIjogMjYsIFwibmFtZVwiOiBcImhhbmRiYWdcIiB9LFxuICB7IFwiaWRcIjogMjcsIFwibmFtZVwiOiBcInRpZVwiIH0sXG4gIHsgXCJpZFwiOiAyOCwgXCJuYW1lXCI6IFwic3VpdGNhc2VcIiB9LFxuICB7IFwiaWRcIjogMjksIFwibmFtZVwiOiBcImZyaXNiZWVcIiB9LFxuICB7IFwiaWRcIjogMzAsIFwibmFtZVwiOiBcInNraXNcIiB9LFxuICB7IFwiaWRcIjogMzEsIFwibmFtZVwiOiBcInNub3dib2FyZFwiIH0sXG4gIHsgXCJpZFwiOiAzMiwgXCJuYW1lXCI6IFwic3BvcnRzIGJhbGxcIiB9LFxuICB7IFwiaWRcIjogMzMsIFwibmFtZVwiOiBcImtpdGVcIiB9LFxuICB7IFwiaWRcIjogMzQsIFwibmFtZVwiOiBcImJhc2ViYWxsIGJhdFwiIH0sXG4gIHsgXCJpZFwiOiAzNSwgXCJuYW1lXCI6IFwiYmFzZWJhbGwgZ2xvdmVcIiB9LFxuICB7IFwiaWRcIjogMzYsIFwibmFtZVwiOiBcInNrYXRlYm9hcmRcIiB9LFxuICB7IFwiaWRcIjogMzcsIFwibmFtZVwiOiBcInN1cmZib2FyZFwiIH0sXG4gIHsgXCJpZFwiOiAzOCwgXCJuYW1lXCI6IFwidGVubmlzIHJhY2tldFwiIH0sXG4gIHsgXCJpZFwiOiAzOSwgXCJuYW1lXCI6IFwiYm90dGxlXCIgfSxcbiAgeyBcImlkXCI6IDQwLCBcIm5hbWVcIjogXCJ3aW5lIGdsYXNzXCIgfSxcbiAgeyBcImlkXCI6IDQxLCBcIm5hbWVcIjogXCJjdXBcIiB9LFxuICB7IFwiaWRcIjogNDIsIFwibmFtZVwiOiBcImZvcmtcIiB9LFxuICB7IFwiaWRcIjogNDMsIFwibmFtZVwiOiBcImtuaWZlXCIgfSxcbiAgeyBcImlkXCI6IDQ0LCBcIm5hbWVcIjogXCJzcG9vblwiIH0sXG4gIHsgXCJpZFwiOiA0NSwgXCJuYW1lXCI6IFwiYm93bFwiIH0sXG4gIHsgXCJpZFwiOiA0NiwgXCJuYW1lXCI6IFwiYmFuYW5hXCIgfSxcbiAgeyBcImlkXCI6IDQ3LCBcIm5hbWVcIjogXCJhcHBsZVwiIH0sXG4gIHsgXCJpZFwiOiA0OCwgXCJuYW1lXCI6IFwic2FuZHdpY2hcIiB9LFxuICB7IFwiaWRcIjogNDksIFwibmFtZVwiOiBcIm9yYW5nZVwiIH0sXG4gIHsgXCJpZFwiOiA1MCwgXCJuYW1lXCI6IFwiYnJvY2NvbGlcIiB9LFxuICB7IFwiaWRcIjogNTEsIFwibmFtZVwiOiBcImNhcnJvdFwiIH0sXG4gIHsgXCJpZFwiOiA1MiwgXCJuYW1lXCI6IFwiaG90IGRvZ1wiIH0sXG4gIHsgXCJpZFwiOiA1MywgXCJuYW1lXCI6IFwicGl6emFcIiB9LFxuICB7IFwiaWRcIjogNTQsIFwibmFtZVwiOiBcImRvbnV0XCIgfSxcbiAgeyBcImlkXCI6IDU1LCBcIm5hbWVcIjogXCJjYWtlXCIgfSxcbiAgeyBcImlkXCI6IDU2LCBcIm5hbWVcIjogXCJjaGFpclwiIH0sXG4gIHsgXCJpZFwiOiA1NywgXCJuYW1lXCI6IFwic29mYVwiIH0sXG4gIHsgXCJpZFwiOiA1OCwgXCJuYW1lXCI6IFwicG90dGVkcGxhbnRcIiB9LFxuICB7IFwiaWRcIjogNTksIFwibmFtZVwiOiBcImJlZFwiIH0sXG4gIHsgXCJpZFwiOiA2MCwgXCJuYW1lXCI6IFwiZGluaW5ndGFibGVcIiB9LFxuICB7IFwiaWRcIjogNjEsIFwibmFtZVwiOiBcInRvaWxldFwiIH0sXG4gIHsgXCJpZFwiOiA2MiwgXCJuYW1lXCI6IFwidHZtb25pdG9yXCIgfSxcbiAgeyBcImlkXCI6IDYzLCBcIm5hbWVcIjogXCJsYXB0b3BcIiB9LFxuICB7IFwiaWRcIjogNjQsIFwibmFtZVwiOiBcIm1vdXNlXCIgfSxcbiAgeyBcImlkXCI6IDY1LCBcIm5hbWVcIjogXCJyZW1vdGVcIiB9LFxuICB7IFwiaWRcIjogNjYsIFwibmFtZVwiOiBcImtleWJvYXJkXCIgfSxcbiAgeyBcImlkXCI6IDY3LCBcIm5hbWVcIjogXCJjZWxsIHBob25lXCIgfSxcbiAgeyBcImlkXCI6IDY4LCBcIm5hbWVcIjogXCJtaWNyb3dhdmVcIiB9LFxuICB7IFwiaWRcIjogNjksIFwibmFtZVwiOiBcIm92ZW5cIiB9LFxuICB7IFwiaWRcIjogNzAsIFwibmFtZVwiOiBcInRvYXN0ZXJcIiB9LFxuICB7IFwiaWRcIjogNzEsIFwibmFtZVwiOiBcInNpbmtcIiB9LFxuICB7IFwiaWRcIjogNzIsIFwibmFtZVwiOiBcInJlZnJpZ2VyYXRvclwiIH0sXG4gIHsgXCJpZFwiOiA3MywgXCJuYW1lXCI6IFwiYm9va1wiIH0sXG4gIHsgXCJpZFwiOiA3NCwgXCJuYW1lXCI6IFwiY2xvY2tcIiB9LFxuICB7IFwiaWRcIjogNzUsIFwibmFtZVwiOiBcInZhc2VcIiB9LFxuICB7IFwiaWRcIjogNzYsIFwibmFtZVwiOiBcInNjaXNzb3JzXCIgfSxcbiAgeyBcImlkXCI6IDc3LCBcIm5hbWVcIjogXCJ0ZWRkeSBiZWFyXCIgfSxcbiAgeyBcImlkXCI6IDc4LCBcIm5hbWVcIjogXCJoYWlyIGRyaWVyXCIgfSxcbiAgeyBcImlkXCI6IDc5LCBcIm5hbWVcIjogXCJ0b290aGJydXNoXCIgfVxuXVxuIiwidHlwZSI6ImRldGVjdG9yIn19LCJzcGVjIjp7ImRlc2NyaXB0aW9uIjoiWU9MTyB2NSB2aWEgcHl0b3JjaCBodWIiLCJoYW5kbGVyIjoibWFpbjpoYW5kbGVyIiwicnVudGltZSI6InB5dGhvbjozLjYiLCJyZXNvdXJjZXMiOnsicmVxdWVzdHMiOnsiY3B1IjoiMjVtIiwibWVtb3J5IjoiMU1pIn19LCJpbWFnZSI6ImN2YXQudWx0cmFseXRpY3MteW9sb3Y1OmxhdGVzdCIsInRhcmdldENQVSI6NzUsInRyaWdnZXJzIjp7Im15SHR0cFRyaWdnZXIiOnsiY2xhc3MiOiIiLCJraW5kIjoiaHR0cCIsIm5hbWUiOiJteUh0dHBUcmlnZ2VyIiwibWF4V29ya2VycyI6Miwid29ya2VyQXZhaWxhYmlsaXR5VGltZW91dE1pbGxpc2Vjb25kcyI6MTAwMDAsImF0dHJpYnV0ZXMiOnsibWF4UmVxdWVzdEJvZHlTaXplIjozMzU1NDQzMn19fSwidm9sdW1lcyI6W3sidm9sdW1lIjp7Im5hbWUiOiJ2b2x1bWUtMSIsImhvc3RQYXRoIjp7InBhdGgiOiIvVXNlcnMvTGlua2luL1N0dWR5L2N2YXQvc2VydmVybGVzcy9jb21tb24ifX0sInZvbHVtZU1vdW50Ijp7Im5hbWUiOiJ2b2x1bWUtMSIsIm1vdW50UGF0aCI6Ii9vcHQvbnVjbGlvL2NvbW1vbiJ9fV0sImJ1aWxkIjp7ImltYWdlIjoiY3ZhdC51bHRyYWx5dGljcy15b2xvdjUiLCJiYXNlSW1hZ2UiOiJ1bHRyYWx5dGljcy95b2xvdjU6bGF0ZXN0LWNwdSIsImRpcmVjdGl2ZXMiOnsicHJlQ29weSI6W3sia2luZCI6IlVTRVIiLCJ2YWx1ZSI6InJvb3QifSx7ImtpbmQiOiJSVU4iLCJ2YWx1ZSI6ImFwdCB1cGRhdGUgXHUwMDI2XHUwMDI2IGFwdCBpbnN0YWxsIC0tbm8taW5zdGFsbC1yZWNvbW1lbmRzIC15IGxpYmdsaWIyLjAtMCJ9LHsia2luZCI6IldPUktESVIiLCJ2YWx1ZSI6Ii9vcHQvbnVjbGlvIn1dfSwiY29kZUVudHJ5VHlwZSI6ImltYWdlIiwidGltZXN0YW1wIjoxNjYzNjc0ODcxfSwicGxhdGZvcm0iOnsiYXR0cmlidXRlcyI6eyJtb3VudE1vZGUiOiJ2b2x1bWUiLCJyZXN0YXJ0UG9saWN5Ijp7Im1heGltdW1SZXRyeUNvdW50IjozLCJuYW1lIjoiYWx3YXlzIn19fSwicmVhZGluZXNzVGltZW91dFNlY29uZHMiOjEyMCwic2VjdXJpdHlDb250ZXh0Ijp7fSwiZXZlbnRUaW1lb3V0IjoiMzBzIn0sInN0YXR1cyI6eyJzdGF0ZSI6InJlYWR5IiwiaHR0cFBvcnQiOjYyMDA3LCJpbnRlcm5hbEludm9jYXRpb25VcmxzIjpbIjE3Mi4xNy4wLjQ6ODA4MCJdfX0=\n", "stderr": "", "exitCode": 0}
nuclio | 22.09.20 12:18:42.908          dashboard.server (D) Handled request {"requestID": "b8b0c9976cff/lHnQhOMvda-000019", "requestMethod": "GET", "requestPath": "/api/functions/ultralytics-yolov5", "requestHeaders": {"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"],"Connection":["close"],"User-Agent":["python-requests/2.26.0"],"X-Nuclio-Function-Namespace":["nuclio"],"X-Nuclio-Invoke-Via":["domain-name"],"X-Nuclio-Project-Name":["cvat"]}, "requestBody": "", "responseStatus": 200, "responseTime": "692.69825ms"}
cvat_server | 2022-09-20 12:18:45,399 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:45.398499 2022] [wsgi:error] [pid 115:tid 275233875712] ERROR:LogProcessingWorker:An error occurred while sending events: HTTPConnectionPool(host='logstash', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4074de0be0>: Failed to establish a new connection: [Errno 111] Connection refused'))
cvat_server | 
cvat_server | 2022-09-20 12:18:45,401 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:45.400436 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | 
cvat_server | 2022-09-20 12:18:45,402 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:45.402347 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 118, in _fetch_events
cvat_server | 
cvat_server | 2022-09-20 12:18:45,403 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:45.403353 2022] [wsgi:error] [pid 115:tid 275233875712]     self._fetch_event()
cvat_server | 
cvat_server | 2022-09-20 12:18:45,404 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:45.404333 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 140, in _fetch_event
cvat_server | 
cvat_server | 2022-09-20 12:18:45,405 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:45.405298 2022] [wsgi:error] [pid 115:tid 275233875712]     self._event = self._queue.get(block=False)
cvat_server | 
cvat_server | 2022-09-20 12:18:45,405 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:45.405852 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/queue.py", line 167, in get
cvat_server | 
cvat_server | 2022-09-20 12:18:45,406 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:45.406495 2022] [wsgi:error] [pid 115:tid 275233875712]     raise Empty
cvat_server | 
cvat_server | 2022-09-20 12:18:45,407 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:45.407228 2022] [wsgi:error] [pid 115:tid 275233875712] _queue.Empty
cvat_server | 
cvat_server | 2022-09-20 12:18:45,408 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:45.407935 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | 
cvat_server | 2022-09-20 12:18:45,409 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:45.408730 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:45.408748 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:45.408753 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:45.408759 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
cvat_server | [Tue Sep 20 12:18:45.408765 2022] [wsgi:error] [pid 115:tid 275233875712]     conn = connection.create_connection(
cvat_server | [Tue Sep 20 12:18:45.408771 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
cvat_server | [Tue Sep 20 12:18:45.408776 2022] [wsgi:error] [pid 115:tid 275233875712]     raise err
cvat_server | [Tue Sep 20 12:18:45.408782 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
cvat_server | [Tue Sep 20 12:18:45.408787 2022] [wsgi:error] [pid 115:tid 275233875712]     sock.connect(sa)
cvat_server | [Tue Sep 20 12:18:45.408792 2022] [wsgi:error] [pid 115:tid 275233875712] ConnectionRefusedError: [Errno 111] Connection refused
cvat_server | [Tue Sep 20 12:18:45.408796 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:45.408800 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:45.408805 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:45.408809 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:45.408814 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
cvat_server | [Tue Sep 20 12:18:45.408820 2022] [wsgi:error] [pid 115:tid 275233875712]     httplib_response = self._make_request(
cvat_server | [Tue Sep 20 12:18:45.408825 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
cvat_server | [Tue Sep 20 12:18:45.408831 2022] [wsgi:error] [pid 115:tid 275233875712]     conn.request(method, url, **httplib_request_kw)
cvat_server | [Tue Sep 20 12:18:45.408837 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
cvat_server | [Tue Sep 20 12:18:45.408842 2022] [wsgi:error] [pid 115:tid 275233875712]     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
cvat_server | [Tue Sep 20 12:18:45.408848 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1256, in request
cvat_server | [Tue Sep 20 12:18:45.408853 2022] [wsgi:error] [pid 115:tid 275233875712]     self._send_request(method, url, body, headers, encode_chunked)
cvat_server | [Tue Sep 20 12:18:45.408858 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
cvat_server | [Tue Sep 20 12:18:45.408863 2022] [wsgi:error] [pid 115:tid 275233875712]     self.endheaders(body, encode_chunked=encode_chunked)
cvat_server | [Tue Sep 20 12:18:45.408868 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
cvat_server | [Tue Sep 20 12:18:45.408873 2022] [wsgi:error] [pid 115:tid 275233875712]     self._send_output(message_body, encode_chunked=encode_chunked)
cvat_server | [Tue Sep 20 12:18:45.408878 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
cvat_server | [Tue Sep 20 12:18:45.408883 2022] [wsgi:error] [pid 115:tid 275233875712]     self.send(msg)
cvat_server | [Tue Sep 20 12:18:45.408888 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 951, in send
cvat_server | [Tue Sep 20 12:18:45.408892 2022] [wsgi:error] [pid 115:tid 275233875712]     self.connect()
cvat_server | [Tue Sep 20 12:18:45.408898 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
cvat_server | [Tue Sep 20 12:18:45.408903 2022] [wsgi:error] [pid 115:tid 275233875712]     conn = self._new_conn()
cvat_server | [Tue Sep 20 12:18:45.408910 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
cvat_server | [Tue Sep 20 12:18:45.408915 2022] [wsgi:error] [pid 115:tid 275233875712]     raise NewConnectionError(
cvat_server | [Tue Sep 20 12:18:45.408921 2022] [wsgi:error] [pid 115:tid 275233875712] urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x4074de0be0>: Failed to establish a new connection: [Errno 111] Connection refused
cvat_server | [Tue Sep 20 12:18:45.408927 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:45.408931 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:45.408935 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:45.408939 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:45.408946 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
cvat_server | [Tue Sep 20 12:18:45.408951 2022] [wsgi:error] [pid 115:tid 275233875712]     resp = conn.urlopen(
cvat_server | [Tue Sep 20 12:18:45.408955 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
cvat_server | [Tue Sep 20 12:18:45.408960 2022] [wsgi:error] [pid 115:tid 275233875712]     retries = retries.increment(
cvat_server | [Tue Sep 20 12:18:45.408965 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
cvat_server | [Tue Sep 20 12:18:45.408970 2022] [wsgi:error] [pid 115:tid 275233875712]     raise MaxRetryError(_pool, url, error or ResponseError(cause))
cvat_server | [Tue Sep 20 12:18:45.408976 2022] [wsgi:error] [pid 115:tid 275233875712] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='logstash', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4074de0be0>: Failed to establish a new connection: [Errno 111] Connection refused'))
cvat_server | [Tue Sep 20 12:18:45.408983 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:45.408987 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:45.408991 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:45.408995 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:45.409001 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 214, in _flush_queued_events
cvat_server | [Tue Sep 20 12:18:45.409006 2022] [wsgi:error] [pid 115:tid 275233875712]     self._send_events(events)
cvat_server | [Tue Sep 20 12:18:45.409010 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 268, in _send_events
cvat_server | [Tue Sep 20 12:18:45.409016 2022] [wsgi:error] [pid 115:tid 275233875712]     self._transport.send(events, use_logging=use_logging)
cvat_server | [Tue Sep 20 12:18:45.409021 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/transport.py", line 364, in send
cvat_server | [Tue Sep 20 12:18:45.409026 2022] [wsgi:error] [pid 115:tid 275233875712]     response = self.__session.post(
cvat_server | [Tue Sep 20 12:18:45.409031 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
cvat_server | [Tue Sep 20 12:18:45.409036 2022] [wsgi:error] [pid 115:tid 275233875712]     return self.request('POST', url, data=data, json=json, **kwargs)
cvat_server | [Tue Sep 20 12:18:45.409041 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
cvat_server | [Tue Sep 20 12:18:45.409046 2022] [wsgi:error] [pid 115:tid 275233875712]     resp = self.send(prep, **send_kwargs)
cvat_server | [Tue Sep 20 12:18:45.409051 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
cvat_server | [Tue Sep 20 12:18:45.409056 2022] [wsgi:error] [pid 115:tid 275233875712]     r = adapter.send(request, **kwargs)
cvat_server | [Tue Sep 20 12:18:45.409061 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
cvat_server | [Tue Sep 20 12:18:45.409066 2022] [wsgi:error] [pid 115:tid 275233875712]     raise ConnectionError(e, request=request)
cvat_server | [Tue Sep 20 12:18:45.409072 2022] [wsgi:error] [pid 115:tid 275233875712] requests.exceptions.ConnectionError: HTTPConnectionPool(host='logstash', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4074de0be0>: Failed to establish a new connection: [Errno 111] Connection refused'))
cvat_server | 
cvat_kibana | {"type":"log","@timestamp":"2022-09-20T12:18:45Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
cvat_kibana | {"type":"log","@timestamp":"2022-09-20T12:18:45Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
cvat_server | 2022-09-20 12:18:47,434 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:47.434476 2022] [wsgi:error] [pid 115:tid 275233875712] ERROR:LogProcessingWorker:An error occurred while sending events: HTTPConnectionPool(host='logstash', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4074c6e130>: Failed to establish a new connection: [Errno 111] Connection refused'))
cvat_server | [Tue Sep 20 12:18:47.434537 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:47.434544 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 118, in _fetch_events
cvat_server | [Tue Sep 20 12:18:47.434550 2022] [wsgi:error] [pid 115:tid 275233875712]     self._fetch_event()
cvat_server | [Tue Sep 20 12:18:47.434602 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 140, in _fetch_event
cvat_server | [Tue Sep 20 12:18:47.434610 2022] [wsgi:error] [pid 115:tid 275233875712]     self._event = self._queue.get(block=False)
cvat_server | [Tue Sep 20 12:18:47.434615 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/queue.py", line 167, in get
cvat_server | [Tue Sep 20 12:18:47.434620 2022] [wsgi:error] [pid 115:tid 275233875712]     raise Empty
cvat_server | [Tue Sep 20 12:18:47.434625 2022] [wsgi:error] [pid 115:tid 275233875712] _queue.Empty
cvat_server | [Tue Sep 20 12:18:47.434629 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:47.434633 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:47.434638 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:47.434642 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:47.434647 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
cvat_server | [Tue Sep 20 12:18:47.434653 2022] [wsgi:error] [pid 115:tid 275233875712]     conn = connection.create_connection(
cvat_server | [Tue Sep 20 12:18:47.434658 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 95, in create_connection
cvat_server | [Tue Sep 20 12:18:47.434664 2022] [wsgi:error] [pid 115:tid 275233875712]     raise err
cvat_server | [Tue Sep 20 12:18:47.434668 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
cvat_server | [Tue Sep 20 12:18:47.434674 2022] [wsgi:error] [pid 115:tid 275233875712]     sock.connect(sa)
cvat_server | [Tue Sep 20 12:18:47.434678 2022] [wsgi:error] [pid 115:tid 275233875712] ConnectionRefusedError: [Errno 111] Connection refused
cvat_server | [Tue Sep 20 12:18:47.434683 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:47.434687 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:47.434692 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:47.434696 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:47.434701 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
cvat_server | [Tue Sep 20 12:18:47.434706 2022] [wsgi:error] [pid 115:tid 275233875712]     httplib_response = self._make_request(
cvat_server | [Tue Sep 20 12:18:47.434711 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 398, in _make_request
cvat_server | [Tue Sep 20 12:18:47.434716 2022] [wsgi:error] [pid 115:tid 275233875712]     conn.request(method, url, **httplib_request_kw)
cvat_server | [Tue Sep 20 12:18:47.434721 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
cvat_server | [Tue Sep 20 12:18:47.434727 2022] [wsgi:error] [pid 115:tid 275233875712]     super(HTTPConnection, self).request(method, url, body=body, headers=headers)
cvat_server | [Tue Sep 20 12:18:47.434732 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1256, in request
cvat_server | [Tue Sep 20 12:18:47.434738 2022] [wsgi:error] [pid 115:tid 275233875712]     self._send_request(method, url, body, headers, encode_chunked)
cvat_server | [Tue Sep 20 12:18:47.434743 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1302, in _send_request
cvat_server | 
cvat_server | 2022-09-20 12:18:47,435 DEBG 'runserver' stderr output:
cvat_server | [Tue Sep 20 12:18:47.434748 2022] [wsgi:error] [pid 115:tid 275233875712]     self.endheaders(body, encode_chunked=encode_chunked)
cvat_server | [Tue Sep 20 12:18:47.434754 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1251, in endheaders
cvat_server | [Tue Sep 20 12:18:47.434759 2022] [wsgi:error] [pid 115:tid 275233875712]     self._send_output(message_body, encode_chunked=encode_chunked)
cvat_server | [Tue Sep 20 12:18:47.434764 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 1011, in _send_output
cvat_server | [Tue Sep 20 12:18:47.434769 2022] [wsgi:error] [pid 115:tid 275233875712]     self.send(msg)
cvat_server | [Tue Sep 20 12:18:47.434774 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/usr/lib/python3.8/http/client.py", line 951, in send
cvat_server | [Tue Sep 20 12:18:47.434778 2022] [wsgi:error] [pid 115:tid 275233875712]     self.connect()
cvat_server | [Tue Sep 20 12:18:47.434783 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
cvat_server | [Tue Sep 20 12:18:47.434788 2022] [wsgi:error] [pid 115:tid 275233875712]     conn = self._new_conn()
cvat_server | [Tue Sep 20 12:18:47.434793 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
cvat_server | [Tue Sep 20 12:18:47.434798 2022] [wsgi:error] [pid 115:tid 275233875712]     raise NewConnectionError(
cvat_server | [Tue Sep 20 12:18:47.434803 2022] [wsgi:error] [pid 115:tid 275233875712] urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x4074c6e130>: Failed to establish a new connection: [Errno 111] Connection refused
cvat_server | [Tue Sep 20 12:18:47.434809 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:47.434813 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:47.434818 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:47.434822 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:47.434827 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
cvat_server | [Tue Sep 20 12:18:47.434832 2022] [wsgi:error] [pid 115:tid 275233875712]     resp = conn.urlopen(
cvat_server | [Tue Sep 20 12:18:47.434837 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 787, in urlopen
cvat_server | [Tue Sep 20 12:18:47.434842 2022] [wsgi:error] [pid 115:tid 275233875712]     retries = retries.increment(
cvat_server | [Tue Sep 20 12:18:47.434847 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
cvat_server | [Tue Sep 20 12:18:47.434852 2022] [wsgi:error] [pid 115:tid 275233875712]     raise MaxRetryError(_pool, url, error or ResponseError(cause))
cvat_server | [Tue Sep 20 12:18:47.434861 2022] [wsgi:error] [pid 115:tid 275233875712] urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='logstash', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4074c6e130>: Failed to establish a new connection: [Errno 111] Connection refused'))
cvat_server | [Tue Sep 20 12:18:47.434867 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:47.434872 2022] [wsgi:error] [pid 115:tid 275233875712] During handling of the above exception, another exception occurred:
cvat_server | [Tue Sep 20 12:18:47.434876 2022] [wsgi:error] [pid 115:tid 275233875712] 
cvat_server | [Tue Sep 20 12:18:47.434880 2022] [wsgi:error] [pid 115:tid 275233875712] Traceback (most recent call last):
cvat_server | [Tue Sep 20 12:18:47.434886 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 214, in _flush_queued_events
cvat_server | [Tue Sep 20 12:18:47.434891 2022] [wsgi:error] [pid 115:tid 275233875712]     self._send_events(events)
cvat_server | [Tue Sep 20 12:18:47.434896 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/worker.py", line 268, in _send_events
cvat_server | [Tue Sep 20 12:18:47.434901 2022] [wsgi:error] [pid 115:tid 275233875712]     self._transport.send(events, use_logging=use_logging)
cvat_server | [Tue Sep 20 12:18:47.434906 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/logstash_async/transport.py", line 364, in send
cvat_server | [Tue Sep 20 12:18:47.434911 2022] [wsgi:error] [pid 115:tid 275233875712]     response = self.__session.post(
cvat_server | [Tue Sep 20 12:18:47.434916 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
cvat_server | [Tue Sep 20 12:18:47.434921 2022] [wsgi:error] [pid 115:tid 275233875712]     return self.request('POST', url, data=data, json=json, **kwargs)
cvat_server | [Tue Sep 20 12:18:47.434926 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
cvat_server | [Tue Sep 20 12:18:47.434931 2022] [wsgi:error] [pid 115:tid 275233875712]     resp = self.send(prep, **send_kwargs)
cvat_server | [Tue Sep 20 12:18:47.434936 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
cvat_server | [Tue Sep 20 12:18:47.434941 2022] [wsgi:error] [pid 115:tid 275233875712]     r = adapter.send(request, **kwargs)
cvat_server | [Tue Sep 20 12:18:47.434946 2022] [wsgi:error] [pid 115:tid 275233875712]   File "/opt/venv/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
cvat_server | [Tue Sep 20 12:18:47.434951 2022] [wsgi:error] [pid 115:tid 275233875712]     raise ConnectionError(e, request=request)
cvat_server | [Tue Sep 20 12:18:47.434957 2022] [wsgi:error] [pid 115:tid 275233875712] requests.exceptions.ConnectionError: HTTPConnectionPool(host='logstash', port=8080): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x4074c6e130>: Failed to establish a new connection: [Errno 111] Connection refused'))
cvat_server | 
cvat_kibana | {"type":"log","@timestamp":"2022-09-20T12:18:48Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}
cvat_kibana | {"type":"log","@timestamp":"2022-09-20T12:18:48Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"No living connections"}
@sizov-kirill sizov-kirill added the bug Something isn't working label Oct 1, 2022
@yasakova-anastasia
Copy link
Contributor

I was unable to reproduce the issue. Maybe it's been fixed by #4907. I will close the issue. Please reopen the issue if you have it again.

@SShimmyo
Copy link

I have same issues. I couldn't use yolov5 for semi-automatic annotation. I would be great if you give me any solutions.
CVAT was deployed on AWS EC2 Ubuntu 20.04

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working nuclio
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@westlinkin @bsekachev @sizov-kirill @yasakova-anastasia @SShimmyo