Cannot deploy CVAT on AWS EC2 instance #1287

rasheeddo · 2020-03-18T13:32:14Z

I created an ubuntu machine on AWS EC2 and installed according to the installation guide, everything seems to be installed properly without any error.
On my AWS EC2 Security Groups, I set Inbound Rules as following image

My docker-compose.override.yml is here

version: "2.3"

services:
  cvat_proxy:
    environment:
      CVAT_HOST: 13.230.9.169
    ports:
      - "7070:80"

13.230.9.169 is my ubuntu instance IPv4 Public IP on AWS.

I did
docker-compose down
docker-compose -f docker-compose.yml -f docker-compose.override.yml build
docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d
Then I go to my chrome browser on my local machine to access that ubuntu AWS instance, and I got this

even I tried both 13.230.9.169:7070 and 13.230.9.169:8080, but both end up at the same page.
here is my docker-compose config

services:
  cvat:
    build:
      args:
        AUTO_SEGMENTATION: "no"
        DJANGO_CONFIGURATION: production
        OPENVINO_TOOLKIT: "no"
        TF_ANNOTATION: "no"
        TZ: Etc/UTC
        USER: django
        http_proxy: ''
        https_proxy: ''
        no_proxy: ''
        socks_proxy: ''
      context: /home/ubuntu/cvat
    container_name: cvat
    depends_on:
      cvat_db:
        condition: service_started
      cvat_redis:
        condition: service_started
    environment:
      ALLOWED_HOSTS: '*'
      DJANGO_MODWSGI_EXTRA_ARGS: ''
    image: cvat
    restart: always
    volumes:
    - cvat_data:/home/django/data:rw
    - cvat_keys:/home/django/keys:rw
    - cvat_logs:/home/django/logs:rw
    - cvat_models:/home/django/models:rw
  cvat_db:
    container_name: cvat_db
    environment:
      POSTGRES_DB: cvat
      POSTGRES_HOST_AUTH_METHOD: trust
      POSTGRES_USER: root
    image: postgres:10-alpine
    networks:
      default:
        aliases:
        - db
    restart: always
    volumes:
    - cvat_db:/var/lib/postgresql/data:rw
  cvat_proxy:
    command: /bin/sh -c "envsubst '$$CVAT_HOST' < /etc/nginx/conf.d/cvat.conf.template
      > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"
    container_name: cvat_proxy
    depends_on:
      cvat:
        condition: service_started
      cvat_ui:
        condition: service_started
    environment:
      CVAT_HOST: 13.230.9.169
    image: nginx:stable-alpine
    ports:
    - 8080:80/tcp
    - 7070:80/tcp
    restart: always
    volumes:
    - /home/ubuntu/cvat/cvat_proxy/conf.d/cvat.conf.template:/etc/nginx/conf.d/cvat.conf.template:ro
    - /home/ubuntu/cvat/cvat_proxy/nginx.conf:/etc/nginx/nginx.conf:ro
  cvat_redis:
    container_name: cvat_redis
    image: redis:4.0-alpine
    networks:
      default:
        aliases:
        - redis
    restart: always
  cvat_ui:
    build:
      args:
        http_proxy: ''
        https_proxy: ''
        no_proxy: ''
        socks_proxy: ''
      context: /home/ubuntu/cvat
      dockerfile: Dockerfile.ui
    container_name: cvat_ui
    depends_on:
      cvat:
        condition: service_started
    networks:
      default:
        aliases:
        - ui
    restart: always
version: '2.3'
volumes:
  cvat_data: {}
  cvat_db: {}
  cvat_keys: {}
  cvat_logs: {}
  cvat_models: {}

and here is my docker ps

I tried the following solutions
#930 (comment)
#1249 (comment)
#1158 (comment)

but none of them work ;( , still I ended up at Welcome to nginx! page, please give me some suggestions. Thank you in advance!

Note: CVAT on my local machine works fine, with localhost:8080 on browser.

The text was updated successfully, but these errors were encountered:

azhavoro · 2020-03-18T15:44:56Z

Hi, thanks for the report, could you please attach the output from the following command
docker exec -it cvat_proxy -T and docker exec -it cvat_ui -T?

azhavoro · 2020-03-18T16:07:12Z

One more question: is the admin page http://13.230.9.169:8080/admin reachable?

rasheeddo · 2020-03-18T17:02:33Z

@azhavoro Hi, thanks for your reply. I stopped that ubuntu instance but now it starts again, the IP has changed to 3.112.247.130, I edited CVAT_HOST on docker-compose.override.yml to the new IP and ran the same process as above

docker-compose down
docker-compose -f docker-compose.yml -f docker-compose.override.yml build
docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d

Then your command docker exec -it cvat_proxy -T gave me this
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"-T\": executable file not found in $PATH": unknown
and the command docker exec -it cvat_ui -T gave me this
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"-T\": executable file not found in $PATH": unknown
I entered the 3.112.247.130:8080/admin, seem like it's reachable, it shows me Django administration login page as this picture

I am quite new to docker and django, please let me know what's happen or what should I do, I am willing to learn. Thank you!

rasheeddo · 2020-03-18T17:08:26Z

I tried login with the same username/password I did on CVAT on my local machine, it links me to this administration page, I tried click on VIEW SITE on top right, but it brought me to the Welcome to nginx! page again.

azhavoro · 2020-03-18T18:26:16Z

oh, my bad , right command is docker exec -it cvat_ui nginx -T and docker exec -it cvat_proxy nginx -T

rasheeddo · 2020-03-18T19:06:17Z

docker exec -it cvat_ui nginx -T gave me this

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/conf.d/default.conf:
server {
    root /usr/share/nginx/html;
    # Any route that doesn't have a file extension (e.g. /devices)
    location / {
        try_files $uri $uri/ /index.html;
        add_header Access-Control-Allow-Origin "*";
    }
}

note that I remember I added add_header Access-Control-Allow-Origin "*"; on react_nginx.conf from this #1011 (comment)
and docker exec -it cvat_proxy nginx -T gave me this

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
worker_processes 2;


events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;
    sendfile      on;
    keepalive_timeout  65;

   include /etc/nginx/conf.d/*.conf;
   client_max_body_size 0;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/conf.d/default.conf:
server {
    listen       80;
    server_name  _ default;
    return       404;
}

server {
    listen       80;
    server_name  3.112.247.130;

    location ~* /api/.*|git/.*|tensorflow/.*|auto_annotation/.*|analytics/.*|static/.*|admin|admin/.*|documentation/.*|dextr/.*|reid/.*  {
        proxy_pass              http://cvat:8080;
        proxy_pass_header       X-CSRFToken;
        proxy_set_header        Host $http_host;
        proxy_pass_header       Set-Cookie;
    }

    location / {
        # workaround for match location by arguments
        error_page 418 = @annotation_ui;

        if ( $query_string ~ "^id=\d+.*" ) { return 418; }

        proxy_pass              http://cvat_ui;
        proxy_pass_header       X-CSRFToken;
        proxy_set_header        Host $http_host;
        proxy_pass_header       Set-Cookie;
    }

    # old annotation ui, will be removed in the future.
    location @annotation_ui {
        proxy_pass              http://cvat:8080;
        proxy_pass_header       X-CSRFToken;
        proxy_set_header        Host $http_host;
        proxy_pass_header       Set-Cookie;
    }
}

azhavoro · 2020-03-18T20:18:54Z

@rasheeddo thank you, both config look correct.... Could you please run docker exec -it cvat_ui cat /usr/share/nginx/html/index.html and post the output?

rasheeddo · 2020-03-19T00:40:13Z

This is an output from that command

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
failed to resize tty, using default size

azhavoro · 2020-03-19T06:39:58Z

@rasheeddo Many thanks for help with localisation of the issue. By some reason cvat_ui image does not build properly and index.html page is not replaced with right index.html fie(or maybe it isn't generated). I need reproduce this issue, what OS and docker are you using?

rasheeddo · 2020-03-19T07:29:30Z

Thanks for your help too, it would be nice if we can make it works or know the reason why it won't, let me know if you need more information about this machine issue.
This CVAT is runnining on the Amazon EC2 instance it's ubuntu 18.04

Description:	Ubuntu 18.04.3 LTS
Release:	18.04
Codename:	bionic

Docker version on this machine is Docker version 19.03.8, build afacb8b7f0
My local machine is ubuntu 16.04 with same docker version.

azhavoro · 2020-03-19T14:03:23Z

@rasheeddo
Could you please run the following command and attach full output?
docker-compose build --no-cache cvat-ui?
After please run docker-compose -up -d and attach the output of docker exec -it cvat_ui ls /usr/share/nginx/html. Thanks.

rasheeddo · 2020-03-19T14:11:37Z

I entered the command docker-compose build --no-cache cvat-ui shows me this ERROR: No such service: cvat-ui
it showed an error, is there something wrong?
docker-compose up -d gave me this

cvat_redis is up-to-date
cvat_db is up-to-date
cvat is up-to-date
cvat_ui is up-to-date
cvat_proxy is up-to-date

and the last command gave me this

50x.html     favicon.ico  index.html
failed to resize tty, using default size

azhavoro · 2020-03-19T14:33:46Z

please change cvat-ui to cvat_ui in the command docker-compose build --no-cache cvat_ui

rasheeddo · 2020-03-19T15:07:06Z

@azhavoro sorry, I changed that to docker-compose build --no-cache cvat_ui then it shows this

Building cvat_ui
Step 1/23 : FROM node:lts-alpine AS cvat-ui
 ---> 927d03058714
Step 2/23 : ARG http_proxy
 ---> Running in 5ab6c28d6529
Removing intermediate container 5ab6c28d6529
 ---> 10461435f458
Step 3/23 : ARG https_proxy
 ---> Running in 70001e2888b7
Removing intermediate container 70001e2888b7
 ---> eb704641f7b2
Step 4/23 : ARG no_proxy
 ---> Running in 036545af3241
Removing intermediate container 036545af3241
 ---> 699b013c78b7
Step 5/23 : ARG socks_proxy
 ---> Running in d7cf855f83ba
Removing intermediate container d7cf855f83ba
 ---> 21efc7175e9e
Step 6/23 : ENV TERM=xterm     http_proxy=${http_proxy}       https_proxy=${https_proxy}     no_proxy=${no_proxy}     socks_proxy=${socks_proxy}
 ---> Running in fff8be7f41b8
Removing intermediate container fff8be7f41b8
 ---> a5b9e57507be
Step 7/23 : ENV LANG='C.UTF-8'      LC_ALL='C.UTF-8'
 ---> Running in 4ff49543354c
Removing intermediate container 4ff49543354c
 ---> 466cbf9a5d1f
Step 8/23 : COPY cvat-core/package*.json /tmp/cvat-core/
 ---> 16470097c102
Step 9/23 : COPY cvat-canvas/package*.json /tmp/cvat-canvas/
 ---> 33c3974ac604
Step 10/23 : COPY cvat-ui/package*.json /tmp/cvat-ui/
 ---> 474827931e4d
Step 11/23 : WORKDIR /tmp/cvat-core/
 ---> Running in 4f25b8b4a86d
Removing intermediate container 4f25b8b4a86d
 ---> 689ec20b1c6b
Step 12/23 : RUN npm install
 ---> Running in dd9fa8230f31
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated mkdirp@0.5.3: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
npm WARN deprecated left-pad@1.3.0: use String.prototype.padStart()
npm WARN deprecated circular-json@0.3.3: CircularJSON is in maintenance only, flatted is its successor.

> jsonpath@1.0.2 postinstall /tmp/cvat-core/node_modules/jsonpath
> node lib/aesprim.js > generated/aesprim-browser.js


ERROR: Service 'cvat_ui' failed to build: The command '/bin/sh -c npm install' returned a non-zero code: 1

rasheeddo · 2020-03-19T15:19:59Z

after that I entered docker-compose -up -d

Define and run multi-container applications with Docker.

Usage:
  docker-compose [-f <arg>...] [options] [COMMAND] [ARGS...]
  docker-compose -h|--help

Options:
  -f, --file FILE             Specify an alternate compose file
                              (default: docker-compose.yml)
  -p, --project-name NAME     Specify an alternate project name
                              (default: directory name)
  --verbose                   Show more output
  --log-level LEVEL           Set log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
  --no-ansi                   Do not print ANSI control characters
  -v, --version               Print version and exit
  -H, --host HOST             Daemon socket to connect to

  --tls                       Use TLS; implied by --tlsverify
  --tlscacert CA_PATH         Trust certs signed only by this CA
  --tlscert CLIENT_CERT_PATH  Path to TLS certificate file
  --tlskey TLS_KEY_PATH       Path to TLS key file
  --tlsverify                 Use TLS and verify the remote
  --skip-hostname-check       Don't check the daemon's hostname against the
                              name specified in the client certificate
  --project-directory PATH    Specify an alternate working directory
                              (default: the path of the Compose file)
  --compatibility             If set, Compose will attempt to convert keys
                              in v3 files to their non-Swarm equivalent
  --env-file PATH             Specify an alternate environment file

Commands:
  build              Build or rebuild services
  config             Validate and view the Compose file
  create             Create services
  down               Stop and remove containers, networks, images, and volumes
  events             Receive real time events from containers
  exec               Execute a command in a running container
  help               Get help on a command
  images             List images
  kill               Kill containers
  logs               View output from containers
  pause              Pause services
  port               Print the public port for a port binding
  ps                 List containers
  pull               Pull service images
  push               Push service images
  restart            Restart services
  rm                 Remove stopped containers
  run                Run a one-off command
  scale              Set number of containers for a service
  start              Start services
  stop               Stop services
  top                Display the running processes
  unpause            Unpause services
  up                 Create and start containers
  version            Show the Docker-Compose version information

and then docker exec -it cvat_ui ls /usr/share/nginx/html gave me this

50x.html     favicon.ico  index.html

Onion-Skins · 2020-03-22T04:49:18Z

Hi rasheeddo, I just deployed CVAT to a cloud VM, and experience similar issue as you when I try to change the ports setting in docker-compose.override.yml. Later on I found that I have to change cvat-proxy's ports setting in docker-compose.yml directly. Don't know why, but you can have a try.
That is, follow the guide exactly and specify CVAT_HOST in docker-compose.override.yml only. Change ports setting in docker-compose.yml if you really need to do so.
One more caveat, using http with basic authentication on public internet is not a good idea in terms of security as the login credentials are transferred as clear text. You may consider adding a nginx reverse proxy in front of cvat-proxy to enable HTTPs encryption.

rasheeddo · 2020-03-22T08:30:10Z

Hi @Onion-Skins , thanks for your method I tried it but still I ended up at "Welcome to nginx!" page.
What I did are
docker-compose down
At cvat_proxy, I edit on CVAT_HOST to my AWS EC2 instance Public IP for both docker-compose.yml and docker-compose.override.yml, and I changed ports to 7070:80
docker-compose -f docker-compose.yml -f docker-compose.override.yml build
docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d
none of error
here is my docker-compose config

services:
  cvat:
    build:
      args:
        AUTO_SEGMENTATION: "no"
        DJANGO_CONFIGURATION: production
        OPENVINO_TOOLKIT: "no"
        TF_ANNOTATION: "no"
        TZ: Etc/UTC
        USER: django
        http_proxy: ''
        https_proxy: ''
        no_proxy: ''
        socks_proxy: ''
      context: /home/ubuntu/cvat
    container_name: cvat
    depends_on:
      cvat_db:
        condition: service_started
      cvat_redis:
        condition: service_started
    environment:
      ALLOWED_HOSTS: '*'
      DJANGO_MODWSGI_EXTRA_ARGS: ''
    image: cvat
    restart: always
    volumes:
    - cvat_data:/home/django/data:rw
    - cvat_keys:/home/django/keys:rw
    - cvat_logs:/home/django/logs:rw
    - cvat_models:/home/django/models:rw
  cvat_db:
    container_name: cvat_db
    environment:
      POSTGRES_DB: cvat
      POSTGRES_HOST_AUTH_METHOD: trust
      POSTGRES_USER: root
    image: postgres:10-alpine
    networks:
      default:
        aliases:
        - db
    restart: always
    volumes:
    - cvat_db:/var/lib/postgresql/data:rw
  cvat_proxy:
    command: /bin/sh -c "envsubst '$$CVAT_HOST' < /etc/nginx/conf.d/cvat.conf.template
      > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"
    container_name: cvat_proxy
    depends_on:
      cvat:
        condition: service_started
      cvat_ui:
        condition: service_started
    environment:
      CVAT_HOST: 54.238.154.132
    image: nginx:stable-alpine
    ports:
    - 7070:80/tcp
    restart: always
    volumes:
    - /home/ubuntu/cvat/cvat_proxy/conf.d/cvat.conf.template:/etc/nginx/conf.d/cvat.conf.template:ro
    - /home/ubuntu/cvat/cvat_proxy/nginx.conf:/etc/nginx/nginx.conf:ro
  cvat_redis:
    container_name: cvat_redis
    image: redis:4.0-alpine
    networks:
      default:
        aliases:
        - redis
    restart: always
  cvat_ui:
    build:
      args:
        http_proxy: ''
        https_proxy: ''
        no_proxy: ''
        socks_proxy: ''
      context: /home/ubuntu/cvat
      dockerfile: Dockerfile.ui
    container_name: cvat_ui
    depends_on:
      cvat:
        condition: service_started
    networks:
      default:
        aliases:
        - ui
    restart: always
version: '2.3'
volumes:
  cvat_data: {}
  cvat_db: {}
  cvat_keys: {}
  cvat_logs: {}
  cvat_models: {}

which OS are you using on cloud VM?

Onion-Skins · 2020-03-22T16:04:08Z

My cloud VM's OS is Ubuntu 18.04.

your docker-compose.override.yml should look like below. Do not specify ports again here.
version: "2.3"

services:
cvat_proxy:
environment:
CVAT_HOST: 54.238.154.132
2. please only modify cvat_proxy ports setting in docker-compose.yml, .(your ports setting modification is correct). Do not change anything else.

Furthermore, you may also try clearing all CVAT data and and then run "docker-compose ... build" again. On ubuntu, you can use this command to clear CVAT data:
sudo rm -rf /var/lib/docker/volumes/cvat_cvat_data/*
sudo rm -rf /var/lib/docker/volumes/cvat_cvat_db/*
sudo rm -rf /var/lib/docker/volumes/cvat_cvat_keys/*
sudo rm -rf /var/lib/docker/volumes/cvat_cvat_logs/*
sudo rm -rf /var/lib/docker/volumes/cvat_cvat_models/*

rasheeddo · 2020-03-22T17:53:49Z

@Onion-Skins I just removed the ports from docker-compose.override.yml, clear all the data from your command, the config on my docker-compose.yml is as below

#
# Copyright (C) 2018-2020 Intel Corporation
#
# SPDX-License-Identifier: MIT
#
version: "2.3"

services:
  cvat_db:
    container_name: cvat_db
    image: postgres:10-alpine
    networks:
      default:
        aliases:
          - db
    restart: always
    environment:
      POSTGRES_USER: root
      POSTGRES_DB: cvat
      POSTGRES_HOST_AUTH_METHOD: trust
    volumes:
      - cvat_db:/var/lib/postgresql/data

  cvat_redis:
    container_name: cvat_redis
    image: redis:4.0-alpine
    networks:
      default:
        aliases:
          - redis
    restart: always

  cvat:
    container_name: cvat
    image: cvat
    restart: always
    depends_on:
      - cvat_redis
      - cvat_db
    build:
      context: .
      args:
        http_proxy:
        https_proxy:
        no_proxy:
        socks_proxy:
        TF_ANNOTATION: "no"
        AUTO_SEGMENTATION: "no"
        USER: "django"
        DJANGO_CONFIGURATION: "production"
        TZ: "Etc/UTC"
        OPENVINO_TOOLKIT: "no"
    environment:
      DJANGO_MODWSGI_EXTRA_ARGS: ""
      ALLOWED_HOSTS: '*'
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
      - cvat_models:/home/django/models

  cvat_ui:
    container_name: cvat_ui
    restart: always
    build:
      context: .
      args:
        http_proxy:
        https_proxy:
        no_proxy:
        socks_proxy:
      dockerfile: Dockerfile.ui

    networks:
      default:
        aliases:
          - ui
    depends_on:
      - cvat

  cvat_proxy:
    container_name: cvat_proxy
    image: nginx:stable-alpine
    restart: always
    depends_on:
      - cvat
      - cvat_ui
    environment:
      CVAT_HOST: localhost
    ports:
      - "7070:80"
    volumes:
      - ./cvat_proxy/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./cvat_proxy/conf.d/cvat.conf.template:/etc/nginx/conf.d/cvat.conf.template:ro
    command: /bin/sh -c "envsubst '$$CVAT_HOST' < /etc/nginx/conf.d/cvat.conf.template > /etc/nginx/conf.d/default.conf && nginx -g 'daemon off;'"

volumes:
  cvat_db:
  cvat_data:
  cvat_keys:
  cvat_logs:
  cvat_models:

and docker-compose.override.yml is as your suggestion

version: "2.3"

services:
  cvat_proxy:
    environment:
      CVAT_HOST: 54.238.154.132

then I run these
docker-compose down
docker-compose -f docker-compose.yml -f docker-compose.override.yml build
docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d
the output from terminal is here

Creating network "cvat_default" with the default driver
Creating cvat_db    ... done
Creating cvat_redis ... done
Creating cvat       ... done
Creating cvat_ui    ... done
Creating cvat_proxy ... done

but again I still cannot access login page just ended up at welcome page, even tried remove cache on browser but still no luck...

yohayonyon · 2020-03-23T07:48:25Z

I saw the same issue and updated on #1264.
Following here.

nmanovic · 2020-03-25T03:39:11Z

#1261

@nmanovic , I have found two issues with the deployment:

nginx proxy cannot work by default with long domain names (e.g. ec2-x-xx-xx-xxxx.us-east-2.compute.amazonaws.com). Need to specify a directive to resolve the issue. If you have the problem it is easy do diagnose. Just run docker logs cvat_proxy and you will see the recommendation from nginx.

When cvat-ui is complied there is a problem with memory consumed by nodejs. Some problems can be solved by use NODE_OPTIONS=--max_old_space_size=200 environment variable. But webpack consumes too much memory. It leads to "JavaScript heap out of memory" problem (webpack/webpack#6389).

Our team will do the best to resolve the issue ASAP.

P.S. I tried to reproduce the issue on t2.micro with 1G RAM.

rasheeddo · 2020-03-26T07:39:44Z

Just more reported update, I rent a virtual machine server with ubuntu 18.04 OS and do all installation. On docker-compose.yml, I changed CVAT_HOST: localhost to CVAT_HOST: 118.27.32.162 as for my VM's IP address. After docker-compose build and docker-compose up -d, no error comes out. Then from my PC, I tried access that cvat app from chrome browser from this 118.27.32.162:8080, but again I ended up at Nginx welcome page, I can access admin page, but still cannot use the app.
My VM has 1GB memory and 100GB SSD. I am not sure ,but I don't think a long domain name is an issue as @nmanovic mentioned.

But when I tried install cvat on my another ubuntu18 PC, and specified the CVAT_HOST as its local ip address as e.i. 192.168.0.247. From my own laptop I can access the app when with 192.168.0.247:8080 ,both PCs are in the same LAN network.

nmanovic · 2020-03-26T09:07:10Z

@rasheeddo , could you please try my PR? Does it work for you? Be sure that all containers are down when you do build (they take some memory from your system). The primary issue is webpack with less.js memory consumption. It was addressed in the PR.

P.S. 1GB isn't enough in general case to annotate data. We don't have exact requirements for the CVAT server but I would say that 4G is a minimum. But with #1316 you should be able to build CVAT on your 1GB instance and create simple tasks. I tested the scenario on an AWS 1GB instance.

rasheeddo · 2020-03-26T10:17:33Z

@nmanovic , thank you for your help, it's working now! I tried on both VM and AWS EC2, with the branch nm/fix_aws_guide

sramirez · 2020-04-13T15:53:16Z

Hi,

I've managed to deploy "develop" branch to deal with the AWS problem, but now images above a couple of MBs can't be upload to CVAT. However, in master it was possible.

Deployment scenario: t1.micro with 1GB mem.

I've been reviewing logs but I can't only see this message (firstly seen on screen): Error: Could not create the task on the server. Work-horse process was terminated unexpectedly (waitpid returned 9).

I could understand that 100 images could blow up 1GB VM, but with a couple of MB is enough to break the upload.

nmanovic · 2020-04-14T19:59:51Z

@sramirez , why is it critical to use t1.micro in your case?

sramirez · 2020-04-17T11:27:29Z

I was thinking on using free tier, I would try with a few RAM more.

azhavoro added the bug Something isn't working label Mar 18, 2020

azhavoro self-assigned this Mar 18, 2020

azhavoro mentioned this issue Mar 18, 2020

nginx not configured #1264

Closed

nmanovic added the duplicate This issue or pull request already exists label Mar 25, 2020

nmanovic assigned nmanovic and unassigned azhavoro Mar 25, 2020

nmanovic added this to the 0.6.1 - Release milestone Mar 25, 2020

nmanovic mentioned this issue Mar 25, 2020

Fix AWS deployment #1316

Merged

rasheeddo closed this as completed Mar 26, 2020

NaLaUer mentioned this issue Nov 23, 2020

Cannot deploy CVAT #2474

Closed

KushBhatnagar mentioned this issue Apr 20, 2022

Not able to login to CVAT UI which is deployed in AWS EC2 instance #4580

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot deploy CVAT on AWS EC2 instance #1287

Cannot deploy CVAT on AWS EC2 instance #1287

rasheeddo commented Mar 18, 2020

azhavoro commented Mar 18, 2020 •

edited

Loading

azhavoro commented Mar 18, 2020

rasheeddo commented Mar 18, 2020

rasheeddo commented Mar 18, 2020

azhavoro commented Mar 18, 2020

rasheeddo commented Mar 18, 2020

azhavoro commented Mar 18, 2020 •

edited

Loading

rasheeddo commented Mar 19, 2020

azhavoro commented Mar 19, 2020

rasheeddo commented Mar 19, 2020

azhavoro commented Mar 19, 2020

rasheeddo commented Mar 19, 2020

azhavoro commented Mar 19, 2020 •

edited

Loading

rasheeddo commented Mar 19, 2020

rasheeddo commented Mar 19, 2020 •

edited

Loading

Onion-Skins commented Mar 22, 2020

rasheeddo commented Mar 22, 2020

Onion-Skins commented Mar 22, 2020 •

edited

Loading

rasheeddo commented Mar 22, 2020

yohayonyon commented Mar 23, 2020

nmanovic commented Mar 25, 2020

rasheeddo commented Mar 26, 2020

nmanovic commented Mar 26, 2020

rasheeddo commented Mar 26, 2020

sramirez commented Apr 13, 2020

nmanovic commented Apr 14, 2020

sramirez commented Apr 17, 2020

Cannot deploy CVAT on AWS EC2 instance #1287

Cannot deploy CVAT on AWS EC2 instance #1287

Comments

rasheeddo commented Mar 18, 2020

azhavoro commented Mar 18, 2020 • edited Loading

azhavoro commented Mar 18, 2020

rasheeddo commented Mar 18, 2020

rasheeddo commented Mar 18, 2020

azhavoro commented Mar 18, 2020

rasheeddo commented Mar 18, 2020

azhavoro commented Mar 18, 2020 • edited Loading

rasheeddo commented Mar 19, 2020

azhavoro commented Mar 19, 2020

rasheeddo commented Mar 19, 2020

azhavoro commented Mar 19, 2020

rasheeddo commented Mar 19, 2020

azhavoro commented Mar 19, 2020 • edited Loading

rasheeddo commented Mar 19, 2020

rasheeddo commented Mar 19, 2020 • edited Loading

Onion-Skins commented Mar 22, 2020

rasheeddo commented Mar 22, 2020

Onion-Skins commented Mar 22, 2020 • edited Loading

rasheeddo commented Mar 22, 2020

yohayonyon commented Mar 23, 2020

nmanovic commented Mar 25, 2020

rasheeddo commented Mar 26, 2020

nmanovic commented Mar 26, 2020

rasheeddo commented Mar 26, 2020

sramirez commented Apr 13, 2020

nmanovic commented Apr 14, 2020

sramirez commented Apr 17, 2020

azhavoro commented Mar 18, 2020 •

edited

Loading

azhavoro commented Mar 18, 2020 •

edited

Loading

azhavoro commented Mar 19, 2020 •

edited

Loading

rasheeddo commented Mar 19, 2020 •

edited

Loading

Onion-Skins commented Mar 22, 2020 •

edited

Loading