Cannot use mounted volume with s3fs (S3 bucket) with cvat

[omar-ogm]

My actions before raising this issue

• Read/searched the docs
• Searched past issues

Hi, I've been really trying to get this to work but there is not a lot of documentation and none of the previous issues seem to be the same I have.
What Im trying to achieve is to use another data storage for CVAT, an s3 bucket on AWS for the volume cvat_data. I've also tried using cvat_share with the same results.

I've mounted the bucket using the tool s3fs as state in the documentation and the I tried modifying the docker-compose use that volume instead of the name volume that comes by default.

My docker-compose.yml

cvat_server:
    container_name: cvat_server
    image: cvat/server:${CVAT_VERSION:-dev}
    restart: always
    depends_on:
      - cvat_redis
      - cvat_opa
    env_file:
      - .env
    environment:
      DJANGO_MODWSGI_EXTRA_ARGS: ''
      ALLOWED_HOSTS: '*'
      CVAT_REDIS_HOST: 'cvat_redis'
      ADAPTIVE_AUTO_ANNOTATION: 'false'
      no_proxy: elasticsearch,kibana,logstash,nuclio,opa,${no_proxy}
      NUMPROCS: 1
    command: -c supervisord/server.conf
    labels:
      - traefik.enable=true
      - traefik.http.services.cvat.loadbalancer.server.port=8080
      - traefik.http.routers.cvat.rule=Host(`${CVAT_HOST:-localhost}`) &&
        PathPrefix(`/api/`, `/git/`, `/opencv/`, `/static/`, `/admin`, `/documentation/`, `/django-rq`)
      - traefik.http.routers.cvat.entrypoints=web
    volumes:
      - cvat_data:/home/django/data
      - cvat_keys:/home/django/keys
      - cvat_logs:/home/django/logs
    networks:
      - cvat

... 

volumes:
  cvat_data:
    driver_opts:
      type: none
      device: /home/ubuntu/mnt/s3-cvat-bucket
      o: bind
  cvat_keys:
  cvat_logs:

• The directory "/home/ubuntu/mnt/s3-cvat-bucket" exist
• The s3fs is mounted successfully, it works (I`ve trying creating some files and those are reflected on AWS), is mounted with the allow_other options to avoid issues with permissions.
• The mount output is this: s3fs on /home/ubuntu/mnt/s3-cvat-bucket type fuse.s3fs (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000,allow_other)

Current Behaviour

When I attempt to run the docker-compose I get the following error:
ERROR: for cvat_server Cannot create container for service cvat_worker_low: failed to chmod on /var/lib/docker/volumes/cvat_cvat_data/_data: chmod /var/lib/docker/volumes/cvat_cvat_data/_data: input/output error

If a use any other folder that I create it works. It seems to be a problem with this mounted folder.

The error seems to be highly related with privileges, and not directly with CVAT I guess. This is the output of the "ls -l" command.
drwxrwxrwx 1 ubuntu ubuntu 0 Jan 1 1970 s3-cvat-bucket-unusuals

I've tried so many things that I dont even remember all. This is all that I've been capable of narrowing down the error.

Some issues that were helpful but didnt help solving the problem #2263, #3935, #1472, #4965

[zhiltsov-max]

@Marishka17, could you look at this problem?

[Marishka17]

Hi @omar-ogm, Could you please try to remove previous volumes, create docker-compose.override.yml file with content

version: '3.3'

services:
  cvat_server:
    volumes:
      - cvat_share:/home/django/share:ro
  cvat_worker_default:
    volumes:
      - cvat_share:/home/django/share:ro

volumes:
  cvat_share:
    driver_opts:
      type: none
      device: /home/ubuntu/mnt/s3-cvat-bucket
      o: bind

and run cvat_server docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d --build cvat_server
Don't use cvat_data volume because we mounted it without ro option and in case of success, additional CVAT files will be written to your s3 bucket.

I can't reproduce this problem on my side and I have no more idea what can be a problem. I see that bucket mounting is correct, I have the same directory permissions and all work for me. I guess this problem is not related to CVAT. If it's possible, could you please try to reproduce this problem on another machine? (e.g use a virtual machine)

[nmanovic]

@omar-ogm , I will close the issue. Feel free to reopen if you can provide more information about the issue.

[KDD2018]

hi @Marishka17, I have the same config, but it dosen't work and has no error. It's unthinkable!

version: '3.3'

services:
cvat_server:
volumes:
- cvat_share:/home/django/share:ro
cvat_worker_default:
volumes:
- cvat_share:/home/django/share:ro

volumes:
cvat_share:
driver_opts:
type: none
device: /home/ubuntu/mnt/s3-cvat-bucket
o: bind