Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't download a very large dataset by cvat_sdk #6026

Closed
ergouu opened this issue Apr 14, 2023 · 0 comments
Closed

Can't download a very large dataset by cvat_sdk #6026

ergouu opened this issue Apr 14, 2023 · 0 comments

Comments

@ergouu
Copy link

ergouu commented Apr 14, 2023

My actions before raising this issue

Steps to Reproduce (for bugs)

This is my code:

        loader=projects_api.retrieve_dataset
        while True:
            # generate dataset zip file
            _,response=loader(
                id=id,
                format=format, # 'ImageNet 1.0'
                location=location, # 'local'
                _parse_response=False
            )
            
            # status code 201 means the zip file has been generated
            if response.status ==201:
                break
            time.sleep(5)

        _,response=loader(
                id=id,
                format=format,
                location=location,
                action='download',
                _parse_response=False
            )
        
        with open(output_file,'wb') as f: # output_file='big_test.zip'
            f.write(response.data)

Expected Behaviour

I want to export a large dataset which contains about 1000+ images by format 'ImageNet 1.0'

Current Behaviour

The zip file has been generated successfully, but the output file specified locally is empty. I think it is beacuse the file is too big to transfer by http session.

CVAT Host, in deffierent network with local host:
/var/lib/docker/volumes/cvat_cvat_data/_data/projects/5/export_cache/dataset_image_net-10.ZIP is which I want to export.

 ergouu@r740:~$ sudo find /var/lib/docker/volumes/cvat_cvat_data/_data/ -name *.ZIP|xargs -i ls -l {}
-rw-r--r-- 1 dell dell 5505275816 Apr 14 10:36 /var/lib/docker/volumes/cvat_cvat_data/_data/projects/3/export_cache/dataset_image_net-10.ZIP
-rw-r--r-- 1 dell dell 3077 Apr 14 11:09 /var/lib/docker/volumes/cvat_cvat_data/_data/projects/3/export_cache/annotations_image_net-10.ZIP
-rw-r--r-- 1 dell dell 106939 Apr 14 08:53 /var/lib/docker/volumes/cvat_cvat_data/_data/projects/9/export_cache/dataset_image_net-10.ZIP
-rw-r--r-- 1 dell dell 93991984216 Apr 14 11:35 /var/lib/docker/volumes/cvat_cvat_data/_data/projects/5/export_cache/dataset_image_net-10.ZIP
-rw-r--r-- 1 dell dell 203245 Apr 14 11:21 /var/lib/docker/volumes/cvat_cvat_data/_data/projects/5/export_cache/annotations_image_net-10.ZIP

Local Host, in Docker:

root@66c2d5f16612:/code# ls -l big_test.zip 
-rw-r--r-- 1 root root 0 Apr 14 11:35 big_test.zip

Possible Solution

Context

Your Environment

CVAT Version:2.3.0

  • Git hash commit (git log -1):
  • Docker version docker version (e.g. Docker 17.0.05):
  • Are you using Docker Swarm or Kubernetes?
  • Operating System and version (e.g. Linux, Windows, MacOS):
  • Code example or link to GitHub repo or gist to reproduce problem:
  • Other diagnostic information / logs:
    Logs from `cvat` container
@ergouu ergouu changed the title Can't download very a large dataset by cvat_sdk Can't download a very large dataset by cvat_sdk Apr 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants