Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fails to export task annotations with "save images" option when using shared storage #6017

Closed
2 tasks done
alkmst-xyz opened this issue Apr 13, 2023 · 4 comments · Fixed by #6193
Closed
2 tasks done
Labels
documentation Documentation should be updated

Comments

@alkmst-xyz
Copy link

My actions before raising this issue

Steps to Reproduce (for bugs)

  1. Start cvat with shared storage option docs.
  2. Create a project and complete annotation task.
  3. Export annotations from "Tasks" page, with "Save Images" toggled.

Expected Behaviour

  1. Annotations (and images) are downloaded correctly when using using shared stoage.

Current Behaviour

  1. Fails to complete download with the following notification.

Screenshot from 2023-04-13 08-22-38

Possible Solution

Maybe the cvat_worker_export service needs to also have access to the cvat_share volume.

Context

Fails to complete annotation export.

Your Environment

  • Git hash commit (git log -1): 61adec5
  • Docker version docker version (e.g. Docker 17.0.05): Docker 23.0.2
  • Are you using Docker Swarm or Kubernetes?: No
  • Operating System and version (e.g. Linux, Windows, MacOS): Ubuntu 22.04
  • Code example or link to GitHub repo or gist to reproduce problem:
  • Other diagnostic information / logs:
Logs from `cvat_worker_export` container
2023-04-13 06:46:11,289 DEBG 'rqworker_export_0' stderr output:
DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2023-04-13 06:46:11,300 DEBG 'rqworker_export_0' stderr output:
DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 90 seconds.

2023-04-13 06:46:11,651 DEBG 'rqworker_export_0' stderr output:
[2023-04-13 06:46:11,648] ERROR cvat.server.task_1: [cvat.apps.dataset_manager.views @ export]: exception occurred
Traceback (most recent call last):
File "/home/django/cvat/apps/dataset_manager/views.py", line 84, in export
export_fn(db_instance.id, temp_file, dst_format,
File "/home/django/cvat/apps/dataset_manager/task.py", line 789, in export_task
task.export(f, exporter, host=server_url, save_images=save_images)
File "/home/django/cvat/apps/dataset_manager/task.py", line 670, in export
exporter(dst_file, temp_dir, task_data, **options)
File "/home/django/cvat/apps/dataset_manager/formats/registry.py", line 36, in call
f_or_cls(*args, **kwargs)
File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1431, in _export_images
_export_task_or_job(dst_file, temp_dir, instance_data,
File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1397, in _export_task_or_job
dump_media_files(instance_data, osp.join(temp_dir, 'images'))
File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1381, in dump_media_files
for frame_id, (frame_data, _) in zip(instance_data.rel_range, frames):
File "/home/django/cvat/apps/engine/frame_provider.py", line 206, in get_frames
yield self.get_frame(idx, quality=quality, out_type=out_type)
File "/home/django/cvat/apps/engine/frame_provider.py", line 196, in get_frame
chunk_reader = loader.load(chunk_number)
File "/home/django/cvat/apps/engine/frame_provider.py", line 86, in load
self.reader_class([self.get_chunk_path(chunk_id, self.quality, self.db_data)[0]]))
File "/home/django/cvat/apps/engine/cache.py", line 46, in get_buf_chunk_with_mime
item = self._get_or_set_cache_item(
File "/home/django/cvat/apps/engine/cache.py", line 39, in _get_or_set_cache_item
item = create_function()
File "/home/django/cvat/apps/engine/cache.py", line 48, in
create_function=lambda: self._prepare_chunk_buff(db_data, quality, chunk_number),
File "/home/django/cvat/apps/engine/cache.py", line 150, in _prepare_chunk_buff
writer.save_as_chunk(images, buff)
File "/home/django/cvat/apps/engine/media_extractors.py", line 646, in save_as_chunk
zip_chunk.write(filename=image, arcname=arcname)
File "/usr/lib/python3.8/zipfile.py", line 1741, in write
zinfo = ZipInfo.from_file(filename, arcname,
File "/usr/lib/python3.8/zipfile.py", line 523, in from_file
st = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/django/share/fruits_dataset/fruits_001.jpg'
ERROR:cvat.server.task_1:[cvat.apps.dataset_manager.views @ export]: exception occurred
Traceback (most recent call last):
File "/home/django/cvat/apps/dataset_manager/views.py", line 84, in export
export_fn(db_instance.id, temp_file, dst_format,
File "/home/django/cvat/apps/dataset_manager/task.py", line 789, in export_task
task.export(f, exporter, host=server_url, save_images=save_images)
File "/home/django/cvat/apps/dataset_manager/task.py", line 670, in export
exporter(dst_file, temp_dir, task_data, **options)
File "/home/django/cvat/apps/dataset_manager/formats/registry.py", line 36, in call
f_or_cls(*args, **kwargs)
File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1431, in _export_images
_export_task_or_job(dst_file, temp_dir, instance_data,
File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1397, in _export_task_or_job
dump_media_files(instance_data, osp.join(temp_dir, 'images'))
File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1381, in dump_media_files
for frame_id, (frame_data, _) in zip(instance_data.rel_range, frames):
File "/home/django/cvat/apps/engine/frame_provider.py", line 206, in get_frames
yield self.get_frame(idx, quality=quality, out_type=out_type)
File "/home/django/cvat/apps/engine/frame_provider.py", line 196, in get_frame
chunk_reader = loader.load(chunk_number)
File "/home/django/cvat/apps/engine/frame_provider.py", line 86, in load
self.reader_class([self.get_chunk_path(chunk_id, self.quality, self.db_data)[0]]))
File "/home/django/cvat/apps/engine/cache.py", line 46, in get_buf_chunk_with_mime
item = self._get_or_set_cache_item(
File "/home/django/cvat/apps/engine/cache.py", line 39, in _get_or_set_cache_item
item = create_function()
File "/home/django/cvat/apps/engine/cache.py", line 48, in
create_function=lambda: self._prepare_chunk_buff(db_data, quality, chunk_number),
File "/home/django/cvat/apps/engine/cache.py", line 150, in _prepare_chunk_buff
writer.save_as_chunk(images, buff)
File "/home/django/cvat/apps/engine/media_extractors.py", line 646, in save_as_chunk
zip_chunk.write(filename=image, arcname=arcname)
File "/usr/lib/python3.8/zipfile.py", line 1741, in write
zinfo = ZipInfo.from_file(filename, arcname,
File "/usr/lib/python3.8/zipfile.py", line 523, in from_file
st = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/django/share/fruits_dataset/fruits_001.jpg'

2023-04-13 06:46:11,651 DEBG 'rqworker_export_0' stderr output:
DEBUG:rq.worker:Handling failed execution of job export:annotations-for-task.id1-in-CVAT_for_images_1.1-format

2023-04-13 06:46:11,652 DEBG 'rqworker_export_0' stderr output:
ERROR:rq.worker:Traceback (most recent call last):
File "/opt/venv/lib/python3.8/site-packages/rq/worker.py", line 936, in perform_job
rv = job.perform()
File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 684, in perform
self._result = self._execute()
File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 690, in _execute
return self.func(*self.args, **self.kwargs)
File "/home/django/cvat/apps/dataset_manager/views.py", line 117, in export_task_as_dataset
return export(dst_format, task_id=task_id, server_url=server_url, save_images=True)
File "/home/django/cvat/apps/dataset_manager/views.py", line 84, in export
export_fn(db_instance.id, temp_file, dst_format,
File "/home/django/cvat/apps/dataset_manager/task.py", line 789, in export_task
task.export(f, exporter, host=server_url, save_images=save_images)
File "/home/django/cvat/apps/dataset_manager/task.py", line 670, in export
exporter(dst_file, temp_dir, task_data, **options)
File "/home/django/cvat/apps/dataset_manager/formats/registry.py", line 36, in call
f_or_cls(*args, **kwargs)
File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1431, in _export_images
_export_task_or_job(dst_file, temp_dir, instance_data,
File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1397, in _export_task_or_job
dump_media_files(instance_data, osp.join(temp_dir, 'images'))
File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1381, in dump_media_files
for frame_id, (frame_data, _) in zip(instance_data.rel_range, frames):
File "/home/django/cvat/apps/engine/frame_provider.py", line 206, in get_frames
yield self.get_frame(idx, quality=quality, out_type=out_type)
File "/home/django/cvat/apps/engine/frame_provider.py", line 196, in get_frame
chunk_reader = loader.load(chunk_number)
File "/home/django/cvat/apps/engine/frame_provider.py", line 86, in load
self.reader_class([self.get_chunk_path(chunk_id, self.quality, self.db_data)[0]]))
File "/home/django/cvat/apps/engine/cache.py", line 46, in get_buf_chunk_with_mime
item = self._get_or_set_cache_item(
File "/home/django/cvat/apps/engine/cache.py", line 39, in _get_or_set_cache_item
item = create_function()
File "/home/django/cvat/apps/engine/cache.py", line 48, in
create_function=lambda: self._prepare_chunk_buff(db_data, quality, chunk_number),
File "/home/django/cvat/apps/engine/cache.py", line 150, in _prepare_chunk_buff
writer.save_as_chunk(images, buff)
File "/home/django/cvat/apps/engine/media_extractors.py", line 646, in save_as_chunk
zip_chunk.write(filename=image, arcname=arcname)
File "/usr/lib/python3.8/zipfile.py", line 1741, in write
zinfo = ZipInfo.from_file(filename, arcname,
File "/usr/lib/python3.8/zipfile.py", line 523, in from_file
st = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/django/share/fruits_dataset/fruits_001.jpg'
Traceback (most recent call last):
File "/opt/venv/lib/python3.8/site-packages/rq/worker.py", line 936, in perform_job
rv = job.perform()
File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 684, in perform
self._result = self._execute()
File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 690, in _execute
return self.func(*self.args, **self.kwargs)
File "/home/django/cvat/apps/dataset_manager/views.py", line 117, in export_task_as_dataset
return export(dst_format, task_id=task_id, server_url=server_url, save_images=True)
File "/home/django/cvat/apps/dataset_manager/views.py", line 84, in export
export_fn(db_instance.id, temp_file, dst_format,
File "/home/django/cvat/apps/dataset_manager/task.py", line 789, in export_task
task.export(f, exporter, host=server_url, save_images=save_images)
File "/home/django/cvat/apps/dataset_manager/task.py", line 670, in export
exporter(dst_file, temp_dir, task_data, **options)
File "/home/django/cvat/apps/dataset_manager/formats/registry.py", line 36, in call
f_or_cls(*args, **kwargs)
File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1431, in _export_images
_export_task_or_job(dst_file, temp_dir, instance_data,
File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1397, in _export_task_or_job
dump_media_files(instance_data, osp.join(temp_dir, 'images'))
File "/home/django/cvat/apps/dataset_manager/formats/cvat.py", line 1381, in dump_media_files
for frame_id, (frame_data, _) in zip(instance_data.rel_range, frames):
File "/home/django/cvat/apps/engine/frame_provider.py", line 206, in get_frames
yield self.get_frame(idx, quality=quality, out_type=out_type)
File "/home/django/cvat/apps/engine/frame_provider.py", line 196, in get_frame
chunk_reader = loader.load(chunk_number)
File "/home/django/cvat/apps/engine/frame_provider.py", line 86, in load
self.reader_class([self.get_chunk_path(chunk_id, self.quality, self.db_data)[0]]))
File "/home/django/cvat/apps/engine/cache.py", line 46, in get_buf_chunk_with_mime
item = self._get_or_set_cache_item(
File "/home/django/cvat/apps/engine/cache.py", line 39, in _get_or_set_cache_item
item = create_function()
File "/home/django/cvat/apps/engine/cache.py", line 48, in
create_function=lambda: self._prepare_chunk_buff(db_data, quality, chunk_number),
File "/home/django/cvat/apps/engine/cache.py", line 150, in _prepare_chunk_buff
writer.save_as_chunk(images, buff)
File "/home/django/cvat/apps/engine/media_extractors.py", line 646, in save_as_chunk
zip_chunk.write(filename=image, arcname=arcname)
File "/usr/lib/python3.8/zipfile.py", line 1741, in write
zinfo = ZipInfo.from_file(filename, arcname,
File "/usr/lib/python3.8/zipfile.py", line 523, in from_file
st = os.stat(filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/django/share/fruits_dataset/fruits_001.jpg'

2023-04-13 06:46:11,652 DEBG 'rqworker_export_0' stderr output:
DEBUG:rq.worker:Invoking exception handler <function rq_exception_handler at 0x7f6310f0df70>

2023-04-13 06:46:11,653 DEBG 'rqworker_export_0' stderr output:
DEBUG:rq.worker:Invoking exception handler <function handle_rq_exception at 0x7f6310f59e50>

2023-04-13 06:46:11,663 DEBG 'rqworker_export_0' stderr output:
DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

2023-04-13 06:46:11,664 DEBG 'rqworker_export_0' stderr output:
DEBUG:rq.worker:*** Listening on export...

2023-04-13 06:46:11,664 DEBG 'rqworker_export_0' stderr output:
DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.

@cherrychengzi
Copy link

This problem is same with me.

@alkmst-xyz
Copy link
Author

For me this is resolved when I mount cvat_share to cvat_worker_export.

cvat_worker_export:
    volumes:
      - cvat_share:/home/django/share:ro

@cherrychengzi
Copy link

cherrychengzi commented Apr 26, 2023 via email

@cherrychengzi
Copy link

cherrychengzi commented May 24, 2023 via email

@bsekachev bsekachev added the documentation Documentation should be updated label May 24, 2023
bsekachev added a commit that referenced this issue May 24, 2023
<!-- Raise an issue to propose your change
(https://github.com/opencv/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution
guide](https://opencv.github.io/cvat/docs/contributing/). -->

<!-- Provide a general summary of your changes in the Title above -->

### Motivation and context
Resolved #5887
Resolved #6017

### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->

### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [ ] I have added a description of my changes into the
[CHANGELOG](https://github.com/opencv/cvat/blob/develop/CHANGELOG.md)
file
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](

https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary

([cvat-canvas](https://github.com/opencv/cvat/tree/develop/cvat-canvas#versioning),

[cvat-core](https://github.com/opencv/cvat/tree/develop/cvat-core#versioning),

[cvat-data](https://github.com/opencv/cvat/tree/develop/cvat-data#versioning)
and

[cvat-ui](https://github.com/opencv/cvat/tree/develop/cvat-ui#versioning))

### License

- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/opencv/cvat/blob/develop/LICENSE) that covers the
project.
  Feel free to contact the maintainers if that's a concern.
mikhail-treskin pushed a commit to retailnext/cvat that referenced this issue Jul 1, 2023
<!-- Raise an issue to propose your change
(https://github.com/opencv/cvat/issues).
It helps to avoid duplication of efforts from multiple independent
contributors.
Discuss your ideas with maintainers to be sure that changes will be
approved and merged.
Read the [Contribution
guide](https://opencv.github.io/cvat/docs/contributing/). -->

<!-- Provide a general summary of your changes in the Title above -->

### Motivation and context
Resolved cvat-ai#5887
Resolved cvat-ai#6017

### How has this been tested?
<!-- Please describe in detail how you tested your changes.
Include details of your testing environment, and the tests you ran to
see how your change affects other areas of the code, etc. -->

### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes
that apply.
If an item isn't applicable for some reason, then ~~explicitly
strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the
pull request.
If you're unsure about any of these, don't hesitate to ask. We're here
to help! -->
- [x] I submit my changes into the `develop` branch
- [ ] I have added a description of my changes into the
[CHANGELOG](https://github.com/opencv/cvat/blob/develop/CHANGELOG.md)
file
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [ ] I have linked related issues (see [GitHub docs](

https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [ ] I have increased versions of npm packages if it is necessary

([cvat-canvas](https://github.com/opencv/cvat/tree/develop/cvat-canvas#versioning),

[cvat-core](https://github.com/opencv/cvat/tree/develop/cvat-core#versioning),

[cvat-data](https://github.com/opencv/cvat/tree/develop/cvat-data#versioning)
and

[cvat-ui](https://github.com/opencv/cvat/tree/develop/cvat-ui#versioning))

### License

- [x] I submit _my code changes_ under the same [MIT License](
https://github.com/opencv/cvat/blob/develop/LICENSE) that covers the
project.
  Feel free to contact the maintainers if that's a concern.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Documentation should be updated
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants