Cannot export dataset (KeyError: 'outside') / can't import backup from older version

[niqbal996]

My actions before raising this issue

• Read/searched the docs
• Searched past issues

I have a dataset that got labelled on CVAT version 2.2 specifically this commit db6bbba. The labelling is finished and now when I try exporting the dataset, it gives me the error either, index out of range or Work-horse was terminated unexpectedly (waitpid returned 9), or Error: Request failed with status code 500. "KeyError: 'outside'\n". I looked at this issue #5600 and thought this may have been fixed in the newer version. I have backed up my projects and tried uploading the project to a version tag of v2.3.0 and v2.4.0. I followed your guide to upgrading from v2.2 to v2.3 from the documents. But the result is stil the same.

Steps to Reproduce (for bugs)

I can provide the backed up cvat project zip file (600 MB) that I created using Backup Project or the backup tar.gz files using the CVAT backup guide from the documentation. The project has a validation split which can be exported easily but the training split roughly 150 images is the problem. You can try uploading it to any CVAT version instance from v2.2, v2.3 or v2.4. The export dataset functionality is not working for me.

Expected Behaviour

I expect that CVAT can generate a dataset for me that I can use for training my CNNs.

Current Behaviour

I can restore the projects using backed up files cvat_data.tar.gz cvat_db.tar.gz and see the images and labels on GUI. But the exported dataset not working. I also tried restoring via the CVAT project backup zip file but it does not work as either on CVAT 2.4, 2.3.

Possible Solution

Should I try migrating my project to a different CVAT version or should i try handpicking with my labels somehow that are faulty maybe? I do not know what could be a potential fix to this.

Context

We are using this dataset for an upcoming research publication on Drone recorded dataset which could be crucial in designing low cost Crop monitoring systems with Drones in Agriculture. It would help alot if i can quickly get access to my labelled dataset in either YOLO or COCO dataset format for my training pipeline.

Your Environment

• Git hash commit (git log -1): db6bbbab
• Docker version: on the original machine where data was labelled

Client: Docker Engine - Community
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:58 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:49 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.9
  GitCommit:        1c90a442489720eec95342e1789ee8a5e1b9536f
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

tried to reproduce on my local machine with Docker version:

Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:08 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:25:58 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.15
  GitCommit:        5b842e528e99d4d4c1686467debf2bd4b88ecd86
 nvidia:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

• Are you using Docker Swarm or Kubernetes? No

• Ubuntu 20.04

• I have used CVAT commit db6bbbab for labelling and when the export dataset failed i tried tag version v2.2.0, v2.3.0 and v2.4.0.

• I have attached some logs of the cvat_server container using docker logs cvat_server -f :

  cvat_server logs

  ` * [cvat_server_logs.txt](https://github.com/opencv/cvat/files/11148122/cvat_server_logs.txt)

I can provide the backup files for reproducibiltiy in any medium possible. Please let me know whats the fastest method for you. Would help a lot if we can quickly resolve this. Thank you.

[zhiltsov-max]

Hi, please attach logs from the cvat_worker_export container, if possible.

I can restore the projects using backed up files cvat_data.tar.gz cvat_db.tar.gz and see the images and labels on GUI.
But the exported dataset not working.

I also tried restoring via the CVAT project backup zip file but it does not work as either on CVAT 2.4, 2.3.

What didn't work exactly in the restored project? Please specify.

[niqbal996]

Hi @zhiltsov-max,
Thank you for the quick response. Here are the logs from the cvat_worker_export container. I have only attached the logs after the export is triggered on GUI.
cvat_worker_export_logs.txt
This is the error that appears when I try to upload the restored project zip file generated from version 2.2 to version 2.4 at the moment.

Error: Request failed with status code 500. "Exception: File with meta information can only be uploaded with video/images \n".

Maybe this could be relevant, at some point during labelling, the labeller put tracks instead of shapes in the images which I removed manually using Remove annotations functionality. I have checked using the tracks filter throughout the task, and it seems there are no tracks left and only shapes/boxes which is what i want. Just putting it here, in case I missed something.

[zhiltsov-max]

Ok. Yes, the issue might be fixed already in the newer releases. Could you please try to use the backup archive, but without the manifest.jsonl file inside? Just remove it from the archive, and try to upload again. If it still doesn't work, please share the backup archive with all personal data removed (without images).

[niqbal996]

I have tried removing the manifest.jsonl but the issue still remain. This was the error:

Could not restore project backup.

Error: Request failed with status code 500. "ValueError: Only one video, archive, pdf, zip or many image, directory can be used simultaneously, but 39 image(s), 0 video(s), 1 archive(s), 0 directory(s), 0 pdf(s), 0 zip(s) found.\n".

I have removed all the images from the dataset and attached the project backup zip file here.
beets_backup2 - Copy.zip

[niqbal996]

I am able to upload the project backup by removing the manifest.jsonl file from the backup into CVAT version 2.3 but it did not work with version 2.4. Though the original problem of exporting the dataset still remains. There are the error logs given by the cvat_worker_default container in version 2.3:

2023-04-06 09:36:13,833 DEBG 'rqworker_default_0' stderr output:
[2023-04-06 09:36:13,833] ERROR cvat.server.task_63: [cvat.apps.dataset_manager.views @ export]: exception occurred
Traceback (most recent call last):
  File "/home/django/cvat/apps/dataset_manager/views.py", line 81, in export
    export_fn(db_instance.id, temp_file, dst_format,
  File "/home/django/cvat/apps/dataset_manager/task.py", line 775, in export_task
    task.export(f, exporter, host=server_url, save_images=save_images)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 657, in export
    exporter(dst_file, task_data, **options)
  File "/home/django/cvat/apps/dataset_manager/formats/registry.py", line 37, in __call__
    f_or_cls(*args, **kwargs)
  File "/home/django/cvat/apps/dataset_manager/formats/coco.py", line 19, in _export
    dataset = Dataset.from_extractors(GetCVATDataExtractor(
  File "/home/django/cvat/apps/dataset_manager/bindings.py", line 1468, in GetCVATDataExtractor
    return CvatTaskOrJobDataExtractor(instance_data, include_images, format_type, dimension)
  File "/home/django/cvat/apps/dataset_manager/bindings.py", line 1302, in __init__
    for frame_data in instance_data.group_by_frame(include_empty=True):
  File "/home/django/cvat/apps/dataset_manager/bindings.py", line 388, in group_by_frame
    for shape in sorted(anno_manager.to_shapes(len(self)),
  File "/home/django/cvat/apps/dataset_manager/annotation.py", line 164, in to_shapes
    return shapes + tracks.to_shapes(end_frame)
  File "/home/django/cvat/apps/dataset_manager/annotation.py", line 364, in to_shapes
    for shape in TrackManager.get_interpolated_shapes(track, 0, end_frame):
  File "/home/django/cvat/apps/dataset_manager/annotation.py", line 762, in get_interpolated_shapes
    if not prev_shape["outside"]:
KeyError: 'outside'

2023-04-06 09:36:13,834 DEBG 'rqworker_default_0' stderr output:
ERROR:cvat.server.task_63:[cvat.apps.dataset_manager.views @ export]: exception occurred
Traceback (most recent call last):
  File "/home/django/cvat/apps/dataset_manager/views.py", line 81, in export
    export_fn(db_instance.id, temp_file, dst_format,
  File "/home/django/cvat/apps/dataset_manager/task.py", line 775, in export_task
    task.export(f, exporter, host=server_url, save_images=save_images)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 657, in export
    exporter(dst_file, task_data, **options)
  File "/home/django/cvat/apps/dataset_manager/formats/registry.py", line 37, in __call__
    f_or_cls(*args, **kwargs)
  File "/home/django/cvat/apps/dataset_manager/formats/coco.py", line 19, in _export
    dataset = Dataset.from_extractors(GetCVATDataExtractor(
  File "/home/django/cvat/apps/dataset_manager/bindings.py", line 1468, in GetCVATDataExtractor
    return CvatTaskOrJobDataExtractor(instance_data, include_images, format_type, dimension)
  File "/home/django/cvat/apps/dataset_manager/bindings.py", line 1302, in __init__
    for frame_data in instance_data.group_by_frame(include_empty=True):
  File "/home/django/cvat/apps/dataset_manager/bindings.py", line 388, in group_by_frame
    for shape in sorted(anno_manager.to_shapes(len(self)),
  File "/home/django/cvat/apps/dataset_manager/annotation.py", line 164, in to_shapes
    return shapes + tracks.to_shapes(end_frame)
  File "/home/django/cvat/apps/dataset_manager/annotation.py", line 364, in to_shapes
    for shape in TrackManager.get_interpolated_shapes(track, 0, end_frame):
  File "/home/django/cvat/apps/dataset_manager/annotation.py", line 762, in get_interpolated_shapes
    if not prev_shape["outside"]:
KeyError: 'outside'

Any ideas why this happens?

[niqbal996]

In my opinion, the tracks.to_shapes() function call should not be executed since I only should have shapes in my dataset and no tracks. I am concerned that maybe since the dataset was mistakenly labelled with tracks at some point, there are still some mishandlings as tracks? I have checked the dataset many times using the filter button and I cannot see any tracks at the moment.

[zhiltsov-max]

I see there are several tracks without shapes inside on the frame 25 in the task 1 (in /task_1/annotations.json, just open and search for "tracks"). I was able to reproduce the problem. After I tried to upload backup with the tracks removed, the export worked for me. Could you please try this?

[niqbal996]

Hi @zhiltsov-max,
I think your fix is on point but unfortunately I still cant manage to upload the project zip file with corrected annotations.json file. I did the correction as following. Both of them gave the same error.

[
    {
        "version": 0,
        "tags": [],
        "shapes":["some shapes here"]
        "tracks": []  "removed whatever was in here"
    }
]

[
    {
        "version": 0,
        "tags": [],
        "shapes":["some shapes here"] "removed tracks category altogether"
    }
]

The error message:

Error: Request failed with status code 500. "Exception: File with meta information can only be uploaded with video/images \n".

[niqbal996]

I have also tried to remove the annotations via the GUI, deleted the image and made a new project export zip file, but these track annotations are still present in the annotations.json file. I can edit it manually for now but just a fix from the GUI side would be much more easier in future.

[zhiltsov-max]

Please check this fixed backup archive (no images inside, please put your images into data/ folders):

beets_backup-fixed.zip

Here are the annotations in COCO format (image names and sizes are different from yours):

project_beets-2023_04_06_20_46_53-coco 1.0.zip

[niqbal996]

okay, I have been able to export the dataset now in both COCO and YOLO. Thank you for your help. Really appreciate it. I did a silly mistake of not removing the manifest.jsonl file even though you told me earlier to remove it from the zip file as well. To summarize, the core issue was that after the labels were mistakenly labelled with tracks, I had removed them using the remove annotations option in the GUI but it led to leftovers in the annotations.json file as tracks with no shape values. I see you already labelled it as bug. You can close this issue.

[Taimourrao]

Hi
While exporting CVAT is giving me this error (Error: Request failed with status code 500. "KeyError: 'outside'\n".)? Can anybody help me in this?

[Taimourrao]

Hi @zhiltsov-max,
Can you please take a look to this issue?

[zhiltsov-max]

@Taimourrao, should be fixed in #6204