Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Export then import using CVAT Video 1.1 #6219

Closed
2 tasks done
realtimshady1 opened this issue Jun 1, 2023 · 7 comments
Closed
2 tasks done

Export then import using CVAT Video 1.1 #6219

realtimshady1 opened this issue Jun 1, 2023 · 7 comments
Labels
need info Need more information to investigate the issue

Comments

@realtimshady1
Copy link

realtimshady1 commented Jun 1, 2023 •
edited
Loading

My actions before raising this issue

Steps to Reproduce (for bugs)

  1. Export a task that contains tracks and shapes using CVAT Video 1.1 format (or Datumaro 1.1)
  2. Upload the labels

Expected Behaviour

If I export the data and then reupload it, it should be uploaded with no errors

Current Behaviour

The annotations fail to be uploaded and returns an error message
image

Possible Solution

If I export then upload using MOT, it works, however MOT format lacks the details fields so that information is lost.

Your Environment

Version: 8fae5d3 (HEAD) Release 2.5.0
Docker version: 20.10.22
Docker Swarm/ Kubernetes: No
OS: Debian GNU/Linux 11 (bullseye)

Here are two snippets of the annotations.xml file

      <labels>
        <label>
          <name>bird</name>
          <color>#ff7f7f</color>
          <type>rectangle</type>
          <attributes>
            <attribute>
              <name>Low Confidence</name>
              <mutable>False</mutable>
              <input_type>checkbox</input_type>
              <default_value>false</default_value>
              <values>false</values>
            </attribute>
          </attributes>
        </label>

    <box frame="10499" keyframe="1" outside="1" occluded="0" xtl="260.00" ytl="238.00" xbr="272.00" ybr="250.00" z_order="0">
      <attribute name="Low Confidence">false</attribute>
    </box>
  </track>
  <track id="105" label="macropod" source="manual">
    <box frame="4" keyframe="1" outside="0" occluded="0" xtl="154.61" ytl="269.66" xbr="163.04" ybr="278.08" z_order="0">
      <attribute name="Low Confidence">false</attribute>
    </box>
    <box frame="9" keyframe="1" outside="1" occluded="0" xtl="154.61" ytl="269.66" xbr="162.89" ybr="278.08" z_order="0">
      <attribute name="Low Confidence">false</attribute>
    </box>
  </track>

Docker Logs after exporting and importing with CVAT format (url removed)

cvat_worker_export      | 2023-06-16 09:45:02,710 DEBG 'rqworker-export-1' stderr output:
cvat_worker_export      | INFO:rq.worker:export: cvat.apps.dataset_manager.views.export_task_annotations(302, 'CVAT for video 1.1', 'http://##########') (export:annotations-for-task.id302-in-CVAT_for_video_1.1-format)
cvat_worker_export      |
cvat_worker_export      | 2023-06-16 09:45:02,711 DEBG 'rqworker-export-1' stderr output:
cvat_worker_export      | DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
cvat_worker_export      |
cvat_worker_export      | 2023-06-16 09:45:02,720 DEBG 'rqworker-export-1' stderr output:
cvat_worker_export      | DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 90 seconds.
cvat_worker_export      |
cvat_server             | INFO:     220.233.199.36:0 - "GET /api/tasks/302/annotations?org=DPE&use_default_location=true&filename=cvat2.zip&format=CVAT+for+video+1.1 HTTP/1.0" 202 Accepted
cvat_server             | 2023-06-16 09:45:02,722 DEBG 'uvicorn-0' stdout output:
cvat_server             | INFO:     220.233.199.36:0 - "GET /api/tasks/302/annotations?org=DPE&use_default_location=true&filename=cvat2.zip&format=CVAT+for+video+1.1 HTTP/1.0" 202 Accepted
cvat_server             |
cvat_worker_export      | 2023-06-16 09:45:03,070 DEBG 'rqworker-export-1' stderr output:
cvat_worker_export      | [2023-06-16 09:45:03,065] INFO cvat.server.task_302: The task '20220728_DJI_0050' is exported as 'CVAT for video 1.1' at '/home/django/data/tasks/302/export_cache/annotations_cvat-for-video-11.ZIP' and available for downloading for the next 10:00:00. Export cache cleaning job is enqueued, id 'f2a00f10-71a0-4feb-9cc0-0e2b415dcccd'
cvat_worker_export      |
cvat_worker_export      | 2023-06-16 09:45:03,071 DEBG 'rqworker-export-1' stderr output:
cvat_worker_export      | INFO:cvat.server.task_302:The task '20220728_DJI_0050' is exported as 'CVAT for video 1.1' at '/home/django/data/tasks/302/export_cache/annotations_cvat-for-video-11.ZIP' and available for downloading for the next 10:00:00. Export cache cleaning job is enqueued, id 'f2a00f10-71a0-4feb-9cc0-0e2b415dcccd'
cvat_worker_export      |
cvat_worker_export      | 2023-06-16 09:45:03,072 DEBG 'rqworker-export-1' stderr output:
cvat_worker_export      | DEBUG:rq.worker:Handling successful execution of job export:annotations-for-task.id302-in-CVAT_for_video_1.1-format
cvat_worker_export      |
cvat_worker_export      | 2023-06-16 09:45:03,074 DEBG 'rqworker-export-1' stderr output:
cvat_worker_export      | INFO:rq.worker:export: Job OK (export:annotations-for-task.id302-in-CVAT_for_video_1.1-format)
cvat_worker_export      |
cvat_worker_export      | 2023-06-16 09:45:03,074 DEBG 'rqworker-export-1' stderr output:
cvat_worker_export      | DEBUG:rq.worker:Result: '/home/django/data/tasks/302/export_cache/annotations_cvat-for-video-11.ZIP'
cvat_worker_export      | INFO:rq.worker:Result is kept for 36000 seconds
cvat_worker_export      |
cvat_worker_export      | 2023-06-16 09:45:03,084 DEBG 'rqworker-export-1' stderr output:
cvat_worker_export      | DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
cvat_worker_export      | DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
cvat_worker_export      |
cvat_worker_export      | 2023-06-16 09:45:03,085 DEBG 'rqworker-export-1' stderr output:
cvat_worker_export      | DEBUG:rq.worker:*** Listening on export...
cvat_worker_export      |
cvat_worker_export      | 2023-06-16 09:45:03,086 DEBG 'rqworker-export-1' stderr output:
cvat_worker_export      | DEBUG:rq.worker:Sent heartbeat to prevent worker timeout. Next one should arrive within 480 seconds.
cvat_worker_export      |
cvat_server             | INFO:     220.233.199.36:0 - "GET /api/tasks/302/annotations?org=DPE&use_default_location=true&filename=cvat2.zip&format=CVAT+for+video+1.1 HTTP/1.0" 201 Created
cvat_server             | 2023-06-16 09:45:05,837 DEBG 'uvicorn-1' stdout output:
cvat_server             | INFO:     220.233.199.36:0 - "GET /api/tasks/302/annotations?org=DPE&use_default_location=true&filename=cvat2.zip&format=CVAT+for+video+1.1 HTTP/1.0" 201 Created
cvat_server             |
cvat_server             | INFO:     192.168.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified
cvat_server             | 2023-06-16 09:45:05,938 DEBG 'uvicorn-0' stdout output:
cvat_server             | INFO:     192.168.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified
cvat_server             |
cvat_server             | INFO:     220.233.199.36:0 - "GET /api/tasks/302/annotations?org=DPE&use_default_location=true&filename=cvat2.zip&format=CVAT+for+video+1.1&action=download HTTP/1.0" 200 OK
cvat_server             | 2023-06-16 09:45:05,943 DEBG 'uvicorn-0' stdout output:
cvat_server             | INFO:     220.233.199.36:0 - "GET /api/tasks/302/annotations?org=DPE&use_default_location=true&filename=cvat2.zip&format=CVAT+for+video+1.1&action=download HTTP/1.0" 200 OK
cvat_server             |
cvat_server             | 2023-06-16 09:45:05,944 DEBG 'uvicorn-0' stderr output:
cvat_server             | ERROR:    Exception in ASGI application
cvat_server             | Traceback (most recent call last):
cvat_server             |   File "/opt/venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi
cvat_server             |     result = await app(  # type: ignore[func-returns-value]
cvat_server             |   File "/opt/venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
cvat_server             |     return await self.app(scope, receive, send)
cvat_server             |   File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/asgi.py", line 160, in __call__
cvat_server             |     await self.handle(scope, receive, send)
cvat_server             |   File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/asgi.py", line 190, in handle
cvat_server             |     await self.send_response(response, send)
cvat_server             |   File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/asgi.py", line 289, in send_response
cvat_server             |     await send(
cvat_server             |   File "/opt/venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 577, in send
cvat_server             |     raise RuntimeError("Response content shorter than Content-Length")
cvat_server             | RuntimeError: Response content shorter than Content-Length
cvat_server             |
cvat_server             | INFO:     220.233.199.36:0 - "POST /api/tasks/302/annotations?org=DPE&use_default_location=true&format=CVAT+1.1&filename=cvat2+(1).zip&conv_mask_to_poly=true HTTP/1.0" 202 Accepted
cvat_server             | 2023-06-16 09:45:14,336 DEBG 'uvicorn-1' stdout output:
cvat_server             | INFO:     220.233.199.36:0 - "POST /api/tasks/302/annotations?org=DPE&use_default_location=true&format=CVAT+1.1&filename=cvat2+(1).zip&conv_mask_to_poly=true HTTP/1.0" 202 Accepted
cvat_server             |
cvat_server             | 2023-06-16 09:45:14,412 DEBG 'uvicorn-0' stderr output:
cvat_server             | [2023-06-16 09:45:14,412] INFO cvat.server: The cleaning job 750b9d2c-3f70-4d55-ae7e-3f514a5fbf30 is queued.The check that the file /home/django/data/tasks/302/tmp/cvat2 (1).zip is deleted will be carried out after 2:00:00.
cvat_server             |
cvat_server             | 2023-06-16 09:45:14,412 DEBG 'uvicorn-0' stderr output:
cvat_server             | INFO:cvat.server:The cleaning job 750b9d2c-3f70-4d55-ae7e-3f514a5fbf30 is queued.The check that the file /home/django/data/tasks/302/tmp/cvat2 (1).zip is deleted will be carried out after 2:00:00.
cvat_server             |
cvat_server             | INFO:     220.233.199.36:0 - "POST /api/tasks/302/annotations/ HTTP/1.0" 201 Created
cvat_server             | 2023-06-16 09:45:14,414 DEBG 'uvicorn-0' stdout output:
cvat_server             | INFO:     220.233.199.36:0 - "POST /api/tasks/302/annotations/ HTTP/1.0" 201 Created
@bsekachev
Copy link
Member

@realtimshady1

Please, fill the issue template. What is CVAT version?

@bsekachev bsekachev added the need info Need more information to investigate the issue label Jun 7, 2023
@realtimshady1
Copy link
Author

I have updated the issue

@bsekachev
Copy link
Member

@realtimshady1

Template is lack of docker logs cvat_server. Without logs we can't even assume anything.

@realtimshady1
Copy link
Author

updated with docker logs

@bsekachev
Copy link
Member

@azhavoro

cvat_server             | 2023-06-16 09:45:05,944 DEBG 'uvicorn-0' stderr output:
cvat_server             | ERROR:    Exception in ASGI application
cvat_server             | Traceback (most recent call last):
cvat_server             |   File "/opt/venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 435, in run_asgi
cvat_server             |     result = await app(  # type: ignore[func-returns-value]
cvat_server             |   File "/opt/venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
cvat_server             |     return await self.app(scope, receive, send)
cvat_server             |   File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/asgi.py", line 160, in __call__
cvat_server             |     await self.handle(scope, receive, send)
cvat_server             |   File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/asgi.py", line 190, in handle
cvat_server             |     await self.send_response(response, send)
cvat_server             |   File "/opt/venv/lib/python3.10/site-packages/django/core/handlers/asgi.py", line 289, in send_response
cvat_server             |     await send(
cvat_server             |   File "/opt/venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 577, in send
cvat_server             |     raise RuntimeError("Response content shorter than Content-Length")
cvat_server             | RuntimeError: Response content shorter than Content-Length

@azhavoro
Copy link
Contributor

@realtimshady1 the same issue #6309, try updating to the latest develop branch or wait until v2.4.7 is released.

@bsekachev this is related to django-sendfile implementation, but doesn't affect file downloading, but I agree will be better to fix.

@realtimshady1
Copy link
Author

realtimshady1 commented Jun 20, 2023 •
edited
Loading

I tried importing using the API and I got this error in the console

Error: Request failed with status code 500. "Traceback (most recent call last):
  File "/opt/venv/lib/python3.10/site-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.10/site-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/engine/utils.py", line 255, in import_resource_with_clean_up_after
    result = func(filename, *args, **kwargs)
  File "/usr/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 863, in import_task_annotations
    task.import_annotations(f, importer, conv_mask_to_poly=conv_mask_to_poly)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 749, in import_annotations
    self.create(task_data.data.serialize())
  File "/home/django/cvat/apps/dataset_manager/task.py", line 698, in create
    self._patch_data(data, PatchAction.CREATE)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 688, in _patch_data
    self._merge_data(_data, jobs[jid]["start"], self.db_task.overlap, self.db_task.dimension)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 692, in _merge_data
    annotation_manager.merge(data, start_frame, overlap, dimension)
  File "/home/django/cvat/apps/dataset_manager/annotation.py", line 161, in merge
    tracks.merge(data.tracks, start_frame, overlap, dimension)
  File "/home/django/cvat/apps/dataset_manager/annotation.py", line 244, in merge
    cost_matrix[i][j] = 1 - self._calc_objects_similarity(
...
  File "/home/django/cvat/apps/dataset_manager/annotation.py", line 793, in interpolate
    is_same_type = shape0["type"] == shape1["type"]
KeyError: 'type'
"..

@Hegelim Hegelim mentioned this issue Jul 12, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
need info Need more information to investigate the issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@realtimshady1 @bsekachev @azhavoro