CVAT Api - Client doesn't return interpolated bounding boxes

[gboeer]

Actions before raising this issue

• I searched the existing issues and did not find anything similar.
• I read/searched the docs

Steps to Reproduce

I can use the API client just fine to access my local CVAT, create projects, download data etc.
However, when it comes to annotated tracks, I can't figure out a way to retrieve the annotations (bounding boxes) which have been interpolated.

I can retrieve the annotations for a task like so:

from cvat_sdk.core import make_client
client = make_client(host=url, credentials=(user, password))
task = client.tasks.retrieve(task_id)
annotations = task.get_annotations()

Afterward, the annotations.tracks contain all bounding boxes except the interpolated ones.
Or in other words, only the keyframes of each track are extracted.

How can I export the interpolated frames as well using the Api ?

Expected Behavior

The retrieved annotations should contain the interpolated bounding boxes as well, and not only the keyframes.

Possible Solution

No response

Context

No response

Environment

No response

[zhiltsov-max]

Hi, it is intended - CVAT returns tracks without interpolation, they are interpolated separately on the client and on the server for export. If you want to get interpolated annotations from the server, export the annotations from the project, task or job.

Examples:

• high-level SDK API 1, 2, (you need to call task.export_dataset())
• low-level SDK API 1
• CLI 1

[gboeer]

Hey, thanks for the info. I know that the interpolated boxes get exported, as you have stated.

Originally, I was trying to use the pytorch Dataset in the CVAT api to directly feed data into my training pipeline.
There again, I saw that the current Datasets do not support track annotations, so I was planning to extend them with this functionality.

I was almost done, meaning I can get the bounding boxes annotated as a track, until I stumbled upon this issue (not having access to the interpolated bounding boxes).

So I guess there is no way around writing my own Dataset which performs a task export and then reads data from disk.

[zhiltsov-max]

Please check if you can use dataset libraries, such as Datumaro or FiftyOne for reading exported annotations. If it's not possible in your case, you might need to parse the exported annotations manually, but the formats are typically simple enough (e.g. COCO, YOLO etc.). CVAT itself uses Datumaro (specifically, our fork) internally for dataset import and export.

[gboeer]

Thanks, I know my way around Fiftyone or Datumaro to handle datasets.
I was just looking forward to using the TaskVisionDataset from CVAT, however since I only work with track annotations there isn't a straightforward way to do so.

And since I saw the TODO about using tracks

cvat/cvat-sdk/cvat_sdk/datasets/task_dataset.py

Line 117 in 022f45d

# TODO: tracks?

I thought it may be possible to implement it myself. Indeed, only the part about the interpolated boxes is missing at my end, but if they aren't exposed over the API I don't think it will be possible to handle tracks with the current API.

[zhiltsov-max]

@gboeer, it would be great to see your contributions in improving this functionality. I think a simple implementation could download a dataset in the background, maybe if some option is enabled.

@SpecLad, please share your thoughts on supporting tracks in Dataset API of SDK.

[SpecLad]

I haven't really given this much thought so far. There is code in the server that can interpolate shapes, but I don't think it's exposed directly. I think that in order to support tracks in the SDK dataset layer, we'd have to provide an API endpoint (or perhaps a parameter to the existing endpoint) that returns tracks as sequences of interpolated shapes.