Deploy skeleton model on CVAT for auto and semi annotations

[OnceUponATimeMathley]

My actions before raising this issue

• Read/searched the docs
• Searched past issues

Expected Behaviour

Current Behaviour

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

• Git hash commit (git log -1):
• Docker version docker version (e.g. Docker 17.0.05):
• Are you using Docker Swarm or Kubernetes?
• Operating System and version (e.g. Linux, Windows, MacOS):
• Code example or link to GitHub repo or gist to reproduce problem:
• Other diagnostic information / logs:
  Logs from `cvat` container

I try to write code for http response for skeleton type annotations on CVAT for deploy deep learning model but get the Could not create annotations
TypeError: Cannot read properties of null (reading 'every'). I also check the annotations.json file in cvat/apps/dataset_manager/tests/assets/annotations.json folder and the attributes is same, open F12 and check response has the same. Can everyone help me. Thanks

[OnceUponATimeMathley]

[bsekachev]

Hi @OnceUponATimeMathley

Probably client code is not prepared to expect skeletons from serverless response because we didn't have such models in the past. Are you going to contribute the model to open source?

[OnceUponATimeMathley]

I create a simple skeleton template and use Draw new skeleton to draw it. I use F12 and press save, so I can have a response and check that json response. I try writing code similarity rectangle and polygon type based on this json. But I get: Cannot read properties of null (reading 'every') ERROR.

I think elements attribute is null because I find some function call with every (image below).
I wonder if there is the documents so I can check the JSON format I need. I recently use CVAT tool and feel the serverless function is cool and try to write some code to test the skeleton type. Thanks.

![Screenshot from 2022-10-27 22-30-39](https://user-images.githubusercontent.com/525773

59/198333221-a074b76e-8f47-4dd2-9e37-9d6dcd4f3842.png)

[bsekachev]

I am not sure I understand you correctly, but SkeletonShape is expected to have the following structure
You can see this type definition in the source code

    id?: number; // optional, only when get the shape from the server
    clientID?: number; // optional, may be null if the object has been just created and is being added to the collection
    label_id: number;
    group: number;
    frame: number;
    source: 'manual' | 'auto';
    attributes: { spec_id: number; value: string }[];
    elements: {
        id?: number;
        attributes: { spec_id: number; value: string }[];
        label_id: number;
        occluded: boolean;
        outside: boolean;
        points: number[];
        type: ShapeType;
    }[];
    occluded: boolean;
    outside: boolean;
    rotation: number;
    z_order: number;
    type: ShapeType;

[OnceUponATimeMathley]

I am not sure I understand you correctly, but SkeletonShape is expected to have the following structure You can see this type definition in the source code

    id?: number; // optional, only when get the shape from the server
    clientID?: number; // optional, may be null if the object has been just created and is being added to the collection
    label_id: number;
    group: number;
    frame: number;
    source: 'manual' | 'auto';
    attributes: { spec_id: number; value: string }[];
    elements: {
        id?: number;
        attributes: { spec_id: number; value: string }[];
        label_id: number;
        occluded: boolean;
        outside: boolean;
        points: number[];
        type: ShapeType;
    }[];
    occluded: boolean;
    outside: boolean;
    rotation: number;
    z_order: number;
    type: ShapeType;

Thanks, In my example, I create a simple skeleton template, it has two points, I have some questions:

First, When I use Draw new skeleton, it contain two points with a bounding box, when I try log the annotations object in cvat-ui/src/components/annotation-page/canvas/canvas-wrapper.tsx folder (you can see in jsonformatter.txt file). I can see the points attributes with 4 point of bounding box. Can I just only need to return the SkeletonShape format.
jsonformatter.txt

Second, I try to write serverless function in function.yaml and main.py. In function.yaml, the annotations.spec define the label,
Is it the same label defined in CVAT tool, Do I need to define svg template for skeleton label?
Thanks so much. I am not good at English.

[bsekachev]

@OnceUponATimeMathley

• Skeleton actually does not have a bounding box. It consists only from two points in your case. Bounding box exists only on user-level for better visualization. Points that you can see in ObjectState.points are simply skeleton elements coordinates, combined into one array (actually just to generalization). You do not need to specify them when you create a new skeleton, just leave undefined value. You can see how we construct ObjectState from raw canvas input here: https://github.com/opencv/cvat/blob/80c72340f89397577de978b278dd408cf2d73fa1/cvat-ui/src/components/annotation-page/canvas/canvas-wrapper.tsx#L399 You can just set a breakpoint to see state after a skeleton is drawn.

Do I need to define svg template for skeleton label?

• That is not obligatory, I believe. Intiailly we can rely on elements the serverless function returns. If the number of elements is equal to the number of nodes in skeleton and all the element's labels are correct, we, obviously can create a skeleton.

Just example. If we have in CVAT skeleton body with elements right_leg and left_leg. I expect serverless output be like:

     type: "skeleton",
     label: "body",
     elements: [{ type: "points", label: "right_leg", points: [x, y], ... }, { type: "points", label: "left_leg", points: [x, y], ... }]
     ....

I do not see reasons to have additional specs, at least for now..

[bsekachev]

For sure, to interpret the serverless response correctly when running detectors online, you need to modify client code here: cvat-ui/src/components/annotation-page/standard-workspace/controls-side-bar/tools-control.tsx

[nmanovic]

@OnceUponATimeMathley , please don't hesitate to contact us if you decide to contribute. We will try to help. I will close the issue for now. I don't know how we can help here.