Skip to content

Commit

Permalink
Added skeleton keypoints detector COCO Whole Body (#7033)
Browse files Browse the repository at this point in the history
### Motivation and context
Resolved #3756
Resolved #5324

Used model is UBody via mmpose
https://mmpose.readthedocs.io/en/latest/model_zoo/wholebody_2d_keypoint.html#topdown-heatmap-hrnet-ubody-coco-wholebody-on-ubody2d

Optional: 
- [ ] Try different detectors from mmdetect
- [ ] GPU support (it very quick on CPU as well)

**Deploy with**: ```./deploy_cpu.sh pytorch/mmpose/ubody2d/nuclio/```

**Recommendations**: redeploy this list of functions if you use any of
them after using this code
 
- tensorflow/faster_rcnn_inception_v2_coco/nuclio/
- tensorflow/matterport/mask_rcnn/nuclio/
- pytorch/facebookresearch/detectron2/retinanet_r101/nuclio/
- openvino/omz/public/yolo-v3-tf/nuclio/
- onnx/WongKinYiu/yolov7/nuclio/
- openvino/omz/public/mask_rcnn_inception_resnet_v2_atrous_coco/nuclio/
-
openvino/omz/public/faster_rcnn_inception_resnet_v2_atrous_coco/nuclio/
- openvino/omz/intel/text-detection-0004/nuclio/
- openvino/omz/intel/semantic-segmentation-adas-0001/nuclio/
- openvino/omz/intel/face-detection-0205/nuclio/
  • Loading branch information
bsekachev authored Nov 6, 2023
1 parent 506c96b commit 630770e
Show file tree
Hide file tree
Showing 42 changed files with 2,494 additions and 1,525 deletions.
6 changes: 6 additions & 0 deletions changelog.d/20231024_123046_boris_keypoints.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
### Added

- CVAT now supports serverless Nuclio functions returning skeleton annotations.
Added keypoint detector supporting skeletons with following classes: body, head, foot, hands.
Deployment command: `./deploy_cpu.sh pytorch/mmpose/hrnet32/nuclio/`
(<https://github.com/opencv/cvat/pull/7033>)
15 changes: 11 additions & 4 deletions cvat-core/src/core-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
//
// SPDX-License-Identifier: MIT

import { ModelKind, ModelReturnType } from './enums';
import { ModelKind, ModelReturnType, ShapeType } from './enums';

export interface ModelAttribute {
name: string;
Expand All @@ -19,17 +19,24 @@ export interface ModelParams {
};
}

export interface ModelTip {
export interface MLModelTip {
message: string;
gif: string;
}

export interface MLModelLabel {
name: string;
type: ShapeType | 'unknown';
attributes: ModelAttribute[];
sublabels?: MLModelLabel[];
svg?: string,
}

export interface SerializedModel {
id?: string | number;
name?: string;
labels?: string[];
labels_v2?: MLModelLabel[];
version?: number;
attributes?: Record<string, ModelAttribute>;
framework?: string;
description?: string;
kind?: ModelKind;
Expand Down
10 changes: 1 addition & 9 deletions cvat-core/src/lambda-manager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,15 +41,7 @@ class LambdaManager {
const { results: functions, count: functionsCount } = functionsResult;

const result = [...lambdaFunctions, ...functions];
const models = [];

for (const model of result) {
models.push(
new MLModel({
...model,
}),
);
}
const models = result.map((serialzedModel) => new MLModel({ ...serialzedModel }));

this.cachedList = models;
return { models, count: lambdaFunctions.length + functionsCount };
Expand Down
18 changes: 8 additions & 10 deletions cvat-core/src/ml-model.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,11 @@
import serverProxy from './server-proxy';
import PluginRegistry from './plugins';
import { decodePreview } from './frames';
import { ModelProviders, ModelKind, ModelReturnType } from './enums';
import {
SerializedModel, ModelAttribute, ModelParams, ModelTip,
ModelProviders, ModelKind, ModelReturnType,
} from './enums';
import {
SerializedModel, ModelParams, MLModelTip, MLModelLabel,
} from './core-types';

export default class MLModel {
Expand All @@ -27,18 +29,14 @@ export default class MLModel {
return this.serialized.name;
}

public get labels(): string[] {
return Array.isArray(this.serialized.labels) ? [...this.serialized.labels] : [];
public get labels(): MLModelLabel[] {
return Array.isArray(this.serialized.labels_v2) ? [...this.serialized.labels_v2] : [];
}

public get version(): number {
return this.serialized.version;
}

public get attributes(): Record<string, ModelAttribute> {
return this.serialized.attributes || {};
}

public get framework(): string {
return this.serialized.framework;
}
Expand Down Expand Up @@ -67,7 +65,7 @@ export default class MLModel {
return result;
}

public get tip(): ModelTip {
public get tip(): MLModelTip {
return {
message: this.serialized.help_message,
gif: this.serialized.animated_gif,
Expand Down Expand Up @@ -146,7 +144,7 @@ Object.defineProperties(MLModel.prototype.delete, {
enumerable: false,
value: async function implementation(this: MLModel): Promise<void> {
if (this.isDeletable) {
await serverProxy.functions.delete(this.id);
await serverProxy.functions.delete(this.id as number);
}
},
},
Expand Down
Loading

0 comments on commit 630770e

Please sign in to comment.