[MMSIG] Support badcase analyze in test #2584

Indigo6 · 2023-07-30T08:54:23Z

Motivation

Analyze badcases in test, supported functions:

Set a sample as a badcase based on loss/accuracy
Show / store visualization of badcases
Store bbox and loss/accuracy info in img name and results json
Also supported draw line with str based color in mmpose/visualization/opencv_backend_visualizer.py

Modification

BC-breaking (Optional)

Use cases (Optional)

Sample config file: configs\body_2d_keypoint\topdown_heatmap\mpii\td-hm_mobilenetv2_8xb64-210e_mpii-256x256_badcase.py

Checklist

Before PR:

I have read and followed the workflow indicated in the CONTRIBUTING.md to create this PR.
Pre-commit or linting tools indicated in CONTRIBUTING.md are used to fix the potential lint issues.
Bug fixes are covered by unit tests, the case that causes the bug should be added in the unit tests.
New functionalities are covered by complete unit tests. If not, please add more unit tests to ensure correctness.
The documentation has been modified accordingly, including docstring or example tutorials.

After PR:

CLA has been signed and all committers have signed the CLA in this PR.

codecov · 2023-07-30T09:10:58Z

Codecov Report

Patch coverage: 81.48% and project coverage change: +0.04% 🎉

Comparison is base (d57b679) 80.81% compared to head (5e400a0) 80.86%.
Report is 16 commits behind head on dev-1.x.

❗ Current head 5e400a0 differs from pull request most recent head 725c1a1. Consider uploading reports for the commit 725c1a1 to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##           dev-1.x    #2584      +/-   ##
===========================================
+ Coverage    80.81%   80.86%   +0.04%     
===========================================
  Files          230      231       +1     
  Lines        14457    14556      +99     
  Branches      2505     2517      +12     
===========================================
+ Hits         11684    11770      +86     
- Misses        2130     2133       +3     
- Partials       643      653      +10

Flag	Coverage Δ
unittests	`80.86% <81.48%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
mmpose/apis/inferencers/mmpose_inferencer.py	`83.56% <ø> (ø)`
mmpose/apis/inferencers/pose3d_inferencer.py	`82.72% <ø> (ø)`
mmpose/datasets/transforms/common_transforms.py	`84.80% <0.00%> (-0.56%)`	⬇️
mmpose/datasets/transforms/formatting.py	`79.76% <ø> (+1.85%)`	⬆️
mmpose/visualization/opencv_backend_visualizer.py	`41.34% <0.00%> (-0.82%)`	⬇️
mmpose/models/pose_estimators/bottomup.py	`70.00% <50.00%> (-0.69%)`	⬇️
mmpose/models/pose_estimators/topdown.py	`67.79% <50.00%> (-0.63%)`	⬇️
mmpose/visualization/local_visualizer.py	`63.40% <60.00%> (+1.09%)`	⬆️
mmpose/models/heads/hybrid_heads/vis_head.py	`83.15% <70.83%> (-4.94%)`	⬇️
mmpose/engine/hooks/badcase_hook.py	`91.66% <91.66%> (ø)`
... and 4 more

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Tau-J · 2023-08-01T05:45:54Z

...s/body_2d_keypoint/topdown_heatmap/mpii/td-hm_mobilenetv2_8xb64-210e_mpii-256x256_badcase.py

+                     badcase=dict(type="BadCaseAnalyzeHook", 
+                                #   metric_type="loss",
+                                  metric_type="accuracy",
+                                  show=True,
+                                  badcase_thr=100,
+                                  out_dir='badcase'))


How about moving badcase into default_runtime.py and disable by defaults? In this way, there is no need to add such a new config

Sounds fine to me. The usage samples can be added to the documentation.

Is the config of badcase in experiment config files able to overwrite that in default_runtime.py?

Tau-J · 2023-08-02T05:22:34Z

configs/_base_/default_runtime.py

+    badcase=dict(type="BadCaseAnalyzeHook", 
+                 metric_type="loss",
+                 badcase_thr=100,
+                 show=True,
+                 out_dir='badcase')


Suggested change

badcase=dict(type="BadCaseAnalyzeHook",

metric_type="loss",

badcase_thr=100,

show=True,

out_dir='badcase')

badcase=dict(type="BadCaseAnalyzeHook",

enable=False,

metric_type="loss",

badcase_thr=100,

show=True,

out_dir='badcase')

Should we set enable=False by defauts?

Yes, I forgot this.

Is it necessary since the default enable is False in __init__ method?

Tau-J · 2023-08-02T05:23:01Z

configs/_base_/default_runtime.py

+    badcase=dict(type="BadCaseAnalyzeHook", 
+                 metric_type="loss",
+                 badcase_thr=100,
+                 show=True,
+                 out_dir='badcase')


Should we set enable=False by defauts?

tools/test.py

mmpose/engine/hooks/badcase_hook.py

CLAassistant · 2023-08-03T07:54:08Z

All committers have signed the CLA.

Tau-J · 2023-08-03T11:38:39Z

mmpose/engine/hooks/badcase_hook.py

+                # draw gt keypoints in blue color
+                self._visualizer.kpt_color = 'blue'
+                self._visualizer.link_color = 'blue'


Suggested change

# draw gt keypoints in blue color

self._visualizer.kpt_color = 'blue'

self._visualizer.link_color = 'blue'

# draw gt keypoints in green color

self._visualizer.kpt_color = 'green'

self._visualizer.link_color = 'green'

Um …… I suggest using blue for easy distinction

mmpose/engine/hooks/badcase_hook.py

mmpose/visualization/opencv_backend_visualizer.py

Co-authored-by: Tau <taujiang@outlook.com>

* update * [Fix] Fix HRFormer log link * [Feature] Add Application 'Just dance' (#2528) * [Docs] Add advanced tutorial of implement new model. (#2539) * [Doc] Update img (#2541) * [Feature] Support MotionBERT (#2482) * [Fix] Fix demo scripts (#2542) * [Fix] Fix Pose3dInferencer keypoint shape bug (#2543) * [Enhance] Add notifications when saving visualization results (#2545) * [Fix] MotionBERT training and flip-test (#2548) * [Docs] Enhance docs (#2555) * [Docs] Fix links in doc (#2557) * [Docs] add details (#2558) * [Refactor] 3d human pose demo (#2554) * [Docs] Update MotionBERT docs (#2559) * [Refactor] Update the arguments of 3d inferencer to align with the demo script (#2561) * [Enhance] Combined dataset supports custom sampling ratio (#2562) * [Docs] Add MultiSourceSampler docs (#2563) * [Doc] Refine docs (#2564) * [Feature][MMSIG] Add UniFormer Pose Estimation to Projects folder (#2501) * [Fix] Check the compatibility of inferencer's input/output (#2567) * [Fix]Fix 3d visualization (#2565) * [Feature] Add bear example in just dance (#2568) * [Doc] Add example and openxlab link for just dance (#2571) * [Fix] Configs' paths of VideoPose3d (#2572) * [Docs] update docs (#2573) * [Fix] Fix new config bug in train.py (#2575) * [Fix] Configs' of MotionBERT (#2574) * [Enhance] Normalization option in 3d human pose demo and inferencer (#2576) * [Fix] Fix the incorrect labels for training vis_head with combined datasets (#2550) * [Enhance] Enhance 3dpose demo and docs (#2578) * [Docs] Enhance Codecs documents (#2580) * [Feature] Add DWPose distilled WholeBody RTMPose models (#2581) * [Docs] Add deployment docs (#2582) * [Fix] Refine 3dpose (#2583) * [Fix] Fix config typo in rtmpose-x (#2585) * [Fix] Fix 3d inferencer (#2593) * [Feature] Add a simple visualize api (#2596) * [Feature][MMSIG] Support badcase analyze in test (#2584) * [Fix] fix bug in flip_bbox with xyxy format (#2598) * [Feature] Support ubody dataset (2d keypoints) (#2588) * [Fix] Fix visualization bug in 3d pose (#2594) * [Fix] Remove use-multi-frames option (#2601) * [Enhance] Update demos (#2602) * [Enhance] wholebody support openpose style visualization (#2609) * [Docs] Documentation regarding 3d pose (#2599) * [CodeCamp2023-533] Migration Deepfashion topdown heatmap algorithms to 1.x (#2597) * [Fix] fix badcase hook (#2616) * [Fix] Update dataset mim downloading source to OpenXLab (#2614) * [Docs] Update docs structure (#2617) * [Docs] Refine Docs (#2619) * [Fix] Fix numpy error (#2626) * [Docs] Update error info and docs (#2624) * [Fix] Fix inferencer argument name (#2627) * [Fix] fix links for coco+aic hrnet (#2630) * [Fix] fix a bug when visualize keypoint indices (#2631) * [Docs] Update rtmpose docs (#2642) * [Docs] update README.md (#2647) * [Docs] Add onnx of RTMPose models (#2656) * [Docs] Fix mmengine link (#2655) * [Docs] Update QR code (#2653) * [Feature] Add DWPose (#2643) * [Refactor] Reorganize distillers (#2658) * [CodeCamp2023-259]Document Writing: Advanced Tutorial - Custom Data Augmentation (#2605) * [Docs] Fix installation docs(#2668) * [Fix] Fix expired links in README (#2673) * [Feature] Support multi-dataset evaluation (#2674) * [Refactor] Specify labels to pack in codecs (#2659) * [Refactor] update mapping tables (#2676) * [Fix] fix link (#2677) * [Enhance] Enable CocoMetric to get ann_file from MessageHub (#2678) * [Fix] fix vitpose pretrained ckpts (#2687) * [Refactor] Refactor YOLOX-Pose into mmpose core package (#2620) * [Fix] Fix typo in COCOMetric(#2691) * [Fix] Fix bug raised by changing bbox_center to input_center (#2693) * [Feature] Surpport EDPose for inference(#2688) * [Refactor] Internet for 3d hand pose estimation (#2632) * [Fix] Change test batch_size of edpose to 1 (#2701) * [Docs] Add OpenXLab Badge (#2698) * [Doc] fix inferencer doc (#2702) * [Docs] Refine dataset config tutorial (#2707) * [Fix] modify yoloxpose test settings (#2706) * [Fix] add compatibility for argument `return_datasample` (#2708) * [Feature] Support ubody3d dataset (#2699) * [Fix] Fix 3d inferencer (#2709) * [Fix] Move ubody3d dataset to wholebody3d (#2712) * [Refactor] Refactor config and dataset file structures (#2711) * [Fix] give more clues when loading img failed (#2714) * [Feature] Add demo script for 3d hand pose (#2710) * [Fix] Fix Internet demo (#2717) * [codecamp: mmpose-315] 300W-LP data set support (#2716) * [Fix] Fix the typo in YOLOX-Pose (#2719) * [Feature] Add detectors trained on humanart (#2724) * [Feature] Add RTMPose-Wholebody (#2721) * [Doc] Fix github action badge in README (#2727) * [Fix] Fix bug of dwpose (#2728) * [Feature] Support hand3d inferencer (#2729) * [Fix] Fix new config of RTMW (#2731) * [Fix] Align visualization color of 3d demo (#2734) * [Fix] Refine h36m data loading and add head_size to PackPoseInputs (#2735) * [Refactor] Align test accuracy for AE (#2737) * [Refactor] Separate evaluation mappings from KeypointConverter (#2738) * [Fix] MotionbertLabel codec (#2739) * [Fix] Fix mask shape (#2740) * [Feature] Add training datasets of RTMW (#2743) * [Doc] update RTMPose README (#2744) * [Fix] skip warnings in demo (#2746) * Bump 1.2 (#2748) * add comments in dekr configs (#2751) --------- Co-authored-by: Peng Lu <penglu2097@gmail.com> Co-authored-by: Yifan Lareina WU <mhsj16lareina@gmail.com> Co-authored-by: Xin Li <7219519+xin-li-67@users.noreply.github.com> Co-authored-by: Indigo6 <40358785+Indigo6@users.noreply.github.com> Co-authored-by: 谢昕辰 <xiexinch@outlook.com> Co-authored-by: tpoisonooo <khj.application@aliyun.com> Co-authored-by: zhengjie.xu <jerryxuzhengjie@gmail.com> Co-authored-by: Mesopotamia <54797851+yzd-v@users.noreply.github.com> Co-authored-by: chaodyna <li0331_1@163.com> Co-authored-by: lwttttt <85999869+lwttttt@users.noreply.github.com> Co-authored-by: Kanji Yomoda <Kanji.yy@gmail.com> Co-authored-by: LiuYi-Up <73060646+LiuYi-Up@users.noreply.github.com> Co-authored-by: ZhaoQiiii <102809799+ZhaoQiiii@users.noreply.github.com> Co-authored-by: Yang-ChangHui <71805205+Yang-Changhui@users.noreply.github.com> Co-authored-by: Xuan Ju <89566272+juxuan27@users.noreply.github.com>

Indigo6 added 10 commits July 26, 2023 20:16

add badcase hook

cc5f248

add loss based badcase analyze

cc9e29e

support accurayc based badcase analyze

c621837

fix configdict bug

e1eadcf

revert cfg

3496be9

add badcase analyze sample cfg

bfd1707

support draw_line with str value color

c658f87

add unit test for badcase hook

5773b69

use str based color

8e29729

rm useless codes and add warnings

4cfc7ab

mm-assistant bot assigned Tau-J Jul 30, 2023

Tau-J reviewed Aug 1, 2023

View reviewed changes

move badcase hook config to default_runtime.py

54eecc3

Tau-J reviewed Aug 2, 2023

View reviewed changes

Tau-J reviewed Aug 3, 2023

View reviewed changes

mmpose/engine/hooks/badcase_hook.py Outdated Show resolved Hide resolved

Indigo6 force-pushed the badcase branch 2 times, most recently from de2a45e to 54eecc3 Compare August 3, 2023 08:13

Indigo6 added 6 commits August 3, 2023 16:23

rename badcase hook and fix linting

271d105

set and sort default cfg of badcase

ac462c7

update badcase or pred show logic

7595b4f

fix linting in test_badcase_hook.py

2d68e40

fix rename bug

371a01f

fix rename bug

34aca1f

Tau-J reviewed Aug 3, 2023

View reviewed changes

mmpose/engine/hooks/badcase_hook.py Show resolved Hide resolved

Tau-J reviewed Aug 3, 2023

View reviewed changes

mmpose/engine/hooks/badcase_hook.py Outdated Show resolved Hide resolved

Tau-J reviewed Aug 3, 2023

View reviewed changes

mmpose/visualization/opencv_backend_visualizer.py Outdated Show resolved Hide resolved

Indigo6 and others added 4 commits August 4, 2023 09:01

Update mmpose/engine/hooks/badcase_hook.py

af3f475

Co-authored-by: Tau <taujiang@outlook.com>

Update mmpose/engine/hooks/badcase_hook.py

8607f12

Co-authored-by: Tau <taujiang@outlook.com>

fix linting

6149d5a

bgr2rgb after mmcv.color_val

725c1a1

Tau-J approved these changes Aug 4, 2023

View reviewed changes

Tau-J merged commit 55bb3e8 into open-mmlab:dev-1.x Aug 4, 2023
5 of 9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MMSIG] Support badcase analyze in test #2584

[MMSIG] Support badcase analyze in test #2584

Indigo6 commented Jul 30, 2023 •

edited

Loading

codecov bot commented Jul 30, 2023 •

edited

Loading

Tau-J Aug 1, 2023

Indigo6 Aug 1, 2023

Indigo6 Aug 3, 2023

Tau-J Aug 2, 2023

Tau-J Aug 2, 2023

Indigo6 Aug 2, 2023

Indigo6 Aug 3, 2023

Tau-J Aug 2, 2023

CLAassistant commented Aug 3, 2023 •

edited

Loading

Tau-J Aug 3, 2023

Indigo6 Aug 4, 2023

[MMSIG] Support badcase analyze in test #2584

[MMSIG] Support badcase analyze in test #2584

Conversation

Indigo6 commented Jul 30, 2023 • edited Loading

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

codecov bot commented Jul 30, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CLAassistant commented Aug 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Indigo6 commented Jul 30, 2023 •

edited

Loading

codecov bot commented Jul 30, 2023 •

edited

Loading

CLAassistant commented Aug 3, 2023 •

edited

Loading