[Fix] Bug in post dark udp (open-mmlab#1079)

* update fap (open-mmlab#1070) * fix a bug in post_dark_udp * fix a bug in post_dark_udp * Modified a bad comment * add unit test for udp * add unit test for udp * use numpy instead math for math ops * fix lint Co-authored-by: Jas <jinsheng@sensetime.com> Co-authored-by: quyang <quyang@qiyi.com> Co-authored-by: ly015 <liyining0712@gmail.com>
wjkim81 · Jan 5, 2022 · 5fcb34a · 5fcb34a
1 parent c06316e
commit 5fcb34a
Show file tree

Hide file tree

Showing 3 changed files with 51 additions and 5 deletions.
diff --git a/docs/en/faq.md b/docs/en/faq.md
@@ -35,6 +35,17 @@ If the contents here do not cover your issue, please create an issue using the [
 
 ## Data
 
+- **How to convert my 2d keypoint dataset to coco-type?**
+
+  You may refer to this conversion [tool](https://github.com/open-mmlab/mmpose/blob/master/tools/dataset/parse_macaquepose_dataset.py) to prepare your data.
+  Here is an [example](https://github.com/open-mmlab/mmpose/blob/master/tests/data/macaque/test_macaque.json) of the coco-type json.
+  In the coco-type json, we need "categories", "annotations" and "images". "categories" contain some basic information of the dataset, e.g. class name and keypoint names.
+  "images" contain image-level information. We need "id", "file_name", "height", "width". Others are optional.
+  Note: (1) It is okay that "id"s are not continuous or not sorted (e.g. 1000, 40, 352, 333 ...).
+
+  "annotations" contain instance-level information. We need "image_id", "id", "keypoints", "num_keypoints", "bbox", "iscrowd", "area", "category_id". Others are optional.
+  Note: (1) "num_keypoints" means the number of visible keypoints. (2) By default, please set "iscrowd: 0". (3) "area" can be calculated using the bbox (area = w * h) (4) Simply set "category_id: 1". (5) The "image_id" in "annotations" should match the "id" in "images".
+
 - **What if my custom dataset does not have bounding box label?**
 
   We can estimate the bounding box of a person as the minimal box that tightly bounds all the keypoints.
@@ -93,6 +104,9 @@ If the contents here do not cover your issue, please create an issue using the [
   Since we do not have the ground-truth for test dataset, we cannot evaluate it 'locally'.
   If you would like to evaluate the performance on test set, you have to upload the pred.mat (which is generated during testing) to the official server via email, according to [the MPII guideline](http://human-pose.mpi-inf.mpg.de/#evaluation).
 
+- **For top-down 2d pose estimation, why predicted joint coordinates can be out of the bounding box (bbox)?**
+  We do not directly use the bbox to crop the image. bbox will be first transformed to center & scale, and the scale will be multiplied by a factor (1.25) to include some context. If the ratio of width/height is different from that of model input (possibly 192/256), we will adjust the bbox.
+
 ## Inference
 
 - **How to run mmpose on CPU?**

diff --git a/mmpose/core/evaluation/top_down_eval.py b/mmpose/core/evaluation/top_down_eval.py
@@ -366,8 +366,29 @@ def post_dark_udp(coords, batch_heatmaps, kernel=3):
     np.log(batch_heatmaps, batch_heatmaps)
     batch_heatmaps = np.transpose(batch_heatmaps,
                                   (2, 3, 0, 1)).reshape(H, W, -1)
-    batch_heatmaps_pad = cv2.copyMakeBorder(
-        batch_heatmaps, 1, 1, 1, 1, borderType=cv2.BORDER_REFLECT)
+
+    # cv2.copyMakeBorder will report an error when input dimension exceeds 512
+    batch_heatmaps_channel = batch_heatmaps.shape[2]
+    if batch_heatmaps_channel > 512:
+        total_group_number = int(np.ceil(batch_heatmaps_channel / 512))
+        splited_batch_heatmaps = []
+        for group_idx in range(total_group_number):
+            splited_batch_heatmap = batch_heatmaps[
+                ..., group_idx *
+                512:min(batch_heatmaps_channel, (group_idx + 1) * 512)]
+            batch_heatmap_pad = cv2.copyMakeBorder(
+                splited_batch_heatmap,
+                1,
+                1,
+                1,
+                1,
+                borderType=cv2.BORDER_REFLECT)
+            splited_batch_heatmaps.append(batch_heatmap_pad)
+        batch_heatmaps_pad = np.concatenate(splited_batch_heatmaps, axis=2)
+    else:
+        batch_heatmaps_pad = cv2.copyMakeBorder(
+            batch_heatmaps, 1, 1, 1, 1, borderType=cv2.BORDER_REFLECT)
+
     batch_heatmaps_pad = np.transpose(
         batch_heatmaps_pad.reshape(H + 2, W + 2, B, K),
         (2, 3, 0, 1)).flatten()

diff --git a/tests/test_evaluation/test_top_down_eval.py b/tests/test_evaluation/test_top_down_eval.py
@@ -32,6 +32,11 @@ def test_keypoints_from_heatmaps():
     center = np.array([[127, 127]])
     scale = np.array([[64 / 200.0, 64 / 200.0]])
 
+    udp_heatmaps = np.ones((32, 17, 64, 64), dtype=np.float32)
+    udp_heatmaps[:, :, 31, 31] = 2
+    udp_center = np.tile([127, 127], (32, 1))
+    udp_scale = np.tile([32, 32], (32, 1))
+
     preds, maxvals = keypoints_from_heatmaps(heatmaps, center, scale)
 
     assert_array_almost_equal(preds, np.array([[[126, 126]]]), decimal=4)
@@ -51,10 +56,16 @@ def test_keypoints_from_heatmaps():
     assert isinstance(preds, np.ndarray)
     assert isinstance(maxvals, np.ndarray)
 
+    # test for udp dimension problem
     preds, maxvals = keypoints_from_heatmaps(
-        heatmaps, center, scale, post_process='unbiased')
-    assert_array_almost_equal(preds, np.array([[[126, 126]]]), decimal=4)
-    assert_array_almost_equal(maxvals, np.array([[[2]]]), decimal=4)
+        udp_heatmaps,
+        udp_center,
+        udp_scale,
+        post_process='default',
+        target_type='GaussianHeatMap',
+        use_udp=True)
+    assert_array_almost_equal(preds, np.tile([76, 76], (32, 17, 1)), decimal=0)
+    assert_array_almost_equal(maxvals, np.tile([2], (32, 17, 1)), decimal=4)
     assert isinstance(preds, np.ndarray)
     assert isinstance(maxvals, np.ndarray)