[Feature] Support UAV123 Dataset in SOT

configs/sot/siamese_rpn/README.md

@@ -14,12 +14,26 @@
 }
 ```
 
-## Results and models on LaSOT dataset
+## Results and models
 
-We observe around 1.0 points fluctuations in Success and 1.5 points fluctuations in Norm percision. We provide the best model.
+### LaSOT
 
-Note that all of checkpoints from 11-th to 20-th epoch need to be evaluated in order to achieve the best results.
+Note that the checkpoints from 10-th to 20-th epoch will be evaluated during training. You can find the best checkpoint from the log file.
+
+We observe around 1.0 points fluctuations in Success and 1.5 points fluctuations in Norm percision. We provide the best model with its configuration and training log.
 
 |    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | Success | Norm precision | Config | Download |
 | :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :----: | :------: | :--------: |
 |    R-50    |  -  |   20e    | 7.54        | 50.0              | 49.9 | 57.9 | [config](siamese_rpn_r50_1x_lasot.py) | [model](https://download.openmmlab.com/mmtracking/sot/siamese_rpn/siamese_rpn_r50_1x_lasot/siamese_rpn_r50_1x_lasot_20201218_051019-3c522eff.pth) | [log](https://download.openmmlab.com/mmtracking/sot/siamese_rpn/siamese_rpn_r50_1x_lasot/siamese_rpn_r50_1x_lasot_20201218_051019.log.json) |
+
+### UAV123
+
+The checkpoints from 10-th to 20-th epoch will be evaluated during training.
+
+After training, you need to pick up the best checkpoint from the log file, then use the best checkpoint to search the hyperparameters on UAV123 following [here](https://github.com/open-mmlab/mmtracking/blob/master/docs/useful_tools_scripts.md#siameserpn-test-time-parameter-search) to achieve the best results.
+
+We observe around xxx points fluctuations in Success and xxx points fluctuations in Norm percision. We provide the best model with its configuration and training log.
+
+|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | Success | Norm precision | Config | Download |
+| :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :----: | :------: | :--------: |
+|    R-50    |  -  |   20e    | 7.54     | -              | 61.8 | 77.3 | [config](siamese_rpn_r50_1x_uav123.py) | [model](https://download.openmmlab.com/mmtracking/sot/siamese_rpn/siamese_rpn_r50_1x_lasot/siamese_rpn_r50_1x_lasot_20201218_051019-3c522eff.pth) | [log](https://download.openmmlab.com/mmtracking/sot/siamese_rpn/siamese_rpn_r50_1x_lasot/siamese_rpn_r50_1x_lasot_20201218_051019.log.json) |

configs/sot/siamese_rpn/siamese_rpn_r50_1x_uav123.py

@@ -0,0 +1,17 @@
+_base_ = ['./siamese_rpn_r50_1x_lasot.py']
+
+# model settings
+model = dict(
+    test_cfg=dict(rpn=dict(penalty_k=0.01, window_influence=0.02, lr=0.46)))
+
+data_root = 'data/'
+# dataset settings
+data = dict(
+    val=dict(
+        type='UAV123Dataset',
+        ann_file=data_root + 'UAV123/annotations/uav123.json',
+        img_prefix=data_root + 'UAV123/data_seq/UAV123'),
+    test=dict(
+        type='UAV123Dataset',
+        ann_file=data_root + 'UAV123/annotations/uav123.json',
+        img_prefix=data_root + 'UAV123/data_seq/UAV123'))

docs/dataset.md

@@ -8,6 +8,7 @@ This page provides the instructions for dataset preparation on existing benchmar
   - [MOT Challenge](https://motchallenge.net/)
 - Single Object Tracking
   - [LaSOT](http://vision.cs.stonybrook.edu/~lasot/)
+  - [UAV123](https://cemse.kaust.edu.sa/ivul/uav123/)
 
 ### 1. Download Datasets
 
@@ -21,7 +22,7 @@ Notes:
 
 - For the training and testing of multi object tracking task, only one of the MOT Challenge dataset (e.g. MOT17) is needed.
 
-- For the training and testing of single object tracking task, the MSCOCO, ILSVRC and LaSOT datasets are needed.
+- For the training and testing of single object tracking task, the MSCOCO, ILSVRC, LaSOT and UAV123 datasets are needed.
 
 ```
 mmtracking
@@ -62,6 +63,14 @@ mmtracking
 |   ├── MOT15/MOT16/MOT17/MOT20
 |   |   ├── train
 |   |   ├── test
+│   │
+│   ├── UAV123
+│   │   ├── data_seq
+│   │   │   ├── UAV123
+│   │   │   │   ├── bike1
+│   │   │   │   ├── boat1
+│   │   ├── anno
+│   │   │   ├── UAV123
 ```
 
 ### 2. Convert Annotations
@@ -83,6 +92,9 @@ python ./tools/convert_datasets/lasot2coco.py -i ./data/lasot/LaSOTTesting -o ./
 # The processing of other MOT Challenge dataset is the same as MOT17
 python ./tools/convert_datasets/mot2coco.py -i ./data/MOT17/ -o ./data/MOT17/annotations --split-train --convert-det
 python ./tools/convert_datasets/mot2reid.py -i ./data/MOT17/ -o ./data/MOT17/reid --val-split 0.2 --vis-threshold 0.3
+
+# UAV123
+python ./tools/convert_datasets/uav2coco.py -i ./data/UAV123/ -o ./data/UAV123/annotations
 ```
 
 The folder structure will be as following after your run these scripts:
@@ -132,6 +144,15 @@ mmtracking
 |   |   ├── reid
 │   │   │   ├── imgs
 │   │   │   ├── meta
+│   │
+│   ├── UAV123
+│   │   ├── data_seq
+│   │   │   ├── UAV123
+│   │   │   │   ├── bike1
+│   │   │   │   ├── boat1
+│   │   ├── anno (the offical annotation files)
+│   │   │   ├── UAV123
+│   │   ├── annotations (the converted annotation file)
 ```
 
 #### The folder of annotations in ILSVRC
@@ -200,3 +221,9 @@ MOT17-02-FRCNN_000009/000081.jpg 3
 For validation, The annotation list `val_20.txt` remains the same as format above.
 
 Images in `reid/imgs` are cropped from raw images in `MOT17/train` by the corresponding `gt.txt`. The value of ground-truth labels should fall in range `[0, num_classes - 1]`.
+
+#### The folder of annotations in UAV123
+
+There are only 1 json files in `data/UAV123/annotations`:
+
+`uav123.json`:  Json file containing the annotations information of the UAV123 dataset.

docs/useful_tools_scripts.md

@@ -111,6 +111,18 @@ python tools/publish_model.py work_dirs/dff_faster_rcnn_r101_dc5_1x_imagenetvid/
 
 The final output filename will be `dff_faster_rcnn_r101_dc5_1x_imagenetvid_20201230-{hash id}.pth`.
 
+## SiameseRPN++ Test-time Parameter Search
+
+`tools/sot_siamrpn_param_search.py` can search the test-time tracking parameters in SiameseRPN++: `penalty_k`, `lr` and `window_influence`. You need to pass the searching range of each parameter into the argparser.
+
+Example on UAV123 dataset:
+
+```shell
+./tools/dist_sot_siamrpn_param_search.sh [${CONFIG_FILE}] [$GPUS] \
+[--checkpoint ${CHECKPOINT}] [--log ${LOG_FILENAME}] [--eval ${EVAL}] \
+[--penalty-k-range 0.05,0.5,0.05] [--lr-range 0.3,0.45,0.02] [--win-infu-range 0.46,0.55,0.02]
+```
+
 ## Miscellaneous
 
 ### Print the entire config

docs_zh-CN/dataset.md

@@ -8,6 +8,7 @@
   - [MOT Challenge](https://motchallenge.net/)
 - 单目标跟踪
   - [LaSOT](http://vision.cs.stonybrook.edu/~lasot/)
+  - [UAV123](https://cemse.kaust.edu.sa/ivul/uav123/)
 
 ### 1. 下载数据集
 
@@ -21,7 +22,7 @@
 
 - 对于多目标跟踪任务的训练和测试，只需要 MOT Challenge 中的任意一个数据集（比如 MOT17）。
 
-- 对于单目标跟踪任务的训练和测试，需要 MSCOCO，ILSVRC 和 LaSOT 数据集。
+- 对于单目标跟踪任务的训练和测试，需要 MSCOCO，ILSVRC, LaSOT 和 UAV123 数据集。
 
 ```
 mmtracking
@@ -62,6 +63,14 @@ mmtracking
 |   ├── MOT15/MOT16/MOT17/MOT20
 |   |   ├── train
 |   |   ├── test
+│   │
+│   ├── UAV123
+│   │   ├── data_seq
+│   │   │   ├── UAV123
+│   │   │   │   ├── bike1
+│   │   │   │   ├── boat1
+│   │   ├── anno
+│   │   │   ├── UAV123
 ```
 
 ### 2. 转换标注格式
@@ -84,6 +93,9 @@ python ./tools/convert_datasets/lasot2coco.py -i ./data/lasot/LaSOTTesting -o ./
 # MOT Challenge中其余数据集的处理与MOT17相同
 python ./tools/convert_datasets/mot2coco.py -i ./data/MOT17/ -o ./data/MOT17/annotations --split-train --convert-det
 python ./tools/convert_datasets/mot2reid.py -i ./data/MOT17/ -o ./data/MOT17/reid --val-split 0.2 --vis-threshold 0.3
+
+# UAV123
+python ./tools/convert_datasets/uav2coco.py -i ./data/UAV123/ -o ./data/UAV123/annotations
 ```
 
 完成以上格式转换后，文件目录结构如下：
@@ -133,6 +145,15 @@ mmtracking
 |   |   ├── reid
 │   │   │   ├── imgs
 │   │   │   ├── meta
+│   │
+│   ├── UAV123
+│   │   ├── data_seq
+│   │   │   ├── UAV123
+│   │   │   │   ├── bike1
+│   │   │   │   ├── boat1
+│   │   ├── anno (the offical annotation files)
+│   │   │   ├── UAV123
+│   │   ├── annotations (the converted annotation file)
 ```
 
 #### ILSVRC的标注文件夹
@@ -202,3 +223,9 @@ MOT17-02-FRCNN_000009/000081.jpg 3
 验证集标注 `val_20.txt` 的结构和上面类似。
 
 `reid/imgs` 中的图片是从 `MOT17/train` 中原始图片根据对应的 `gt.txt` 裁剪得到。真实类别标签值在 `[0, num_classes - 1]` 范围内。
+
+#### UAV123的标注文件夹
+
+在 `data/UAV123/annotations` 中只有一个 json 文件:
+
+`uav123.json`: 包含 UAV123 数据集标注信息的 json 文件。

docs_zh-CN/useful_tools_scripts.md

@@ -111,6 +111,18 @@ python tools/publish_model.py work_dirs/dff_faster_rcnn_r101_dc5_1x_imagenetvid/
 
 最后输出的文件名为 `dff_faster_rcnn_r101_dc5_1x_imagenetvid_20201230-{hash id}.pth`。
 
+## SiameseRPN++ 测试参数搜索
+
+`tools/sot_siamrpn_param_search.py` 用来搜索 SiameseRPN++ 测试时的跟踪相关参数： `penalty_k`, `lr` 和 `window_influence`。你需要在参数解析器中传入前面每个参数的搜索范围。
+
+在 UAV123 上的超参搜索范例：
+
+```shell
+./tools/dist_sot_siamrpn_param_search.sh [${CONFIG_FILE}] [$GPUS] \
+[--checkpoint ${CHECKPOINT}] [--log ${LOG_FILENAME}] [--eval ${EVAL}] \
+[--penalty-k-range 0.05,0.5,0.05] [--lr-range 0.3,0.45,0.02] [--win-infu-range 0.46,0.55,0.02]
+```
+
 ## 其它有用的工具脚本
 
 ### 输出完整的配置

mmtrack/datasets/__init__.py

@@ -9,10 +9,13 @@
 from .parsers import CocoVID
 from .pipelines import PIPELINES
 from .reid_dataset import ReIDDataset
+from .sot_test_dataset import SOTTestDataset
 from .sot_train_dataset import SOTTrainDataset
+from .uav123_dataset import UAV123Dataset
 
 __all__ = [
     'DATASETS', 'PIPELINES', 'build_dataloader', 'build_dataset', 'CocoVID',
     'CocoVideoDataset', 'ImagenetVIDDataset', 'MOTChallengeDataset',
-    'LaSOTDataset', 'SOTTrainDataset', 'ReIDDataset'
+    'ReIDDataset', 'SOTTrainDataset', 'SOTTestDataset', 'LaSOTDataset',
+    'UAV123Dataset'
 ]

mmtrack/datasets/lasot_dataset.py

@@ -1,24 +1,17 @@
 # Copyright (c) OpenMMLab. All rights reserved.
 import numpy as np
-from mmcv.utils import print_log
 from mmdet.datasets import DATASETS
 
-from mmtrack.core.evaluation import eval_sot_ope
-from .coco_video_dataset import CocoVideoDataset
+from .sot_test_dataset import SOTTestDataset
 
 
 @DATASETS.register_module()
-class LaSOTDataset(CocoVideoDataset):
+class LaSOTDataset(SOTTestDataset):
     """LaSOT dataset for the testing of single object tracking.
 
     The dataset doesn't support training mode.
     """
 
-    CLASSES = (0, )
-
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
-
     def _parse_ann_info(self, img_info, ann_info):
         """Parse bbox annotations.
 
@@ -39,60 +32,3 @@ def _parse_ann_info(self, img_info, ann_info):
         ignore = ann_info[0]['full_occlusion'] or ann_info[0]['out_of_view']
         ann = dict(bboxes=gt_bboxes, labels=gt_labels, ignore=ignore)
         return ann
-
-    def evaluate(self, results, metric=['track'], logger=None):
-        """Evaluation in OPE protocol.
-
-        Args:
-            results (dict): Testing results of the dataset.
-            metric (str | list[str]): Metrics to be evaluated. Options are
-                'track'.
-            logger (logging.Logger | str | None): Logger used for printing
-                related information during evaluation. Default: None.
-
-        Returns:
-            dict[str, float]: OPE style evaluation metric (i.e. success,
-            norm precision and precision).
-        """
-        if isinstance(metric, list):
-            metrics = metric
-        elif isinstance(metric, str):
-            metrics = [metric]
-        else:
-            raise TypeError('metric must be a list or a str.')
-        allowed_metrics = ['track']
-        for metric in metrics:
-            if metric not in allowed_metrics:
-                raise KeyError(f'metric {metric} is not supported.')
-
-        eval_results = dict()
-        if 'track' in metrics:
-            assert len(self.data_infos) == len(results['track_results'])
-            print_log('Evaluate OPE Benchmark...', logger=logger)
-            inds = [
-                i for i, _ in enumerate(self.data_infos) if _['frame_id'] == 0
-            ]
-            num_vids = len(inds)
-            inds.append(len(self.data_infos))
-
-            track_bboxes = [
-                list(
-                    map(lambda x: x[:4],
-                        results['track_results'][inds[i]:inds[i + 1]]))
-                for i in range(num_vids)
-            ]
-
-            ann_infos = [self.get_ann_info(_) for _ in self.data_infos]
-            ann_infos = [
-                ann_infos[inds[i]:inds[i + 1]] for i in range(num_vids)
-            ]
-            track_eval_results = eval_sot_ope(
-                results=track_bboxes, annotations=ann_infos)
-            eval_results.update(track_eval_results)
-
-            for k, v in eval_results.items():
-                if isinstance(v, float):
-                    eval_results[k] = float(f'{(v):.3f}')
-            print_log(eval_results, logger=logger)
-
-        return eval_results