[Feature] Support TrackingNet dataset in SOT (#268)

* init doc * update zh-CN docs * fix some translation error * fix some translation error * add a blank between English and Chinese * fix some translation error * fix zh-CN docs * fix zh-CN docs * support uav * support trackingnet * simplify uav config * tiny changes in sot_siamrpn_params_search.py * add sot_test_dataset.py * refactor sot test dataset class * update siamrpn README.md * [wip]uav unittest * tiny changes in sot unittest * init trackinget * fix docstring * convert uav to uav123 * add ignore key for evaluation * fix docstring * fix docstring and rename file * remove 'ignore' key when converting uav dataset * fix some typos * refactor tools structure * fix docs * trackingnet docs * trackingnet docs * fix error * del ignore * fix typo Co-authored-by: zhangjingwei <zhangjingwei@pjlab.org.cn> Co-authored-by: zhangjingwei <zhangjingwei@sensetime.com>
open-mmlab · Sep 13, 2021 · 5a57163 · 5a57163
1 parent 1fa9280
commit 5a57163
Show file tree

Hide file tree

Showing 36 changed files with 367 additions and 75 deletions.
diff --git a/configs/sot/siamese_rpn/README.md b/configs/sot/siamese_rpn/README.md
@@ -32,8 +32,14 @@ The checkpoints from 10-th to 20-th epoch will be evaluated during training.
 
 After training, you need to pick up the best checkpoint from the log file, then use the best checkpoint to search the hyperparameters on UAV123 following [here](https://github.com/open-mmlab/mmtracking/blob/master/docs/useful_tools_scripts.md#siameserpn-test-time-parameter-search) to achieve the best results.
 
-We observe around xxx points fluctuations in Success and xxx points fluctuations in Norm percision. We provide the best model with its configuration and training log.
-
 |    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | Success | Norm precision | Config | Download |
 | :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :----: | :------: | :--------: |
 |    R-50    |  -  |   20e    | 7.54     | -              | 61.8 | 77.3 | [config](siamese_rpn_r50_1x_uav123.py) | [model](https://download.openmmlab.com/mmtracking/sot/siamese_rpn/siamese_rpn_r50_1x_lasot/siamese_rpn_r50_1x_lasot_20201218_051019-3c522eff.pth) &#124; [log](https://download.openmmlab.com/mmtracking/sot/siamese_rpn/siamese_rpn_r50_1x_lasot/siamese_rpn_r50_1x_lasot_20201218_051019.log.json) |
+
+### TrackingNet
+
+The best model on LaSOT is submitted to [the evaluation server on TrackingNet Chanllenge](http://eval.tracking-net.org/web/challenges/challenge-page/39/submission). We provide the best model with its configuration and training log.
+
+|    Backbone     |  Style  | Lr schd | Mem (GB) | Inf time (fps) | Success | Norm precision | Config | Download |
+| :-------------: | :-----: | :-----: | :------: | :------------: | :----: | :----: | :------: | :--------: |
+|    R-50    |  -  |   20e    |      | -              | 70.6 | 77.6 | [config](siamese_rpn_r50_1x_trackingnet.py) | [model](https://download.openmmlab.com/mmtracking/sot/siamese_rpn/siamese_rpn_r50_1x_lasot/siamese_rpn_r50_1x_lasot_20201218_051019-3c522eff.pth) &#124; [log](https://download.openmmlab.com/mmtracking/sot/siamese_rpn/siamese_rpn_r50_1x_lasot/siamese_rpn_r50_1x_lasot_20201218_051019.log.json) |
diff --git a/configs/sot/siamese_rpn/siamese_rpn_r50_1x_trackingnet.py b/configs/sot/siamese_rpn/siamese_rpn_r50_1x_trackingnet.py
@@ -0,0 +1,10 @@
+_base_ = ['./siamese_rpn_r50_1x_lasot.py']
+
+data_root = 'data/'
+# dataset settings
+data = dict(
+    test=dict(
+        type='TrackingNetTestDataset',
+        ann_file=data_root +
+        'trackingnet/TEST/annotations/trackingnet_test.json',
+        img_prefix=data_root + 'trackingnet/TEST/frames'))
diff --git a/configs/sot/siamese_rpn/siamese_rpn_r50_1x_uav123.py b/configs/sot/siamese_rpn/siamese_rpn_r50_1x_uav123.py
@@ -7,10 +7,6 @@
 data_root = 'data/'
 # dataset settings
 data = dict(
-    val=dict(
-        type='UAV123Dataset',
-        ann_file=data_root + 'UAV123/annotations/uav123.json',
-        img_prefix=data_root + 'UAV123/data_seq/UAV123'),
     test=dict(
         type='UAV123Dataset',
         ann_file=data_root + 'UAV123/annotations/uav123.json',

diff --git a/docs/dataset.md b/docs/dataset.md
@@ -9,6 +9,7 @@ This page provides the instructions for dataset preparation on existing benchmar
 - Single Object Tracking
   - [LaSOT](http://vision.cs.stonybrook.edu/~lasot/)
   - [UAV123](https://cemse.kaust.edu.sa/ivul/uav123/)
+  - [TrackingNet](https://tracking-net.org/)
 
 ### 1. Download Datasets
 
@@ -22,7 +23,7 @@ Notes:
 
 - For the training and testing of multi object tracking task, only one of the MOT Challenge dataset (e.g. MOT17) is needed.
 
-- For the training and testing of single object tracking task, the MSCOCO, ILSVRC, LaSOT and UAV123 datasets are needed.
+- For the training and testing of single object tracking task, the MSCOCO, ILSVRC, LaSOT, UAV123 and TrackingNet datasets are needed.
 
 ```
 mmtracking
@@ -71,6 +72,11 @@ mmtracking
 │   │   │   │   ├── boat1
 │   │   ├── anno
 │   │   │   ├── UAV123
+│   │
+│   ├── trackingnet
+│   │   ├── TEST
+│   │   │   ├── anno
+│   │   │   ├── zips
 ```
 
 ### 2. Convert Annotations
@@ -80,21 +86,27 @@ In this case, you need to convert the offical annotations to this style. We prov
 
 ```shell
 # ImageNet DET
-python ./tools/convert_datasets/imagenet2coco_det.py -i ./data/ILSVRC -o ./data/ILSVRC/annotations
+python ./tools/convert_datasets/ilsvrc/imagenet2coco_det.py -i ./data/ILSVRC -o ./data/ILSVRC/annotations
 
 # ImageNet VID
-python ./tools/convert_datasets/imagenet2coco_vid.py -i ./data/ILSVRC -o ./data/ILSVRC/annotations
+python ./tools/convert_datasets/ilsvrc/imagenet2coco_vid.py -i ./data/ILSVRC -o ./data/ILSVRC/annotations
 
 # LaSOT
-python ./tools/convert_datasets/lasot2coco.py -i ./data/lasot/LaSOTTesting -o ./data/lasot/annotations
+python ./tools/convert_datasets/lasot/lasot2coco.py -i ./data/lasot/LaSOTTesting -o ./data/lasot/annotations
 
 # MOT17
 # The processing of other MOT Challenge dataset is the same as MOT17
-python ./tools/convert_datasets/mot2coco.py -i ./data/MOT17/ -o ./data/MOT17/annotations --split-train --convert-det
-python ./tools/convert_datasets/mot2reid.py -i ./data/MOT17/ -o ./data/MOT17/reid --val-split 0.2 --vis-threshold 0.3
+python ./tools/convert_datasets/mot/mot2coco.py -i ./data/MOT17/ -o ./data/MOT17/annotations --split-train --convert-det
+python ./tools/convert_datasets/mot/mot2reid.py -i ./data/MOT17/ -o ./data/MOT17/reid --val-split 0.2 --vis-threshold 0.3
 
 # UAV123
-python ./tools/convert_datasets/uav2coco.py -i ./data/UAV123/ -o ./data/UAV123/annotations
+python ./tools/convert_datasets/uav123/uav2coco.py -i ./data/UAV123/ -o ./data/UAV123/annotations
+
+# TrackingNet
+# unzip files in 'TEST/zips/*.zip'
+bash ./tools/convert_datasets/trackingnet/unzip_trackingnet_test.sh ./data/trackingnet/TEST
+# generate testset annotaions
+python ./tools/convert_datasets/trackingnet/trackingnet2coco.py -i ./data/trackingnet/TEST/ -o ./data/trackingnet/TEST/annotations
 ```
 
 The folder structure will be as following after your run these scripts:
@@ -153,6 +165,15 @@ mmtracking
 │   │   ├── anno (the offical annotation files)
 │   │   │   ├── UAV123
 │   │   ├── annotations (the converted annotation file)
+│   │
+│   ├── trackingnet
+│   │   ├── TEST
+│   │   │   ├── anno (the offical annotation files)
+│   │   │   ├── zips
+│   │   │   ├── annotations (the converted annotation file)
+│   │   │   ├── frames (the unzipped folders)
+│   │   │   │   ├── 0-6LB4FqxoE_0
+│   │   │   │   ├── 07Ysk1C0ZX0_0
 ```
 
 #### The folder of annotations in ILSVRC
@@ -227,3 +248,11 @@ Images in `reid/imgs` are cropped from raw images in `MOT17/train` by the corres
 There are only 1 json files in `data/UAV123/annotations`:
 
 `uav123.json`:  Json file containing the annotations information of the UAV123 dataset.
+
+#### The folder of frames and annotations in TrackingNet
+
+There are 511 video directories of TrackingNet testset in `data/trackingnet/TEST/frames`, and each video directory contains all images of the video.
+
+There are only 1 json files in `data/trackingnet/TEST/annotations`:
+
+`trackingnet_test.json`:  Json file containing the annotations information of the testing set in TrackingNet dataset.
diff --git a/docs/model_zoo.md b/docs/model_zoo.md
@@ -5,7 +5,7 @@
 - We use distributed training.
 - All pytorch-style pretrained backbones on ImageNet are from PyTorch model zoo.
 - For fair comparison with other codebases, we report the GPU memory as the maximum value of `torch.cuda.max_memory_allocated()` for all 8 GPUs. Note that this value is usually less than what `nvidia-smi` shows.
-- We report the inference time as the total time of network forwarding and post-processing, excluding the data loading time. Results are obtained with the script `tools/benchmark.py` which computes the average time on 2000 images.
+- We report the inference time as the total time of network forwarding and post-processing, excluding the data loading time. Results are obtained with the script `tools/analysis/benchmark.py` which computes the average time on 2000 images.
 - Speed benchmark environments
 
     HardWare

diff --git a/docs/useful_tools_scripts.md b/docs/useful_tools_scripts.md
@@ -2,7 +2,7 @@ We provide lots of useful tools under the `tools/` directory.
 
 ## MOT Test-time Parameter Search
 
-`tools/mot_param_search.py` can search the parameters of the `tracker` in MOT models.
+`tools/analysis/mot/mot_param_search.py` can search the parameters of the `tracker` in MOT models.
 It is used as the same manner with `tools/test.py` but different in the configs.
 
 Here is an example that shows how to modify the configs:
@@ -43,9 +43,21 @@ Here is an example that shows how to modify the configs:
 
     Then the script will test the totally 12 cases and log the results.
 
+## SiameseRPN++ Test-time Parameter Search
+
+`tools/analysis/sot/sot_siamrpn_param_search.py` can search the test-time tracking parameters in SiameseRPN++: `penalty_k`, `lr` and `window_influence`. You need to pass the searching range of each parameter into the argparser.
+
+Example on UAV123 dataset:
+
+```shell
+./tools/analysis/sot/dist_sot_siamrpn_param_search.sh [${CONFIG_FILE}] [$GPUS] \
+[--checkpoint ${CHECKPOINT}] [--log ${LOG_FILENAME}] [--eval ${EVAL}] \
+[--penalty-k-range 0.05,0.5,0.05] [--lr-range 0.3,0.45,0.02] [--win-infu-range 0.46,0.55,0.02]
+```
+
 ## Log Analysis
 
-`tools/analyze_logs.py` plots loss/mAP curves given a training log file.
+`tools/analysis/analyze_logs.py` plots loss/mAP curves given a training log file.
 
  ```shell
 python tools/analyze_logs.py plot_curve [--keys ${KEYS}] [--title ${TITLE}] [--legend ${LEGEND}] [--backend ${BACKEND}] [--style ${STYLE}] [--out ${OUT_FILE}]
@@ -56,25 +68,25 @@ Examples:
 - Plot the classification loss of some run.
 
     ```shell
-    python tools/analyze_logs.py plot_curve log.json --keys loss_cls --legend loss_cls
+    python tools/analysis/analyze_logs.py plot_curve log.json --keys loss_cls --legend loss_cls
     ```
 
 - Plot the classification and regression loss of some run, and save the figure to a pdf.
 
     ```shell
-    python tools/analyze_logs.py plot_curve log.json --keys loss_cls loss_bbox --out losses.pdf
+    python tools/analysis/analyze_logs.py plot_curve log.json --keys loss_cls loss_bbox --out losses.pdf
     ```
 
 - Compare the bbox mAP of two runs in the same figure.
 
     ```shell
-    python tools/analyze_logs.py plot_curve log1.json log2.json --keys bbox_mAP --legend run1 run2
+    python tools/analysis/analyze_logs.py plot_curve log1.json log2.json --keys bbox_mAP --legend run1 run2
     ```
 
 - Compute the average training speed.
 
     ```shell
-    python tools/analyze_logs.py cal_train_time log.json [--include-outliers]
+    python tools/analysis/analyze_logs.py cal_train_time log.json [--include-outliers]
     ```
 
     The output is expected to be like the following.
@@ -91,7 +103,7 @@ Examples:
 
 ### Prepare a model for publishing
 
-`tools/publish_model.py` helps users to prepare their model for publishing.
+`tools/analysis/publish_model.py` helps users to prepare their model for publishing.
 
 Before you upload a model to AWS, you may want to
 
@@ -100,35 +112,23 @@ Before you upload a model to AWS, you may want to
 3. compute the hash of the checkpoint file and append the hash id to the filename.
 
 ```shell
-python tools/publish_model.py ${INPUT_FILENAME} ${OUTPUT_FILENAME}
+python tools/analysis/publish_model.py ${INPUT_FILENAME} ${OUTPUT_FILENAME}
 ```
 
 E.g.,
 
 ```shell
-python tools/publish_model.py work_dirs/dff_faster_rcnn_r101_dc5_1x_imagenetvid/latest.pth dff_faster_rcnn_r101_dc5_1x_imagenetvid.pth
+python tools/analysis/publish_model.py work_dirs/dff_faster_rcnn_r101_dc5_1x_imagenetvid/latest.pth dff_faster_rcnn_r101_dc5_1x_imagenetvid.pth
 ```
 
 The final output filename will be `dff_faster_rcnn_r101_dc5_1x_imagenetvid_20201230-{hash id}.pth`.
 
-## SiameseRPN++ Test-time Parameter Search
-
-`tools/sot_siamrpn_param_search.py` can search the test-time tracking parameters in SiameseRPN++: `penalty_k`, `lr` and `window_influence`. You need to pass the searching range of each parameter into the argparser.
-
-Example on UAV123 dataset:
-
-```shell
-./tools/dist_sot_siamrpn_param_search.sh [${CONFIG_FILE}] [$GPUS] \
-[--checkpoint ${CHECKPOINT}] [--log ${LOG_FILENAME}] [--eval ${EVAL}] \
-[--penalty-k-range 0.05,0.5,0.05] [--lr-range 0.3,0.45,0.02] [--win-infu-range 0.46,0.55,0.02]
-```
-
 ## Miscellaneous
 
 ### Print the entire config
 
-`tools/print_config.py` prints the whole config verbatim, expanding all its imports.
+`tools/analysis/print_config.py` prints the whole config verbatim, expanding all its imports.
 
 ```shell
-python tools/print_config.py ${CONFIG} [-h] [--options ${OPTIONS [OPTIONS...]}]
+python tools/analysis/print_config.py ${CONFIG} [-h] [--options ${OPTIONS [OPTIONS...]}]
 ```
diff --git a/docs_zh-CN/dataset.md b/docs_zh-CN/dataset.md
@@ -9,6 +9,7 @@
 - 单目标跟踪
   - [LaSOT](http://vision.cs.stonybrook.edu/~lasot/)
   - [UAV123](https://cemse.kaust.edu.sa/ivul/uav123/)
+  - [TrackingNet](https://tracking-net.org/)
 
 ### 1. 下载数据集
 
@@ -22,7 +23,7 @@
 
 - 对于多目标跟踪任务的训练和测试，只需要 MOT Challenge 中的任意一个数据集（比如 MOT17）。
 
-- 对于单目标跟踪任务的训练和测试，需要 MSCOCO，ILSVRC, LaSOT 和 UAV123 数据集。
+- 对于单目标跟踪任务的训练和测试，需要 MSCOCO，ILSVRC, LaSOT, UAV123 和 TrackingNet 数据集。
 
 ```
 mmtracking
@@ -71,6 +72,11 @@ mmtracking
 │   │   │   │   ├── boat1
 │   │   ├── anno
 │   │   │   ├── UAV123
+│   │
+│   ├── trackingnet
+│   │   ├── TEST
+│   │   │   ├── anno
+│   │   │   ├── zips
 ```
 
 ### 2. 转换标注格式
@@ -81,21 +87,27 @@ mmtracking
 
 ```shell
 # ImageNet DET
-python ./tools/convert_datasets/imagenet2coco_det.py -i ./data/ILSVRC -o ./data/ILSVRC/annotations
+python ./tools/convert_datasets/ilsvrc/imagenet2coco_det.py -i ./data/ILSVRC -o ./data/ILSVRC/annotations
 
 # ImageNet VID
-python ./tools/convert_datasets/imagenet2coco_vid.py -i ./data/ILSVRC -o ./data/ILSVRC/annotations
+python ./tools/convert_datasets/ilsvrc/imagenet2coco_vid.py -i ./data/ILSVRC -o ./data/ILSVRC/annotations
 
 # LaSOT
-python ./tools/convert_datasets/lasot2coco.py -i ./data/lasot/LaSOTTesting -o ./data/lasot/annotations
+python ./tools/convert_datasets/lasot/lasot2coco.py -i ./data/lasot/LaSOTTesting -o ./data/lasot/annotations
 
 # MOT17
 # MOT Challenge中其余数据集的处理与MOT17相同
-python ./tools/convert_datasets/mot2coco.py -i ./data/MOT17/ -o ./data/MOT17/annotations --split-train --convert-det
-python ./tools/convert_datasets/mot2reid.py -i ./data/MOT17/ -o ./data/MOT17/reid --val-split 0.2 --vis-threshold 0.3
+python ./tools/convert_datasets/mot/mot2coco.py -i ./data/MOT17/ -o ./data/MOT17/annotations --split-train --convert-det
+python ./tools/convert_datasets/mot/mot2reid.py -i ./data/MOT17/ -o ./data/MOT17/reid --val-split 0.2 --vis-threshold 0.3
 
 # UAV123
-python ./tools/convert_datasets/uav2coco.py -i ./data/UAV123/ -o ./data/UAV123/annotations
+python ./tools/convert_datasets/uav123/uav2coco.py -i ./data/UAV123/ -o ./data/UAV123/annotations
+
+# TrackingNet
+# 解压目录 'TEST/zips' 下的所有 '*.zip' 文件
+bash ./tools/convert_datasets/trackingnet/unzip_trackingnet_test.sh ./data/trackingnet/TEST
+# 生成测试集标注
+python ./tools/convert_datasets/trackingnet/trackingnet2coco.py -i ./data/trackingnet/TEST/ -o ./data/trackingnet/TEST/annotations
 ```
 
 完成以上格式转换后，文件目录结构如下：
@@ -154,6 +166,15 @@ mmtracking
 │   │   ├── anno (the offical annotation files)
 │   │   │   ├── UAV123
 │   │   ├── annotations (the converted annotation file)
+│   │
+│   ├── trackingnet
+│   │   ├── TEST
+│   │   │   ├── anno (the offical annotation files)
+│   │   │   ├── zips
+│   │   │   ├── annotations (the converted annotation file)
+│   │   │   ├── frames (the unzipped folders)
+│   │   │   │   ├── 0-6LB4FqxoE_0
+│   │   │   │   ├── 07Ysk1C0ZX0_0
 ```
 
 #### ILSVRC的标注文件夹
@@ -229,3 +250,11 @@ MOT17-02-FRCNN_000009/000081.jpg 3
 在 `data/UAV123/annotations` 中只有一个 json 文件:
 
 `uav123.json`: 包含 UAV123 数据集标注信息的 json 文件。
+
+#### TrackingNet的标注和视频帧文件夹
+
+在 `data/trackingnet/TEST/frames` 文件夹下有 TrackingNet 测试集的 511 个视频目录， 每个视频目录下面包含该视频所有图片。
+
+在 `data/trackingnet/TEST/annotations` 中只有一个 json 文件：
+
+`trackingnet_test.json`： 包含 TrackingNet 测试集标注信息的 json 文件。
diff --git a/docs_zh-CN/model_zoo.md b/docs_zh-CN/model_zoo.md
@@ -5,7 +5,7 @@
 - 我们默认使用分布式训练。
 - 所有 pytorch 类型的预训练骨干网络都是来自 Pytorch 的模型库。
 - 为了与其他代码库进行公平比较，我们以全部 8 个 GPU 的 `torch.cuda.max_memory_allocated()` 的最大值作为 GPU 显存使用量。请注意，此值通常小于 `nvidia-smi` 显示的值。
-- 该推理时间不包含数据加载时间，推理时间结果是通过脚本 `tools/benchmark.py` 获得的，该脚本计算处理 2000 张图像的平均时间。
+- 该推理时间不包含数据加载时间，推理时间结果是通过脚本 `tools/analysis/benchmark.py` 获得的，该脚本计算处理 2000 张图像的平均时间。
 - 速度基准测试的环境如下：
 
     硬件环境：