Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature]: Evaluation with TensorRT backend #5198

Merged
merged 11 commits into from
Jun 1, 2021
Merged

[Feature]: Evaluation with TensorRT backend #5198

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
10f99bc
evaluate trt models
RunningLeon May 13, 2021
55b8b8d
update version of onnx
RunningLeon May 14, 2021
0132b69
update maskrcnn results
RunningLeon May 17, 2021
d889a12
add backend argument
RunningLeon May 18, 2021
1ca37c5
update fcos results
RunningLeon May 21, 2021
46dcf1b
update
RunningLeon May 21, 2021
8ed6eec
fix bug
RunningLeon May 24, 2021
b68454f
Merge branch 'master' into trt_eval_tool
RunningLeon May 24, 2021
3af71bd
update doc
RunningLeon May 24, 2021
bc8bae8
fix conflicts with master
RunningLeon May 28, 2021
abb83d8
fix conflicts with master
RunningLeon Jun 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 22 additions & 10 deletions docs/tutorials/onnx2tensorrt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
- [How to convert models from ONNX to TensorRT](#how-to-convert-models-from-onnx-to-tensorrt)
- [Prerequisite](#prerequisite)
- [Usage](#usage)
- [How to evaluate the exported models](#how-to-evaluate-the-exported-models)
- [List of supported models convertable to TensorRT](#list-of-supported-models-convertable-to-tensorrt)
- [Reminders](#reminders)
- [FAQs](#faqs)
Expand All @@ -28,6 +29,7 @@ python tools/deployment/onnx2tensorrt.py \
--trt-file ${TRT_FILE} \
--input-img ${INPUT_IMAGE_PATH} \
--shape ${IMAGE_SHAPE} \
--max-shape ${MAX_IMAGE_SHAPE} \
--mean ${IMAGE_MEAN} \
--std ${IMAGE_STD} \
--dataset ${DATASET_NAME} \
Expand All @@ -42,6 +44,7 @@ Description of all arguments:
- `--trt-file`: The Path of output TensorRT engine file. If not specified, it will be set to `tmp.trt`.
- `--input-img` : The path of an input image for tracing and conversion. By default, it will be set to `demo/demo.jpg`.
- `--shape`: The height and width of model input. If not specified, it will be set to `400 600`.
- `--max-shape`: The maximum height and width of model input. If not specified, it will be set to the same as `--shape`.
- `--mean` : Three mean values for the input image. If not specified, it will be set to `123.675 116.28 103.53`.
- `--std` : Three std values for the input image. If not specified, it will be set to `58.395 57.12 57.375`.
- `--dataset` : The dataset name for the input model. If not specified, it will be set to `coco`.
Expand All @@ -65,23 +68,32 @@ python tools/deployment/onnx2tensorrt.py \
--verify \
```

## How to evaluate the exported models

We prepare a tool `tools/deplopyment/test.py` to evaluate TensorRT models.

Please refer to following links for more information.

- [how-to-evaluate-the-exported-models](pytorch2onnx.md#how-to-evaluate-the-exported-models)
- [results-and-models](pytorch2onnx.md#results-and-models)

## List of supported models convertable to TensorRT

The table below lists the models that are guaranteed to be convertable to TensorRT.

| Model | Config | Status |
| :----------: | :--------------------------------------------------: | :----: |
| SSD | `configs/ssd/ssd300_coco.py` | Y |
| FSAF | `configs/fsaf/fsaf_r50_fpn_1x_coco.py` | Y |
| FCOS | `configs/fcos/fcos_r50_caffe_fpn_4x4_1x_coco.py` | Y |
| YOLOv3 | `configs/yolo/yolov3_d53_mstrain-608_273e_coco.py` | Y |
| RetinaNet | `configs/retinanet/retinanet_r50_fpn_1x_coco.py` | Y |
| Faster R-CNN | `configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py` | Y |
| Mask R-CNN | `configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py` | Y |
| Model | Config | Dynamic Shape | Batch Inference | Note |
| :----------: | :--------------------------------------------------: | :-----------: | :-------------: | :---: |
| SSD | `configs/ssd/ssd300_coco.py` | Y | Y | |
| FSAF | `configs/fsaf/fsaf_r50_fpn_1x_coco.py` | Y | Y | |
| FCOS | `configs/fcos/fcos_r50_caffe_fpn_4x4_1x_coco.py` | Y | Y | |
| YOLOv3 | `configs/yolo/yolov3_d53_mstrain-608_273e_coco.py` | Y | Y | |
| RetinaNet | `configs/retinanet/retinanet_r50_fpn_1x_coco.py` | Y | Y | |
| Faster R-CNN | `configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py` | Y | Y | |
| Mask R-CNN | `configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py` | Y | Y | |

Notes:

- *All models above are tested with Pytorch==1.6.0 and TensorRT-7.2.1.6.Ubuntu-16.04.x86_64-gnu.cuda-10.2.cudnn8.0*
- *All models above are tested with Pytorch==1.6.0, onnx==1.7.0 and TensorRT-7.2.1.6.Ubuntu-16.04.x86_64-gnu.cuda-10.2.cudnn8.0*

## Reminders

Expand Down
73 changes: 43 additions & 30 deletions docs/tutorials/pytorch2onnx.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,19 @@
<!-- TOC -->

- [Tutorial 8: Pytorch to ONNX (Experimental)](#tutorial-8-pytorch-to-onnx-experimental)
- [How to convert models from Pytorch to ONNX](#how-to-convert-models-from-pytorch-to-onnx)
- [Prerequisite](#prerequisite)
- [Usage](#usage)
- [Description of all arguments](#description-of-all-arguments)
- [How to evaluate ONNX models with ONNX Runtime](#how-to-evaluate-onnx-models-with-onnx-runtime)
- [Prerequisite](#prerequisite-1)
- [Usage](#usage-1)
- [Description of all arguments](#description-of-all-arguments-1)
- [Results and Models](#results-and-models)
- [List of supported models exportable to ONNX](#list-of-supported-models-exportable-to-onnx)
- [The Parameters of Non-Maximum Suppression in ONNX Export](#the-parameters-of-non-maximum-suppression-in-onnx-export)
- [Reminders](#reminders)
- [FAQs](#faqs)
- [How to convert models from Pytorch to ONNX](#how-to-convert-models-from-pytorch-to-onnx)
- [Prerequisite](#prerequisite)
- [Usage](#usage)
- [Description of all arguments](#description-of-all-arguments)
- [How to evaluate the exported models](#how-to-evaluate-the-exported-models)
- [Prerequisite](#prerequisite-1)
- [Usage](#usage-1)
- [Description of all arguments](#description-of-all-arguments-1)
- [Results and Models](#results-and-models)
- [List of supported models exportable to ONNX](#list-of-supported-models-exportable-to-onnx)
- [The Parameters of Non-Maximum Suppression in ONNX Export](#the-parameters-of-non-maximum-suppression-in-onnx-export)
- [Reminders](#reminders)
- [FAQs](#faqs)

<!-- TOC -->

Expand Down Expand Up @@ -85,14 +85,12 @@ python tools/deployment/pytorch2onnx.py \
--verify \
--dynamic-export \
--cfg-options \
model.test_cfg.nms_pre=200 \
model.test_cfg.max_per_img=200 \
model.test_cfg.deploy_nms_pre=300 \
model.test_cfg.deploy_nms_pre=-1 \
```

## How to evaluate ONNX models with ONNX Runtime
## How to evaluate the exported models

We prepare a tool `tools/deplopyment/test.py` to evaluate ONNX models with ONNX Runtime backend.
We prepare a tool `tools/deplopyment/test.py` to evaluate ONNX models with ONNXRuntime and TensorRT.

### Prerequisite

Expand All @@ -102,13 +100,16 @@ We prepare a tool `tools/deplopyment/test.py` to evaluate ONNX models with ONNX
pip install onnx onnxruntime-gpu
```

- Install TensorRT by referring to [how-to-build-tensorrt-plugins-in-mmcv](https://mmcv.readthedocs.io/en/latest/tensorrt_plugin.html#how-to-build-tensorrt-plugins-in-mmcv)(optional)

### Usage

```bash
python tools/deployment/test.py \
${CONFIG_FILE} \
${ONNX_FILE} \
${MODEL_FILE} \
--out ${OUTPUT_FILE} \
--backend ${BACKEND} \
--format-only ${FORMAT_ONLY} \
--eval ${EVALUATION_METRICS} \
--show-dir ${SHOW_DIRECTORY} \
Expand All @@ -120,8 +121,9 @@ python tools/deployment/test.py \
### Description of all arguments

- `config`: The path of a model config file.
- `model`: The path of a ONNX model file.
- `model`: The path of an input model file.
- `--out`: The path of output result file in pickle format.
- `--backend`: Backend for input model to run and should be `onnxruntime` or `tensorrt`.
- `--format-only` : Format the output results without perform evaluation. It is useful when you want to format the result to a specific format and submit it to the test server. If not specified, it will be set to `False`.
- `--eval`: Evaluation metrics, which depends on the dataset, e.g., "bbox", "segm", "proposal" for COCO, and "mAP", "recall" for PASCAL VOC.
- `--show-dir`: Directory where painted images will be saved
Expand All @@ -138,60 +140,69 @@ python tools/deployment/test.py \
<th align="center">Metric</th>
<th align="center">PyTorch</th>
<th align="center">ONNX Runtime</th>
<th align="center">TensorRT</th>
</tr >
<tr >
<td align="center">FCOS</td>
<td align="center"><code>configs/fcos/fcos_r50_caffe_fpn_gn-head_4x4_1x_coco.py</code></td>
<td align="center">Box AP</td>
<td align="center">36.6</td>
<td align="center">36.5</td>
<td align="center">36.3</td>
</tr>
<tr >
<td align="center">FSAF</td>
<td align="center"><code>configs/fsaf/fsaf_r50_fpn_1x_coco.py</code></td>
<td align="center">Box AP</td>
<td align="center">36.0</td>
<td align="center">36.0</td>
<td align="center">35.9</td>
</tr>
<tr >
<td align="center">RetinaNet</td>
<td align="center"><code>configs/retinanet/retinanet_r50_fpn_1x_coco.py</code></td>
<td align="center">Box AP</td>
<td align="center">36.5</td>
<td align="center">36.4</td>
<td align="center">36.3</td>
</tr>
<tr >
<td align="center" align="center" >SSD</td>
<td align="center" align="center"><code>configs/ssd/ssd300_coco.py</code></td>
<td align="center" align="center">Box AP</td>
<td align="center" align="center">25.6</td>
<td align="center" align="center">25.6</td>
<td align="center" align="center">25.6</td>
</tr>
<tr >
<td align="center">YOLOv3</td>
<td align="center"><code>configs/yolo/yolov3_d53_mstrain-608_273e_coco.py</code></td>
<td align="center">Box AP</td>
<td align="center">33.5</td>
<td align="center">33.5</td>
<td align="center">33.5</td>
</tr>
<tr >
<td align="center">Faster R-CNN</td>
<td align="center"><code>configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py</code></td>
<td align="center">Box AP</td>
<td align="center">37.4</td>
<td align="center">37.4</td>
<td align="center">37.0</td>
</tr>
<tr >
<td align="center" rowspan="2">Mask R-CNN</td>
<td align="center" rowspan="2"><code>configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py</code></td>
<td align="center">Box AP</td>
<td align="center">38.2</td>
<td align="center">38.1</td>
<td align="center">37.7</td>
</tr>
<tr>
<td align="center">Mask AP</td>
<td align="center">34.7</td>
<td align="center">33.7</td>
<td align="center">33.3</td>
</tr>
<tr >
<td align="center">CornerNet</td>
Expand All @@ -206,25 +217,27 @@ Notes:

- All ONNX models are evaluated with dynamic shape on coco dataset and images are preprocessed according to the original config file. Note that CornerNet is evaluated without test-time flip, since currently only single-scale evaluation is supported with ONNX Runtime.

- Mask AP of Mask R-CNN drops by 1% for ONNXRuntime. The main reason is that the predicted masks are directly interpolated to original image in PyTorch, while they are at first interpolated to the preprocessed input image of the model and then to original image in ONNXRuntime.
- Mask AP of Mask R-CNN drops by 1% for ONNXRuntime. The main reason is that the predicted masks are directly interpolated to original image in PyTorch, while they are at first interpolated to the preprocessed input image of the model and then to original image in other backend.

## List of supported models exportable to ONNX

The table below lists the models that are guaranteed to be exportable to ONNX and runnable in ONNX Runtime.

| Model | Config | Dynamic Shape | Batch Inference | Note |
| :----------: | :-----------------------------------------------------------------: | :-----------: | :-------------: | :-----: |
| FCOS | `configs/fcos/fcos_r50_caffe_fpn_gn-head_4x4_1x_coco.py` | Y | Y | |
| FSAF | `configs/fsaf/fsaf_r50_fpn_1x_coco.py` | Y | Y | |
| RetinaNet | `configs/retinanet/retinanet_r50_fpn_1x_coco.py` | Y | Y | |
| SSD | `configs/ssd/ssd300_coco.py` | Y | Y | |
| YOLOv3 | `configs/yolo/yolov3_d53_mstrain-608_273e_coco.py` | Y | Y | |
| Faster R-CNN | `configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py` | Y | Y | |
| Mask R-CNN | `configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py` | Y | Y | |
| Model | Config | Dynamic Shape | Batch Inference | Note |
| :----------: | :-----------------------------------------------------------------: | :-----------: | :-------------: | :---------------------------------------------------------------------------: |
| FCOS | `configs/fcos/fcos_r50_caffe_fpn_gn-head_4x4_1x_coco.py` | Y | Y | |
| FSAF | `configs/fsaf/fsaf_r50_fpn_1x_coco.py` | Y | Y | |
| RetinaNet | `configs/retinanet/retinanet_r50_fpn_1x_coco.py` | Y | Y | |
| SSD | `configs/ssd/ssd300_coco.py` | Y | Y | |
| YOLOv3 | `configs/yolo/yolov3_d53_mstrain-608_273e_coco.py` | Y | Y | |
| Faster R-CNN | `configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py` | Y | Y | |
| Mask R-CNN | `configs/mask_rcnn/mask_rcnn_r50_fpn_1x_coco.py` | Y | Y | |
| CornerNet | `configs/cornernet/cornernet_hourglass104_mstest_10x5_210e_coco.py` | Y | N | no flip, no batch inference, tested with torch==1.7.0 and onnxruntime==1.5.1. |

Notes:

- Minimum required version of MMCV is `1.3.5`

- *All models above are tested with Pytorch==1.6.0 and onnxruntime==1.5.1*, except for CornerNet. For more details about the
torch version when exporting CornerNet to ONNX, which involves `mmcv::cummax`, please refer to the [Known Issues](https://github.com/open-mmlab/mmcv/blob/master/docs/onnxruntime_op.md#known-issues) in mmcv.

Expand Down
Loading