Support for img norm in graph during onnx export #1027

nick-clarifai · 2021-11-09T21:55:49Z

Motivation

The purpose of this PR is to provide an option that does the image norm step of inference within the graph itself during the export to an ONNX model.

Modification

`mmseg/datasets/pipelines/transforms.py`:

an init flag is added which causes the image norm to be skipped during __call__ and causes the flag to be passed along in the img_norm_cfg

`mmseg/models/segmentors/base.py`

forward modified to do image normalization to the input if img_norm_cfg['normalize_in_graph'] is True

`tools/pytorch2onnx.py`:

command line arg added
normalize step in test pipeline is modified to skip normalization but still add img_norm_cfg
input is cloned in verification step so the pre-image norm input can be compared

Use cases

This would allow one to have a totally contained model that doesn't require extra python code for inference so that the ONNX model could easily be used in a setting such as NVIDIA Triton.

…specified

CLAassistant · 2021-11-09T21:55:53Z

All committers have signed the CLA.

codecov · 2021-11-10T05:33:34Z

Codecov Report

Attention: Patch coverage is 27.27273% with 8 lines in your changes missing coverage. Please review.

Project coverage is 89.98%. Comparing base (9975c67) to head (f62cf69).
Report is 257 commits behind head on master.

Files with missing lines	Patch %	Lines
mmseg/models/segmentors/base.py	0.00%	6 Missing and 1 partial ⚠️
mmseg/datasets/pipelines/transforms.py	75.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1027      +/-   ##
==========================================
- Coverage   90.11%   89.98%   -0.14%     
==========================================
  Files         125      125              
  Lines        7262     7277      +15     
  Branches     1206     1210       +4     
==========================================
+ Hits         6544     6548       +4     
- Misses        515      524       +9     
- Partials      203      205       +2

Flag	Coverage Δ
unittests	`89.98% <27.27%> (-0.14%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Junjun2016 · 2021-11-11T01:57:13Z

Hi @nick-clarifai
Thanks for your contribution.
We will review it in the near future.

mmseg/models/segmentors/base.py

RockeyCoss · 2021-11-15T08:25:28Z

Please merge the master branch into your branch, thank you.

Junjun2016 · 2021-11-16T12:49:47Z

Hi @RunningLeon
Please review it.

mmseg/models/segmentors/base.py

RunningLeon · 2021-11-17T03:05:40Z

@nick-clarifai Hi, thanks for your PR. If you add --normalize-in-graph in pytorch2onnx.py, then you might need to support it in downstream tool onnx2tensorrt.py as well.

nick-clarifai · 2021-11-18T19:47:57Z

@nick-clarifai Hi, thanks for your PR. If you add --normalize-in-graph in pytorch2onnx.py, then you might need to support it in downstream tool onnx2tensorrt.py as well.

Yeah this makes sense. The changes I can see being necessary would be allowing image normalization in the preprocessing step to be skipped and adding a command line arg to skip normalization? I'm happy to make these changes if we're on the same page. Let me know if there's something else I'm forgetting there.

RunningLeon · 2021-11-19T06:18:20Z

@nick-clarifai Hi, thanks for your PR. If you add --normalize-in-graph in pytorch2onnx.py, then you might need to support it in downstream tool onnx2tensorrt.py as well.

Yeah this makes sense. The changes I can see being necessary would be allowing image normalization in the preprocessing step to be skipped and adding a command line arg to skip normalization? I'm happy to make these changes if we're on the same page. Let me know if there's something else I'm forgetting there.

Yes. --normalize-in-graph should be added to onnx2tensorrt.py as well.

nick-clarifai · 2021-11-19T20:45:17Z

Thanks for the feedback. onnx2tensorrt.py now has an option to skip image normalization in the preprocessing step. I didn't think it made sense to name the arg --normalize-in-graph because the tensorrt model will only do so if the input onnx model does normalize in graph, so I named the arg --skip-normalize instead. Let me know if you suggest any more changes.

RunningLeon · 2021-11-22T02:41:16Z

Thanks for the feedback. onnx2tensorrt.py now has an option to skip image normalization in the preprocessing step. I didn't think it made sense to name the arg --normalize-in-graph because the tensorrt model will only do so if the input onnx model does normalize in graph, so I named the arg --skip-normalize instead. Let me know if you suggest any more changes.

Actually, skip-normalize works for me.

nick-clarifai · 2021-12-20T20:04:33Z

Is there something else that should be done before this could be merged? Let me know if there's anything I can do.

tools/pytorch2onnx.py

tools/onnx2tensorrt.py

RunningLeon · 2021-12-23T11:47:32Z

Is there something else that should be done before this could be merged? Let me know if there's anything I can do.

@nick-clarifai Hi, sorry for the late reply. After resolve the rest comments, this PR is ready to go merge for me.

nick-clarifai · 2021-12-30T16:03:00Z

Is there something else that should be done before this could be merged? Let me know if there's anything I can do.

@nick-clarifai Hi, sorry for the late reply. After resolve the rest comments, this PR is ready to go merge for me.

Thanks for the feedback @RunningLeon. I've made the requested changes.

RunningLeon

LGTM

MengzhangLI · 2022-01-16T06:08:44Z

Hi, @nick-clarifai nick, sorry for late reply.

This pr is closely to merge, could you please fix unit test problem? Your modification is not coveraged by unit test.

Best,

nick-clarifai · 2022-02-01T20:41:41Z

Hi, @nick-clarifai nick, sorry for late reply.

This pr is closely to merge, could you please fix unit test problem? Your modification is not coveraged by unit test.

Best,

Sure, can you help me decide which is the best test to write? Should I write a test that runs the code from pytorch2onnx.py with the verification enabled? Or should I do something like add a test to tests/test_models/test_segmentors/test_encoder_decoder.py that makes sure the added code in mmseg/models/segmentors/base.py is run?

…ab#1027)

OpenMMLab-Assistant005 · 2023-04-14T08:00:14Z

Hi @nick-clarifai ！We are grateful for your efforts in helping improve this open-source project during your personal time.
Welcome to join OpenMMLab Special Interest Group (SIG) private channel on Discord, where you can share your experiences, ideas, and build connections with like-minded peers. To join the SIG channel, simply message moderator— OpenMMLab on Discord or briefly share your open-source contributions in the #introductions channel and we will assist you. Look forward to seeing you there! Join us ：https://discord.gg/UjgXkPWNqA
If you have a WeChat account，welcome to join our community on WeChat. You can add our assistant ：openmmlabwx. Please add "mmsig + Github ID" as a remark when adding friends：）
Thank you again for your contribution❤

* map location to cpu when load checkpoint (open-mmlab#1007) * [Enhancement] Support minus output feature index in mobilenet_v3 (open-mmlab#1005) * fix typo in mobilenet_v3 * fix typo in mobilenet_v3 * use -1 to indicate output tensors from final stage * support negative out_indices * [Enhancement] inference speed and flops tools. (open-mmlab#986) * add the function to test the dummy forward speed of models. * add tools to test the flops and inference speed of multiple models. * [Fix] Update pose tracking demo to be compatible with latest mmtrakcing (open-mmlab#1014) * update mmtracking demo * support both track_bboxes and track_results * add docstring * [Fix] fix skeleton_info of coco wholebody dataset (open-mmlab#1010) * fix wholebody base dataset * fix lint * fix lint Co-authored-by: ly015 <liyining0712@gmail.com> * [Feature] Add ViPNAS models for wholebody keypoint detection (open-mmlab#1009) * add configs * add dark configs * add checkpoint and readme * update webcam demo * fix model path in webcam demo * fix unittest * [Fix] Fix bbox label visualization (open-mmlab#1020) * update model metafiles (open-mmlab#1001) * update hourglass ae .md (open-mmlab#1027) * [Feature] Add ViPNAS mbv3 (open-mmlab#1025) * add vipnas mbv3 * test other variants * submission for mmpose * add unittest * add readme * update .yml * fix lint * rebase * fix pytest Co-authored-by: jin-s13 <jinsheng13@foxmail.com> * [Enhancement] Set a random seed when the user does not set a seed (open-mmlab#1030) * fix randseed * fix lint * fix import * fix isort * update yapf hook * revert yapf version * add cfg file for flops and speed test, change the bulid_posenet to init_pose_model and fix some typo in cfg (open-mmlab#1028) * [Enhancement] Add more functions for speed test tool (open-mmlab#1034) * add batch size and device args in speed test script, and remove MMDataParallel warper * add vipnas_mbv3 model * fix dead link (open-mmlab#1038) * Skip CI when some specific files were changed (open-mmlab#1041) * update sigmas (open-mmlab#1040) * add more configs, ckpts and logs for HRNet on PoseTrack18 (open-mmlab#1035) * [Feature] Add PoseWarper dataset (open-mmlab#1006) * add PoseWarper dataset and base class * modify pipelines related to video * add unittest for PoseWarper dataset * add unittest for evaluation function in posetrack18-realted dataset, and add some annotations json files * fix typo * fix unittest CI failure * fix typo * add PoseWarper dataset and base class * modify pipelines related to video * add unittest for PoseWarper dataset * add unittest for evaluation function in posetrack18-realted dataset, and add some annotations json files * fix typo * fix unittest CI failure * fix typo * modify some methods in the base class to improve code coverage rate * recover some mistakenly-deleted notes * remove test_dataset_info part for the new TopDownPoseTrack18VideoDataset class * cancel uncompleted previous runs (open-mmlab#1053) * [Doc] Add inference speed results (open-mmlab#1044) * add docs related to inference speed results * add corresponding Chinese docs and fix some typos * add Chinese docs in readthedocs * remove the massive table in readme * minor modification to wording Co-authored-by: ly015 <liyining0712@gmail.com> * [Feature] Add PoseWarper detector model (open-mmlab#932) * Add top down video detector module * Add PoseWarper neck * add function _freeze_stages * fix typo * modify PoseWarper detector and PoseWarperNeck * fix typo * modify posewarper detector and neck * Delete top_down_video.py change the base class of `PoseWarper` detector from `TopDownVideo` to `TopDown` * fix spell typo * modify detector and neck * add unittest for detector and neck * modify unittest for posewarper forward * Add top down video detector module * Add PoseWarper neck * add function _freeze_stages * fix typo * modify PoseWarper detector and PoseWarperNeck * fix typo * modify posewarper detector and neck * Delete top_down_video.py change the base class of `PoseWarper` detector from `TopDownVideo` to `TopDown` * fix spell typo * modify detector and neck * add unittest for detector and neck * modify unittest for posewarper forward * modify dependency on mmcv version in posewarper neck * reduce memory cost in test * modify flops tool for more flexible input format * Add top down video detector module * Add PoseWarper neck * add function _freeze_stages * fix typo * modify PoseWarper detector and PoseWarperNeck * fix typo * modify posewarper detector and neck * Delete top_down_video.py change the base class of `PoseWarper` detector from `TopDownVideo` to `TopDown` * fix spell typo * modify detector and neck * add unittest for detector and neck * modify unittest for posewarper forward * Add PoseWarper neck * modify PoseWarper detector and PoseWarperNeck * modify posewarper detector and neck * Delete top_down_video.py change the base class of `PoseWarper` detector from `TopDownVideo` to `TopDown` * fix spell typo * modify detector and neck * add unittest for detector and neck * modify unittest for posewarper forward * modify dependency on mmcv version in posewarper neck * reduce memory cost in test * modify flops tool for more flexible input format * modify the posewarper detector description * modify some arguments and related fields * modify default values for some args * fix readthedoc bulid typo * fix ignore path (open-mmlab#1059) * [Doc] Add related docs for PoseWarper (open-mmlab#1036) * add related docs for PoseWarper * add related readme docs for posewarper * modify related args in posewarper stage2 config * modify posewarper stage2 config path * add description about val_boxes path for data preparation (open-mmlab#1060) * bump version to v0.21.0 (open-mmlab#1061) * [Feature] Add ViPNAS_Mbv3 wholebody model (open-mmlab#1055) * add vipnas mbv3 coco_wholebody * add vipnas mbv3 coco_wholebody md&yml * fix lint Co-authored-by: ly015 <liyining0712@gmail.com> Co-authored-by: Lumin <30328525+luminxu@users.noreply.github.com> Co-authored-by: zengwang430521 <zengwang430521@gmail.com> Co-authored-by: Jas <jinsheng@sensetime.com> Co-authored-by: jin-s13 <jinsheng13@foxmail.com> Co-authored-by: Qikai Li <87690686+liqikai9@users.noreply.github.com> Co-authored-by: QwQ2000 <396707050@qq.com>

nick-clarifai added 13 commits September 28, 2021 17:09

add support for normalization in graph

f113cd7

add support for uint8 input

d8f9fe0

add back missing line

7970216

ensure model is normalized for verify, ensure norm only happens when …

93b1051

…specified

Merge branch 'master' of https://github.com/nick-clarifai/mmsegmentation

b3bd490

fix img norm bug, disable verify temporarily

a2ce38b

fix verify in onnx export

aa7be96

disable constant folding

3215bb8

Merge branch 'master' into onnx-img-norm

fdac3b8

restore constant folding

7c0d776

fix formatting for flake

9b8aee4

update docstring in transforms.Normalize

6f142d7

add normalize-in-graph to pytorch2onnx documentation

cebae08

Junjun2016 requested review from MengzhangLI and removed request for MengzhangLI November 10, 2021 05:27

Junjun2016 reviewed Nov 11, 2021

View reviewed changes

mmseg/models/segmentors/base.py Outdated Show resolved Hide resolved

nick-clarifai added 2 commits November 15, 2021 11:36

Merge branch 'master' into onnx-img-norm

eabb004

add img norm for all imgs in forward of BaseSegmentor

ad82c48

Junjun2016 requested a review from RunningLeon November 16, 2021 12:49

RunningLeon reviewed Nov 17, 2021

View reviewed changes

mmseg/models/segmentors/base.py Show resolved Hide resolved

add img normalization skip arg to onnx2tensorrt

4ef719f

Merge branch 'master' into onnx-img-norm

3712e34

Merge branch 'master' into onnx-img-norm

7e24fa7

RunningLeon requested a review from Junjun2016 December 23, 2021 07:14

RunningLeon reviewed Dec 23, 2021

View reviewed changes

tools/pytorch2onnx.py Show resolved Hide resolved

RunningLeon reviewed Dec 23, 2021

View reviewed changes

tools/onnx2tensorrt.py Show resolved Hide resolved

ensure inputs to graphs are float in onnx and tensorrt export

f62cf69

RunningLeon approved these changes Dec 31, 2021

View reviewed changes

Junjun2016 approved these changes Dec 31, 2021

View reviewed changes

MengzhangLI added WIP Work in process Merging PR waited for merging labels Jan 16, 2022

MengzhangLI approved these changes Jan 17, 2022

View reviewed changes

aravind-h-v pushed a commit to aravind-h-v/mmsegmentation that referenced this pull request Mar 27, 2023

Add --dataloader_num_workers to the DDPM training example (open-mml…

a6314a8

…ab#1027)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for img norm in graph during onnx export #1027

Support for img norm in graph during onnx export #1027

nick-clarifai commented Nov 9, 2021 •

edited

Loading

CLAassistant commented Nov 9, 2021 •

edited

Loading

codecov bot commented Nov 10, 2021 •

edited

Loading

Junjun2016 commented Nov 11, 2021

RockeyCoss commented Nov 15, 2021

Junjun2016 commented Nov 16, 2021

RunningLeon commented Nov 17, 2021

nick-clarifai commented Nov 18, 2021 •

edited

Loading

RunningLeon commented Nov 19, 2021

nick-clarifai commented Nov 19, 2021

RunningLeon commented Nov 22, 2021

nick-clarifai commented Dec 20, 2021

RunningLeon commented Dec 23, 2021

nick-clarifai commented Dec 30, 2021

RunningLeon left a comment

MengzhangLI commented Jan 16, 2022

nick-clarifai commented Feb 1, 2022

OpenMMLab-Assistant005 commented Apr 14, 2023

Support for img norm in graph during onnx export #1027

Are you sure you want to change the base?

Support for img norm in graph during onnx export #1027

Conversation

nick-clarifai commented Nov 9, 2021 • edited Loading

Motivation

Modification

mmseg/datasets/pipelines/transforms.py:

mmseg/models/segmentors/base.py

tools/pytorch2onnx.py:

Use cases

CLAassistant commented Nov 9, 2021 • edited Loading

codecov bot commented Nov 10, 2021 • edited Loading

Codecov Report

Junjun2016 commented Nov 11, 2021

RockeyCoss commented Nov 15, 2021

Junjun2016 commented Nov 16, 2021

RunningLeon commented Nov 17, 2021

nick-clarifai commented Nov 18, 2021 • edited Loading

RunningLeon commented Nov 19, 2021

nick-clarifai commented Nov 19, 2021

RunningLeon commented Nov 22, 2021

nick-clarifai commented Dec 20, 2021

RunningLeon commented Dec 23, 2021

nick-clarifai commented Dec 30, 2021

RunningLeon left a comment

Choose a reason for hiding this comment

MengzhangLI commented Jan 16, 2022

nick-clarifai commented Feb 1, 2022

OpenMMLab-Assistant005 commented Apr 14, 2023

nick-clarifai commented Nov 9, 2021 •

edited

Loading

`mmseg/datasets/pipelines/transforms.py`:

`mmseg/models/segmentors/base.py`

`tools/pytorch2onnx.py`:

CLAassistant commented Nov 9, 2021 •

edited

Loading

codecov bot commented Nov 10, 2021 •

edited

Loading

nick-clarifai commented Nov 18, 2021 •

edited

Loading