[Feature] Add mmrazor support (open-mmlab#220)

* Torchscript support (open-mmlab#159) * support torchscript * add nms * add torchscript configs and update deploy process and dump-info * typescript -> torchscript * add torchscript custom extension support * add ts custom ops again * support mmseg unet * [WIP] add optimizer for torchscript (open-mmlab#119) * add passes * add python api * Torchscript optimizer python api (open-mmlab#121) * add passes * add python api * use python api instead of executable * Merge Master, update optimizer (open-mmlab#151) * [Feature] add yolox ncnn (open-mmlab#29) * add yolox ncnn * add ncnn android performance of yolox * add ut * fix lint * fix None bugs for ncnn * test codecov * test codecov * add device * fix yapf * remove if-else for img shape * use channelshuffle optimize * change benchmark after channelshuffle * fix yapf * fix yapf * fuse continuous reshape * fix static shape deploy * fix code * drop pad * only static shape * fix static * fix docstring * Added mask overlay to output image, changed fprintf info messages to … (open-mmlab#55) * Added mask overlay to output image, changed fprintf info messages to stdout * Improved box filtering (filter area/score), make sure roi coordinates stay within bounds * clang-format * Support UNet in mmseg (open-mmlab#77) * Repeatdataset in train has no CLASSES & PALETTE * update result for unet * update docstring for mmdet * remove ppl for unet in docs * fix ort wrap about input type (open-mmlab#81) * Fix memleak (open-mmlab#86) * delete [] * fix build error when enble MMDEPLOY_ACTIVE_LEVEL * fix lint * [Doc] Nano benchmark and tutorial (open-mmlab#71) * add cls benchmark * add nano zh-cn benchmark and en tutorial * add device row * add doc path to index.rst * fix typo * [Fix] fix missing deploy_core (open-mmlab#80) * fix missing deploy_core * mv flag to demo * target link * [Docs] Fix links in Chinese doc (open-mmlab#84) * Fix docs in Chinese link * Fix links * Delete symbolic link and add links to html * delete files * Fix link * [Feature] Add docker files (open-mmlab#67) * add gpu and cpu dockerfile * fix lint * fix cpu docker and remove redundant * use pip instead * add build arg and readme * fix grammar * update readme * add chinese doc for dockerfile and add docker build to build.md * grammar * refine dockerfiles * add FAQs * update Dpplcv_DIR for SDK building * remove mmcls * add sdk demos * fix typo and lint * update FAQs * [Fix]fix check_env (open-mmlab#101) * fix check_env * update * Replace convert_syncbatchnorm in mmseg (open-mmlab#93) * replace convert_syncbatchnorm with revert_sync_batchnorm from mmcv * change logger * [Doc] Update FAQ for TensorRT (open-mmlab#96) * update FAQ * comment * [Docs]: Update doc for openvino installation (open-mmlab#102) * fix docs * fix docs * fix docs * fix mmcv version * fix docs * rm blank line * simplify non batch nms (open-mmlab#99) * [Enhacement] Allow test.py to save evaluation results (open-mmlab#108) * Add log file * Delete debug code * Rename logger * resolve comments * [Enhancement] Support mmocr v0.4+ (open-mmlab#115) * support mmocr v0.4+ * 0.4.0 -> 0.4.1 * fix onnxruntime wrapper for gpu inference (open-mmlab#123) * fix ncnn wrapper for ort-gpu * resolve comment * fix lint * Fix typo (open-mmlab#132) * lock mmcls version (open-mmlab#131) * [Enhancement] upgrade isort in pre-commit config (open-mmlab#141) * [Enhancement] upgrade isort in pre-commit config by refering to mmflow pr open-mmlab#87 * fix lint * remove .isort.cfg and put its known_third_party to setup.cfg * Fix ci for mmocr (open-mmlab#144) * fix mmocr unittests * remove useless * lock mmdet maximum version to 2.20 * pip install -U numpy * Fix capture_output (open-mmlab#125) Co-authored-by: hanrui1sensetime <83800577+hanrui1sensetime@users.noreply.github.com> Co-authored-by: Johannes L <tehkillerbee@users.noreply.github.com> Co-authored-by: RunningLeon <mnsheng@yeah.net> Co-authored-by: VVsssssk <88368822+VVsssssk@users.noreply.github.com> Co-authored-by: lvhan028 <lvhan_028@163.com> Co-authored-by: AllentDan <41138331+AllentDan@users.noreply.github.com> Co-authored-by: Yifan Zhou <singlezombie@163.com> Co-authored-by: 杨培文 (Yang Peiwen) <915505626@qq.com> Co-authored-by: Semyon Bevzyuk <semen.bevzuk@gmail.com> * configs for all tasks * use torchvision roi align * remote unnecessary code * fix ut * fix ut * export * det dynamic * det dynamic * add ut * fix ut * add ut and docs * fix ut * skip torchscript ut if no ops available * add torchscript option to build.md * update benchmark and resolve comments * resolve conflicts * rename configs * fix mrcnn cuda test * remove useless * add version requirements to docs and comments to codes * enable empty image exporting for torchscript and accelerate ORT inference for MRCNN * rebase * update example for torchscript.md * update FAQs for torchscript.md * resolve comments * only use torchvision roi_align for torchscript * fix ut * use torchvision roi align when pool model is avg * resolve comments Co-authored-by: grimoire <streetyao@live.com> Co-authored-by: grimoire <yaoqian@sensetime.com> Co-authored-by: hanrui1sensetime <83800577+hanrui1sensetime@users.noreply.github.com> Co-authored-by: Johannes L <tehkillerbee@users.noreply.github.com> Co-authored-by: RunningLeon <mnsheng@yeah.net> Co-authored-by: VVsssssk <88368822+VVsssssk@users.noreply.github.com> Co-authored-by: lvhan028 <lvhan_028@163.com> Co-authored-by: Yifan Zhou <singlezombie@163.com> Co-authored-by: 杨培文 (Yang Peiwen) <915505626@qq.com> Co-authored-by: Semyon Bevzyuk <semen.bevzuk@gmail.com> * remove roi_align plugin for ORT (open-mmlab#258) * remove roi_align plugin * remove ut * skip single_roi_extractor UT for ORT in CI * move align to symbolic and update docs * recover UT * resolve comments * add mmcls example * add mmcls/mmdet/mmseg and their corresponding tests * add test data * simplify test data * add requirement in optional.txt * fix setup problem when adding mmrazor requirement * use get_codebase_config * change mmrazor requirement Co-authored-by: AllentDan <41138331+AllentDan@users.noreply.github.com> Co-authored-by: grimoire <streetyao@live.com> Co-authored-by: grimoire <yaoqian@sensetime.com> Co-authored-by: hanrui1sensetime <83800577+hanrui1sensetime@users.noreply.github.com> Co-authored-by: Johannes L <tehkillerbee@users.noreply.github.com> Co-authored-by: RunningLeon <mnsheng@yeah.net> Co-authored-by: VVsssssk <88368822+VVsssssk@users.noreply.github.com> Co-authored-by: lvhan028 <lvhan_028@163.com> Co-authored-by: Yifan Zhou <singlezombie@163.com> Co-authored-by: 杨培文 (Yang Peiwen) <915505626@qq.com> Co-authored-by: Semyon Bevzyuk <semen.bevzuk@gmail.com>
lvhan028 · Jun 3, 2022 · c53a806 · c53a806
1 parent 493ee64
commit c53a806
Show file tree

Hide file tree

Showing 13 changed files with 332 additions and 10 deletions.
diff --git a/mmdeploy/codebase/base/task.py b/mmdeploy/codebase/base/task.py
@@ -7,7 +7,8 @@
 import torch
 from torch.utils.data import DataLoader, Dataset
 
-from mmdeploy.utils import get_backend_config, get_codebase, get_root_logger
+from mmdeploy.utils import (get_backend_config, get_codebase,
+                            get_codebase_config, get_root_logger)
 from mmdeploy.utils.dataset import is_can_sort_dataset, sort_dataset
 
 
@@ -284,3 +285,21 @@ def get_model_name(self) -> str:
             str: the name of the model.
         """
         pass
+
+    @property
+    def from_mmrazor(self) -> bool:
+        """Whether the codebase from mmrazor.
+
+        Returns:
+            bool: From mmrazor or not.
+
+        Raises:
+            TypeError: An error when type of `from_mmrazor` is not boolean.
+        """
+        codebase_config = get_codebase_config(self.deploy_cfg)
+        from_mmrazor = codebase_config.get('from_mmrazor', False)
+        if not isinstance(from_mmrazor, bool):
+            raise TypeError('`from_mmrazor` attribute must be boolean type! '
+                            f'but got: {from_mmrazor}')
+
+        return from_mmrazor
diff --git a/mmdeploy/codebase/mmcls/deploy/classification.py b/mmdeploy/codebase/mmcls/deploy/classification.py
@@ -95,7 +95,11 @@ def init_pytorch_model(self,
             nn.Module: An initialized torch model generated by OpenMMLab
                 codebases.
         """
-        from mmcls.apis import init_model
+        if self.from_mmrazor:
+            from mmrazor.apis import init_mmcls_model as init_model
+        else:
+            from mmcls.apis import init_model
+
         model = init_model(self.model_cfg, model_checkpoint, self.device,
                            cfg_options)
 

diff --git a/mmdeploy/codebase/mmdet/deploy/object_detection.py b/mmdeploy/codebase/mmdet/deploy/object_detection.py
@@ -89,7 +89,11 @@ def init_pytorch_model(self,
             nn.Module: An initialized torch model generated by other OpenMMLab
                 codebases.
         """
-        from mmdet.apis import init_detector
+        if self.from_mmrazor:
+            from mmrazor.apis import init_mmdet_model as init_detector
+        else:
+            from mmdet.apis import init_detector
+
         model = init_detector(self.model_cfg, model_checkpoint, self.device,
                               cfg_options)
         return model.eval()

diff --git a/mmdeploy/codebase/mmseg/deploy/segmentation.py b/mmdeploy/codebase/mmseg/deploy/segmentation.py
@@ -89,7 +89,11 @@ def init_pytorch_model(self,
                 codebases.
         """
         from mmcv.cnn.utils import revert_sync_batchnorm
-        from mmseg.apis import init_segmentor
+        if self.from_mmrazor:
+            from mmrazor.apis import init_mmseg_model as init_segmentor
+        else:
+            from mmseg.apis import init_segmentor
+
         model = init_segmentor(self.model_cfg, model_checkpoint, self.device)
         model = revert_sync_batchnorm(model)
         return model.eval()

diff --git a/requirements/optional.txt b/requirements/optional.txt
@@ -3,6 +3,7 @@ mmdet>=2.19.0,<=2.20.0
 mmedit
 mmocr>=0.3.0,<=0.4.1
 mmpose>=0.24.0,<=0.25.1
+mmrazor>=0.3.0
 mmsegmentation
 onnxruntime>=1.8.0
 openvino-dev
diff --git a/tests/test_codebase/test_mmcls/data/mmrazor_model.py b/tests/test_codebase/test_mmcls/data/mmrazor_model.py
@@ -0,0 +1,31 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+_base_ = 'model.py'
+
+norm_cfg = dict(type='BN')
+
+mutator = dict(
+    type='OneShotMutator',
+    placeholder_mapping=dict(
+        all_blocks=dict(
+            type='OneShotOP',
+            choices=dict(
+                shuffle_3x3=dict(
+                    type='ShuffleBlock', kernel_size=3, norm_cfg=norm_cfg),
+                shuffle_5x5=dict(
+                    type='ShuffleBlock', kernel_size=5, norm_cfg=norm_cfg),
+                shuffle_7x7=dict(
+                    type='ShuffleBlock', kernel_size=7, norm_cfg=norm_cfg),
+                shuffle_xception=dict(
+                    type='ShuffleXception', norm_cfg=norm_cfg),
+            ))))
+
+algorithm = dict(
+    type='SPOS',
+    architecture=dict(
+        type='MMClsArchitecture',
+        model={{_base_.model}},
+    ),
+    mutator=mutator,
+    distiller=None,
+    mutable_cfg='tests/test_codebase/test_mmcls/data/mmrazor_mutable_cfg.yaml',
+    retraining=True)
diff --git a/tests/test_codebase/test_mmcls/data/mmrazor_mutable_cfg.yaml b/tests/test_codebase/test_mmcls/data/mmrazor_mutable_cfg.yaml
@@ -0,0 +1,60 @@
+stage_0_block_0:
+  chosen:
+  - shuffle_7x7
+stage_0_block_1:
+  chosen:
+  - shuffle_5x5
+stage_0_block_2:
+  chosen:
+  - shuffle_3x3
+stage_0_block_3:
+  chosen:
+  - shuffle_5x5
+stage_1_block_0:
+  chosen:
+  - shuffle_7x7
+stage_1_block_1:
+  chosen:
+  - shuffle_3x3
+stage_1_block_2:
+  chosen:
+  - shuffle_7x7
+stage_1_block_3:
+  chosen:
+  - shuffle_3x3
+stage_2_block_0:
+  chosen:
+  - shuffle_7x7
+stage_2_block_1:
+  chosen:
+  - shuffle_3x3
+stage_2_block_2:
+  chosen:
+  - shuffle_7x7
+stage_2_block_3:
+  chosen:
+  - shuffle_xception
+stage_2_block_4:
+  chosen:
+  - shuffle_3x3
+stage_2_block_5:
+  chosen:
+  - shuffle_3x3
+stage_2_block_6:
+  chosen:
+  - shuffle_3x3
+stage_2_block_7:
+  chosen:
+  - shuffle_3x3
+stage_3_block_0:
+  chosen:
+  - shuffle_xception
+stage_3_block_1:
+  chosen:
+  - shuffle_7x7
+stage_3_block_2:
+  chosen:
+  - shuffle_xception
+stage_3_block_3:
+  chosen:
+  - shuffle_xception
diff --git a/tests/test_codebase/test_mmcls/test_classification.py b/tests/test_codebase/test_mmcls/test_classification.py
@@ -1,6 +1,8 @@
 # Copyright (c) OpenMMLab. All rights reserved.
+import copy
 import os
 from tempfile import NamedTemporaryFile, TemporaryDirectory
+from typing import Any
 
 import mmcv
 import numpy as np
@@ -37,9 +39,33 @@
 img = np.random.rand(*img_shape, 3)
 
 
-def test_init_pytorch_model():
+@pytest.mark.parametrize('from_mmrazor', [True, False, '123', 0])
+def test_init_pytorch_model(from_mmrazor: Any):
     from mmcls.models.classifiers.base import BaseClassifier
-    model = task_processor.init_pytorch_model(None)
+    if from_mmrazor is False:
+        _task_processor = task_processor
+    else:
+        _model_cfg_path = 'tests/test_codebase/test_mmcls/data/' \
+            'mmrazor_model.py'
+        _model_cfg = load_config(_model_cfg_path)[0]
+        _model_cfg.algorithm.architecture.model.type = 'mmcls.ImageClassifier'
+        _model_cfg.algorithm.architecture.model.backbone = dict(
+            type='SearchableShuffleNetV2', widen_factor=1.0)
+        _deploy_cfg = copy.deepcopy(deploy_cfg)
+        _deploy_cfg.codebase_config['from_mmrazor'] = from_mmrazor
+        _task_processor = build_task_processor(_model_cfg, _deploy_cfg, 'cpu')
+
+    if not isinstance(from_mmrazor, bool):
+        with pytest.raises(
+                TypeError,
+                match='`from_mmrazor` attribute must be '
+                'boolean type! '
+                f'but got: {from_mmrazor}'):
+            _ = _task_processor.from_mmrazor
+        return
+    assert from_mmrazor == _task_processor.from_mmrazor
+
+    model = _task_processor.init_pytorch_model(None)
     assert isinstance(model, BaseClassifier)
 
 

diff --git a/tests/test_codebase/test_mmdet/data/mmrazor_model.py b/tests/test_codebase/test_mmdet/data/mmrazor_model.py
@@ -0,0 +1,34 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+_base_ = 'model.py'
+
+norm_cfg = dict(type='BN', requires_grad=True)
+mutator = dict(
+    type='OneShotMutator',
+    placeholder_mapping=dict(
+        all_blocks=dict(
+            type='OneShotOP',
+            choices=dict(
+                shuffle_3x3=dict(
+                    type='ShuffleBlock', norm_cfg=norm_cfg, kernel_size=3),
+                shuffle_5x5=dict(
+                    type='ShuffleBlock', norm_cfg=norm_cfg, kernel_size=5),
+                shuffle_7x7=dict(
+                    type='ShuffleBlock', norm_cfg=norm_cfg, kernel_size=7),
+                shuffle_xception=dict(
+                    type='ShuffleXception',
+                    norm_cfg=norm_cfg,
+                ),
+            ))))
+
+algorithm = dict(
+    type='DetNAS',
+    architecture=dict(
+        type='MMDetArchitecture',
+        model={{_base_.model}},
+    ),
+    mutator=mutator,
+    pruner=None,
+    distiller=None,
+    retraining=True,
+    mutable_cfg='tests/test_codebase/test_mmdet/data/mmrazor_mutable_cfg.yaml',
+)
diff --git a/tests/test_codebase/test_mmdet/data/mmrazor_mutable_cfg.yaml b/tests/test_codebase/test_mmdet/data/mmrazor_mutable_cfg.yaml
@@ -0,0 +1,60 @@
+stage_0_block_0:
+  chosen:
+  - shuffle_7x7
+stage_0_block_1:
+  chosen:
+  - shuffle_5x5
+stage_0_block_2:
+  chosen:
+  - shuffle_7x7
+stage_0_block_3:
+  chosen:
+  - shuffle_3x3
+stage_1_block_0:
+  chosen:
+  - shuffle_7x7
+stage_1_block_1:
+  chosen:
+  - shuffle_5x5
+stage_1_block_2:
+  chosen:
+  - shuffle_5x5
+stage_1_block_3:
+  chosen:
+  - shuffle_7x7
+stage_2_block_0:
+  chosen:
+  - shuffle_xception
+stage_2_block_1:
+  chosen:
+  - shuffle_xception
+stage_2_block_2:
+  chosen:
+  - shuffle_5x5
+stage_2_block_3:
+  chosen:
+  - shuffle_xception
+stage_2_block_4:
+  chosen:
+  - shuffle_3x3
+stage_2_block_5:
+  chosen:
+  - shuffle_3x3
+stage_2_block_6:
+  chosen:
+  - shuffle_xception
+stage_2_block_7:
+  chosen:
+  - shuffle_5x5
+stage_3_block_0:
+  chosen:
+  - shuffle_xception
+stage_3_block_1:
+  chosen:
+  - shuffle_5x5
+stage_3_block_2:
+  chosen:
+  - shuffle_xception
+stage_3_block_3:
+  chosen:
+  - shuffle_7x7
diff --git a/tests/test_codebase/test_mmdet/test_object_detection.py b/tests/test_codebase/test_mmdet/test_object_detection.py
@@ -1,6 +1,8 @@
 # Copyright (c) OpenMMLab. All rights reserved.
+import copy
 import os
 from tempfile import NamedTemporaryFile, TemporaryDirectory
+from typing import Any
 
 import mmcv
 import numpy as np
@@ -48,9 +50,33 @@
 img = np.random.rand(*img_shape, 3)
 
 
-def test_init_pytorch_model():
+@pytest.mark.parametrize('from_mmrazor', [True, False, '123', 0])
+def test_init_pytorch_model(from_mmrazor: Any):
     from mmdet.models import BaseDetector
-    model = task_processor.init_pytorch_model(None)
+    if from_mmrazor is False:
+        _task_processor = task_processor
+    else:
+        _model_cfg_path = 'tests/test_codebase/test_mmdet/data/' \
+            'mmrazor_model.py'
+        _model_cfg = load_config(_model_cfg_path)[0]
+        _model_cfg.algorithm.architecture.model.type = 'mmdet.YOLOV3'
+        _model_cfg.algorithm.architecture.model.backbone.type = \
+            'mmcls.SearchableShuffleNetV2'
+        _deploy_cfg = copy.deepcopy(deploy_cfg)
+        _deploy_cfg.codebase_config['from_mmrazor'] = from_mmrazor
+        _task_processor = build_task_processor(_model_cfg, _deploy_cfg, 'cpu')
+
+    if not isinstance(from_mmrazor, bool):
+        with pytest.raises(
+                TypeError,
+                match='`from_mmrazor` attribute must be '
+                'boolean type! '
+                f'but got: {from_mmrazor}'):
+            _ = _task_processor.from_mmrazor
+        return
+    assert from_mmrazor == _task_processor.from_mmrazor
+
+    model = _task_processor.init_pytorch_model(None)
     assert isinstance(model, BaseDetector)
 
 

diff --git a/tests/test_codebase/test_mmseg/data/mmrazor_model.py b/tests/test_codebase/test_mmseg/data/mmrazor_model.py
@@ -0,0 +1,28 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+_base_ = 'model.py'
+
+# algorithm setting
+algorithm = dict(
+    type='GeneralDistill',
+    architecture=dict(
+        type='MMSegArchitecture',
+        model={{_base_.model}},
+    ),
+    distiller=dict(
+        type='SingleTeacherDistiller',
+        teacher={{_base_.model}},
+        teacher_trainable=False,
+        components=[
+            dict(
+                student_module='decode_head.conv_seg',
+                teacher_module='decode_head.conv_seg',
+                losses=[
+                    dict(
+                        type='ChannelWiseDivergence',
+                        name='loss_cwd_logits',
+                        tau=1,
+                        loss_weight=5,
+                    )
+                ])
+        ]),
+)