Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support DyHead #6823

Merged
merged 19 commits into from
Feb 22, 2022
Merged

[Feature] Support DyHead #6823

merged 19 commits into from
Feb 22, 2022

Conversation

shinya7y
Copy link
Contributor

@shinya7y shinya7y commented Dec 17, 2021

Motivation

This PR aims to support DyHead (CVPR 2021).
Dynamic Head: Unifying Object Detection Heads with Attentions https://arxiv.org/abs/2106.08322
This will close #6634

I trained ATSS+DyHead (R-50) only. Please train other models if needed.

Figure for README.md:
DyHead_fig1

Checklist

  • Add DYReLU
  • Add DyHead
  • Add docstring
  • Add unit tests
  • Add method's README.md
  • Add metafile.yml and update model-index.yml
  • Update README.md, README_zh-CN.md
  • Align inference accuracy of ATSS+DyHead (R-50)
    converted weight
    box AP: 42.3 at 85ed81f (I think this is reasonable alignment without the need for changing existing code much.)
  • Align training accuracy of ATSS+DyHead (R-50)
Method Backbone code config lr batch size conv_offset box AP
ATSS+DyHead R-50 mmdet (8011283) simple 0.01 16 (4x4) zero init 43.3
ATSS+DyHead R-50 mmdet (c72c660) repro 0.01 16 (4x4) PyTorch default 42.5
ATSS+DyHead R-50 Detectron2 - 0.01 16 (8?x2?) PyTorch default 42.4
ATSS+DyHead R-50 mrbench (paper) - 0.02 ? (8x?) ? 42.6

@codecov
Copy link

codecov bot commented Dec 17, 2021

Codecov Report

Merging #6823 (3a5fbe1) into dev (ffff556) will increase coverage by 0.13%.
The diff coverage is 95.09%.

Impacted file tree graph

@@            Coverage Diff             @@
##              dev    #6823      +/-   ##
==========================================
+ Coverage   62.46%   62.59%   +0.13%     
==========================================
  Files         330      332       +2     
  Lines       26215    26443     +228     
  Branches     4436     4472      +36     
==========================================
+ Hits        16376    16553     +177     
- Misses       8968     9007      +39     
- Partials      871      883      +12     
Flag Coverage Δ
unittests 62.57% <95.09%> (+0.12%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmdet/models/utils/se_layer.py 90.24% <90.90%> (+0.77%) ⬆️
mmdet/models/necks/dyhead.py 96.05% <96.05%> (ø)
mmdet/models/dense_heads/atss_head.py 89.15% <100.00%> (+0.13%) ⬆️
mmdet/models/necks/__init__.py 100.00% <100.00%> (ø)
mmdet/models/utils/__init__.py 100.00% <100.00%> (ø)
mmdet/core/visualization/image.py 68.10% <0.00%> (-11.10%) ⬇️
mmdet/models/roi_heads/bbox_heads/sabl_head.py 76.11% <0.00%> (-0.33%) ⬇️
mmdet/core/mask/structures.py 80.43% <0.00%> (-0.01%) ⬇️
mmdet/datasets/utils.py 75.60% <0.00%> (ø)
mmdet/core/evaluation/__init__.py 100.00% <0.00%> (ø)
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ffff556...3a5fbe1. Read the comment docs.

@ZwwWayne ZwwWayne requested a review from RangiLyu December 22, 2021 16:23
@ZwwWayne ZwwWayne added the WIP Working in progress label Dec 22, 2021
@jshilong
Copy link
Collaborator

jshilong commented Jan 5, 2022

Thanks for your amazing work.
Does the table mean this version is higher than the original implementation?

@shinya7y
Copy link
Contributor Author

shinya7y commented Jan 5, 2022

Yes, the version (8011283) achieves higher AP.
Although the config configs/dyhead/atss_r50_fpn_dyhead_1x_coco.py will work as a minimum example to use DyHead in mmdet, it has many implementation differences. I will add another config to align accuracy.

@shinya7y shinya7y changed the base branch from master to dev January 12, 2022 22:00
@shinya7y shinya7y changed the title [WIP] Support DyHead [Feature] Support DyHead Jan 12, 2022
@RangiLyu RangiLyu added enhancement New feature or request planned feature and removed WIP Working in progress labels Jan 13, 2022
Copy link
Member

@RangiLyu RangiLyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. By the way, should the models be uploaded to the openmmlab server before merging? @ZwwWayne

@hhaAndroid
Copy link
Collaborator

@shinya7y Please resolve the conflict.

@hhaAndroid
Copy link
Collaborator

@shinya7y Please resolve the conflict.

@ZwwWayne
Copy link
Collaborator

LGTM. By the way, should the models be uploaded to the openmmlab server before merging? @ZwwWayne

Yes, we can do that. Is there any link so that I can download them and upload them to our model zoo server?

@@ -0,0 +1,111 @@
_base_ = [
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name for_reproduction might be confusing. Can we keep the naming convention and rename it as atss_r50_caffe_fpn_dyhead_1x_coco.py?

in_channels=256,
out_channels=256,
num_blocks=6,
zero_init_offset=False)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest leaving comments to indicate it is different from that of the pytorch backbone.

type='DeltaXYWHBBoxCoder',
target_means=[.0, .0, .0, .0],
target_stds=[0.1, 0.1, 0.2, 0.2]),
loss_cls=dict(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggest leaving comments to indicate it is different from that using Pytorch backbone

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should also comment other places if they are different from the config of using PyTorch backbone,

@ZwwWayne
Copy link
Collaborator

Hi @shinya7y ,
Thanks for the great work. This PR can be merged after resolving the last several comments. Let's make it in this release~

@shinya7y
Copy link
Contributor Author

LGTM. By the way, should the models be uploaded to the openmmlab server before merging? @ZwwWayne

Yes, we can do that. Is there any link so that I can download them and upload them to our model zoo server?

As written in README.md, the model weights are
https://github.com/shinya7y/weights/releases/download/v1.0.0/atss_r50_fpn_dyhead_for_reproduction_4x4_1x_coco_20220107_213939-162888e6.pth
https://github.com/shinya7y/weights/releases/download/v1.0.0/atss_r50_fpn_dyhead_4x4_1x_coco_20211219_023314-eaa620c6.pth

I will update the links if you could let me know new URLs.


| Method | Backbone | Style | Setting | Lr schd | Mem (GB) | Inf time (fps) | box AP | Config | Download |
|:------:|:--------:|:-------:|:------------:|:-------:|:--------:|:--------------:|:------:|:------:|:--------:|
| ATSS | R-50 | caffe | reproduction | 1x | 5.4 | 13.2 | 42.5 | [config](./atss_r50_fpn_dyhead_for_reproduction_1x_coco.py) | [model](https://github.com/shinya7y/weights/releases/download/v1.0.0/atss_r50_fpn_dyhead_for_reproduction_4x4_1x_coco_20220107_213939-162888e6.pth) &#124; [log](https://github.com/shinya7y/weights/releases/download/v1.0.0/atss_r50_fpn_dyhead_for_reproduction_4x4_1x_coco_20220107_213939.log.json) |
Copy link
Collaborator

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is https://download.openmmlab.com better?

| Method | Backbone | Style | Setting | Lr schd | Mem (GB) | Inf time (fps) | box AP | Config | Download |
|:------:|:--------:|:-------:|:------------:|:-------:|:--------:|:--------------:|:------:|:------:|:--------:|
| ATSS | R-50 | caffe | reproduction | 1x | 5.4 | 13.2 | 42.5 | [config](./atss_r50_fpn_dyhead_for_reproduction_1x_coco.py) | [model](https://github.com/shinya7y/weights/releases/download/v1.0.0/atss_r50_fpn_dyhead_for_reproduction_4x4_1x_coco_20220107_213939-162888e6.pth) &#124; [log](https://github.com/shinya7y/weights/releases/download/v1.0.0/atss_r50_fpn_dyhead_for_reproduction_4x4_1x_coco_20220107_213939.log.json) |
| ATSS | R-50 | pytorch | simple | 1x | 4.9 | 13.7 | 43.3 | [config](./atss_r50_fpn_dyhead_1x_coco.py) | [model](https://github.com/shinya7y/weights/releases/download/v1.0.0/atss_r50_fpn_dyhead_4x4_1x_coco_20211219_023314-eaa620c6.pth) &#124; [log](https://github.com/shinya7y/weights/releases/download/v1.0.0/atss_r50_fpn_dyhead_4x4_1x_coco_20211219_023314.log.json) |
Copy link
Collaborator

@ZwwWayne
Copy link
Collaborator

Models have been uploaded.

@ZwwWayne ZwwWayne merged commit 0b205c6 into open-mmlab:dev Feb 22, 2022
ZwwWayne added a commit that referenced this pull request Feb 26, 2022
* [Enhancement] Upgrade isort in pre-commit hook (#7130)

* upgrade isort to v5.10.1

* replace known_standard_library with extra_standard_library

* upgrade isort to v5.10.1

replace known_standard_library with extra_standard_library

* imports order changes

* [Fix] cannot to save the best checkpoint when the key_score is None (#7101)

* [Fix] Fix MixUp transform filter boxes failing case. Added test case (#7080)

* [Fix] Update the version limitation of mmcv-full and pytorch in CI. (#7133)

* Update

* Update build.yml

* Update build.yml

* [Feature] Support TIMMBackbone (#7020)

* add TIMMBackbone

based on
open-mmlab/mmpretrain#427
open-mmlab/mmsegmentation#998

* update and clean

* fix unit test

* Revert

* add example configs

* Create 2_new_data_model.md (#6476)

fix some typo

Co-authored-by: PJLAB\huanghaian <1286304229@qq.com>

* [FIX] add Ci of pytorch 1.10 and comments for bbox clamp (#7081) (#7083)

* add comments for bbox clamp

* add CI of pytorch1.10

* add ci of pytorch1.10.1

* mmcv1.9.0->mmcv1.9

* add ci of pytorch1.10

* Add daily issue owners (#7163)

* Add code owners

Signed-off-by: del-zhenwu <dele.zhenwu@gmail.com>

* Update code owners

Signed-off-by: del-zhenwu <dele.zhenwu@gmail.com>

* [Feature] Support visualization for Panoptic Segmentation (#7041)

* First commit of v2

* split the functions

* Support to show panoptic result

* temp

* Support to show gt

* support show gt

* fix lint

* Support to browse datasets

* Fix unit tests

* Fix findContours

* fix comments

* Fix pre-commit

* fix lint

* Add the type of an argument

* [Fix] confusion_matrix.py analysis tool handling NaNs (#7147)

* [Fix] Added missing property in SABLHead (#7091)

* Added missing property in SABLHead

* set pre-commit-hooks to v0.1.0

* set maskdownlint to v0.11.0

* pre-commit-hooks

Co-authored-by: Cedric Luo <luochunhua1996@outlook.com>

* Update config.md (#7215)

* [Fix] Fix wrong img name in onnx2tensorrt.py (#7157)

* [Docs] fix albumentations installed way (#7143)

* Update config.md

fix some typos

Co-authored-by: Jamie <jamiechoi1995@users.noreply.github.com>
Co-authored-by: BigDong <yudongwang@tju.edu.cn>

* [Feature] Support DyHead (#6823)

* add DyHead

* move and update DYReLU

* update

* replace stack with sum to reduce memory

* clean and update

* update to align inference accuracy (incomplete)

* fix pad

* update to align training accuracy and pick #6867

* add README and metafile

* update docs

* resolve comments

* revert picking 6867

* update README.md

* update metafile.yml

* resolve comments and update urls

* Fix broken colab link (#7218)

* [Fix] Fix wrong img name in onnx2tensorrt.py (#7157)

* [Docs] fix albumentations installed way (#7143)

* Fix broken colab link

Co-authored-by: Jamie <jamiechoi1995@users.noreply.github.com>
Co-authored-by: BigDong <yudongwang@tju.edu.cn>

* Remove the inplace addition in `FPN` (#7175)

* [Fix] Fix wrong img name in onnx2tensorrt.py (#7157)

* [Docs] fix albumentations installed way (#7143)

* Remove the inplace addition in `FPN`

* update

Co-authored-by: Jamie <jamiechoi1995@users.noreply.github.com>
Co-authored-by: BigDong <yudongwang@tju.edu.cn>
Co-authored-by: PJLAB\huanghaian <1286304229@qq.com>

* [Feature] Support OpenImages Dataset (#6331)

* [Feature] support openimage group of eval

* [Feature] support openimage group of eval

* support openimage dataset

* support openimage challenge dataset

* fully support OpenImages-V6 and OpenImages Challenge 2019

* Fix some logic error

* update config file

* fix get data_infos error

* fully support OpenImages evaluation

* update OpenImages config files

* [Feature] support OpenImages datasets

* fix bug

* support load image metas from pipeline

* fix bug

* fix get classes logic error

* update code

* support get image metas

* support openimags

* support collect image metas

* support Open Images

* fix openimages logic

* minor fix

* add a new function to compute openimages tpfp

* minor fix

* fix ci error

* minor fix

* fix indication

* minor fix

* fix returns

* fix returns

* fix returns

* fix returns

* fix returns

* minor fix

* update readme

* support loading image level labels and fix some logic

* minor fix

* minor fix

* add class names

* minor fix

* minor fix

* minor fix

* add openimages test unit

* minor fix

* minor fix

* fix test unit

* minor fix

* fix logic error

* minor fix

* fully support openimages

* minor fix

* fix docstring

* fix docstrings in readthedocs

* update get image metas script

* label_description_file -> label_file

* update openimages readme

* fix test unit

* fix test unit

* minor fix

* update readme file

* Update get_image_metas.py

* [Enhance] Speed up SimOTA matching. (#7098)

* [Feature] Add Maskformer to mmdet (#7212)

* first commit

* add README

* move model description from config to readme

add description for binary_input

add description for dice loss

add a independent panoptic gt processing function

add a independent panoptic gt processing function

remove compatibility of pretrain in maskformer

* update comments in maskformer_head

* update docs format

* Add deprecation message for deploy tools (#7242)

* Add CI for windows (#7228)

* [Fix] Fix wrong img name in onnx2tensorrt.py (#7157)

* [Docs] fix albumentations installed way (#7143)

* Add mmrotate (#7216)

* fix description for args in CSPDarknet (#7187)

* Update build.yml

* Update build.yml

* Update build.yml

* Update build.yml

* Update build.yml

* Update build.yml

* Update build.yml

* Update build.yml

* Update build.yml

* Update build.yml

* Update build.yml

* Update build.yml

* fix test_find_latest_checkpoint

* fix data_infos__default_db_directories

* fix test_custom_classes_override_default

* Update test_custom_dataset.py

* Update test_common.py

* Update assign_result.py

* use os.path.join

* fix bug

* Update test_common.py

* Update assign_result.py

* Update sampling_result.py

* os.path -> osp

* os.path -> osp

Co-authored-by: Jamie <jamiechoi1995@users.noreply.github.com>
Co-authored-by: BigDong <yudongwang@tju.edu.cn>
Co-authored-by: Hyeokjoon Kwon <dsazz801@gmail.com>

* add Chinese version of init_cfg (#7188)

* [Fix] Fix wrong img name in onnx2tensorrt.py (#7157)

* [Docs] fix albumentations installed way (#7143)

* Create init_cfg.md

* Update docs/zh_cn/tutorials/init_cfg.md

Co-authored-by: Haian Huang(深度眸) <1286304229@qq.com>

* update init_cfg.md

* update init_cfg.md

* update init_cfg.md

* update init_cfg.md

Co-authored-by: Jamie <jamiechoi1995@users.noreply.github.com>
Co-authored-by: BigDong <yudongwang@tju.edu.cn>
Co-authored-by: Haian Huang(深度眸) <1286304229@qq.com>

* update MaskFormer readme and docs (#7241)

* update docs for maskformer

* update readme

* update readme format

* update link

* update json link

* update format of ConfigDict

* update format of function returns

* uncomment main in deployment/test.py

* [Feature] ResNet Strikes Back. (#7001)

* [Feature] ResNet Strikes Back.

* add more cfg

* add readme

* update

* update

* update

* update

* update

* update

* Maskformer Visualization (#7247)

* maskformer visualization

* compatible with models that do not contain arg of pretrained

* compatible with models that do not contain arg of pretrained

* Bump versions to v2.22.0 (#7240)

* Bump versions to v2.22.0

* Fix comments and add the latest PRs

* fix the id of contributor

* relax the version of mmcv

* Add ResNet Strikes Back

* Update README_zh-CN.md

* Update README.md

* fix typo

* Update README_zh-CN.md

Co-authored-by: Wenwei Zhang <40779233+ZwwWayne@users.noreply.github.com>

* Maskformer metafile and rsb readme format (#7250)

* [Fix] Fix Open Images testunit to avoid error in Windows CI (#7252)

* [Feature] support openimage group of eval

* [Feature] support openimage group of eval

* support openimage dataset

* support openimage challenge dataset

* fully support OpenImages-V6 and OpenImages Challenge 2019

* Fix some logic error

* update config file

* fix get data_infos error

* fully support OpenImages evaluation

* update OpenImages config files

* [Feature] support OpenImages datasets

* fix bug

* support load image metas from pipeline

* fix bug

* fix get classes logic error

* update code

* support get image metas

* support openimags

* support collect image metas

* support Open Images

* fix openimages logic

* minor fix

* add a new function to compute openimages tpfp

* minor fix

* fix ci error

* minor fix

* fix indication

* minor fix

* fix returns

* fix returns

* fix returns

* fix returns

* fix returns

* minor fix

* update readme

* support loading image level labels and fix some logic

* minor fix

* minor fix

* add class names

* minor fix

* minor fix

* minor fix

* add openimages test unit

* minor fix

* minor fix

* fix test unit

* minor fix

* fix logic error

* minor fix

* fully support openimages

* minor fix

* fix docstring

* fix docstrings in readthedocs

* update get image metas script

* label_description_file -> label_file

* update openimages readme

* fix test unit

* fix test unit

* minor fix

* update readme file

* Update get_image_metas.py

* fix oid testunit to avoid some error in windows

* minor fix to avoid some error in windows

* minor fix

* add comments in oid test unit

* minor fix

Co-authored-by: Cedric Luo <luochunhua1996@outlook.com>
Co-authored-by: LuooChen <33743370+LuooChen@users.noreply.github.com>
Co-authored-by: Daniel van Sabben Alsina <13849741+dvansa@users.noreply.github.com>
Co-authored-by: jbwang1997 <jbwang1997@gmail.com>
Co-authored-by: Yosuke Shinya <42844407+shinya7y@users.noreply.github.com>
Co-authored-by: siatwangmin <csustwangmin@163.com>
Co-authored-by: PJLAB\huanghaian <1286304229@qq.com>
Co-authored-by: del-zhenwu <dele.zhenwu@gmail.com>
Co-authored-by: Guangchen Lin <347630870@qq.com>
Co-authored-by: VIKASH RANJAN <ranjanvikash26@gmail.com>
Co-authored-by: Range King <RangeKingHZ@gmail.com>
Co-authored-by: Jamie <jamiechoi1995@users.noreply.github.com>
Co-authored-by: BigDong <yudongwang@tju.edu.cn>
Co-authored-by: Haofan Wang <frankmiracle@outlook.com>
Co-authored-by: Zhijian Liu <zhijianliu.cs@gmail.com>
Co-authored-by: BigDong <yudongwang1226@gmail.com>
Co-authored-by: RangiLyu <lyuchqi@gmail.com>
Co-authored-by: Yue Zhou <592267829@qq.com>
Co-authored-by: Hyeokjoon Kwon <dsazz801@gmail.com>
Co-authored-by: Kevin Ye <1752391457@qq.com>
ZwwWayne pushed a commit that referenced this pull request Jul 18, 2022
* add DyHead

* move and update DYReLU

* update

* replace stack with sum to reduce memory

* clean and update

* update to align inference accuracy (incomplete)

* fix pad

* update to align training accuracy and pick #6867

* add README and metafile

* update docs

* resolve comments

* revert picking 6867

* update README.md

* update metafile.yml

* resolve comments and update urls
ZwwWayne pushed a commit to ZwwWayne/mmdetection that referenced this pull request Jul 19, 2022
* add DyHead

* move and update DYReLU

* update

* replace stack with sum to reduce memory

* clean and update

* update to align inference accuracy (incomplete)

* fix pad

* update to align training accuracy and pick open-mmlab#6867

* add README and metafile

* update docs

* resolve comments

* revert picking 6867

* update README.md

* update metafile.yml

* resolve comments and update urls
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request planned feature
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

dynamic head
5 participants