[Feature] Support Segmenter #955

rstrudel · 2021-10-12T13:44:56Z

Motivation

Support Segmenter . Copy of #952 not using the master branch.

Modification

I added configuration files to train segmenter on ADE20K and Cityscapes. I also added a script to convert the original ViT checkpoints in JAX to checkpoints compatible with the ViT class of mmsegmentation.

To be done

use img_norm_cfg=[127.5, 127.5, 127.5] as default for ViT checkpoints
checkpoints, I reported the performances in the readme for the ones I trained

Update results (2022-01-06)

config		mIoU (SS)	Official Repo mIoU (SS/MS)	Paddleseg results mIoU (SS)
segmenter_vit-t_mask_8x1_512x512_160k_ade20k	mode='slide', crop_size=(512, 512), stride=(480, 480)	39.99	38.1 / 38.8	Not shown
segmenter_vit-s_linear_8x1_512x512_160k_ade20k	mode='slide', crop_size=(512, 512), stride=(480, 480)	45.75	Not shown	45.48
segmenter_vit-s_mask_8x1_512x512_160k_ade20k	mode='slide', crop_size=(512, 512), stride=(480, 480)	46.19	45.3 / 46.9	45.15
segmenter_vit-b_mask_8x1_512x512_160k_ade20k	mode='slide', crop_size=(512, 512), stride=(480, 480)	49.6	48.5 / 50.0	48.49
segmenter_vit-l_mask_8x1_512x512_160k_ade20k	mode='slide', crop_size=(640, 640), stride=(608, 608)	52.16	51.8 / 53.6	Not shown

MengzhangLI · 2021-10-12T14:02:36Z

Hi, @rstrudel thank you very much for your nice PR.

For the convenience of your time, I would review and modify your code in your forked branch to enable it concord on our code base style. I try to finish first modification (For example, our docstring and pre-commit standard) in the next week, let's keep contact.

Thanks again and wish you all the best.

Junjun2016 · 2021-10-24T18:57:38Z

Hi @rstrudel
The CI has been fixed, please merge the master branch.
Could you please also update this PR according to previous comments?

rstrudel · 2021-10-25T08:26:46Z

Hi @Junjun2016 ,
Thanks, I will work on this during the week.
Robin

RockeyCoss · 2021-11-15T08:23:12Z

Please merge the master branch into your branch, thank you.

rstrudel · 2021-11-15T15:19:02Z

Hi @RockeyCoss ,
It should be done now. Sorry for the delay, I should be able to help after CVPR deadline.
Robin

MengzhangLI · 2021-11-15T18:11:42Z

Hi @RockeyCoss , It should be done now. Sorry for the delay, I should be able to help after CVPR deadline. Robin

It's fine, Robin.

Can't wait to read your paper submitted to CVPR'22 and wish it would be accepted.

Best,

MengzhangLI · 2021-11-24T13:13:12Z

Hi, Robin!

What's up. Do you plan to continue finishing this pr? Feel free to contact us anytime!

Best,

rstrudel · 2021-11-25T10:23:28Z

Hi @MengzhangLI ,
I am on it, I have time now and I will ping you if have questions. I will update the PR soon and check with you what's missing.
Best,
Robin

rstrudel · 2021-11-25T12:44:56Z

Is it possible to set
img_norm_cfg=dict(mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=True)
globally ? E.g. if you use Segmenter then this will be the default normalization before passing an input to the backbone.

MengzhangLI · 2021-11-25T12:54:57Z

Is it possible to set img_norm_cfg=dict(mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=True) globally ? E.g. if you use Segmenter then this will be the default normalization before passing an input to the backbone.

Hi, @rstrudel . It is not easy to change this parameter in MMSegmentation becuase img_norm_cfg is a dict which contains list. So it can not be passed easily like other pure dict parameters.

If dict(mean=[127.5, 127.5, 127.5], std=[127.5, 127.5, 127.5], to_rgb=True) is default setting of your pretrained model, can we set it specifically in model config file like upernet_swin_tiny_patch4_window7_512x512_160k_ade20k_pretrain_224x224_1K.py? You can see many new values of those parameters.

Best,

configs/segmenter/segmenter_vit-l_mask_8x1_512x512_160k_ade20k.py

configs/segmenter/segmenter_vit-s_linear_8x1_512x512_160k_ade20k.py

configs/segmenter/segmenter_vit-s_mask_8x1_512x512_160k_ade20k.py

configs/segmenter/segmenter_vit-t_mask_8x1_512x512_160k_ade20k.py

Junjun2016

LGTM, except for some minor revision.

xvjiarui

LGTM except for the comment for the convert script.

tools/model_converters/vitjax2mmseg.py

…into dev_mmseg

* segmenter: add model * update * readme: update * config: update * segmenter: update readme * segmenter: update * segmenter: update * segmenter: update * configs: set checkpoint path to pretrain folder * segmenter: modify vit-s/lin, remove data config * rreadme: update * configs: transfer from _base_ to segmenter * configs: add 8x1 suffix * configs: remove redundant lines * configs: cleanup * first attempt * swipe CI error * Update mmseg/models/decode_heads/__init__.py Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn> * segmenter_linear: use fcn backbone * segmenter_mask: update * models: add segmenter vit * decoders: yapf+remove unused imports * apply precommit * segmenter/linear_head: fix * segmenter/linear_header: fix * segmenter: fix mask transformer * fix error * segmenter/mask_head: use trunc_normal init * refactor segmenter head * Fetch upstream (#1) * [Feature] Change options to cfg-option (open-mmlab#1129) * [Feature] Change option to cfg-option * add expire date and fix the docs * modify docstring * [Fix] Add  in metafile open-mmlab#1127 * [Fix] Fix correct num_classes of HRNet in LoveDA dataset open-mmlab#1136 * Bump to v0.20.1 (open-mmlab#1138) * bump version 0.20.1 * bump version 0.20.1 * [Fix] revise --option to --options open-mmlab#1140 Co-authored-by: Rockey <41846794+RockeyCoss@users.noreply.github.com> Co-authored-by: MengzhangLI <mcmong@pku.edu.cn> * decode_head: switch from linear to fcn * fix init list formatting * configs: remove variants, keep only vit-s on ade * align inference metric of vit-s-mask * configs: add vit t/b/l * Update mmseg/models/decode_heads/segmenter_mask_head.py Co-authored-by: Miao Zheng <76149310+MeowZheng@users.noreply.github.com> * Update mmseg/models/decode_heads/segmenter_mask_head.py Co-authored-by: Miao Zheng <76149310+MeowZheng@users.noreply.github.com> * Update mmseg/models/decode_heads/segmenter_mask_head.py Co-authored-by: Miao Zheng <76149310+MeowZheng@users.noreply.github.com> * Update mmseg/models/decode_heads/segmenter_mask_head.py Co-authored-by: Miao Zheng <76149310+MeowZheng@users.noreply.github.com> * Update mmseg/models/decode_heads/segmenter_mask_head.py Co-authored-by: Miao Zheng <76149310+MeowZheng@users.noreply.github.com> * model_converters: use torch instead of einops * setup: remove einops * segmenter_mask: fix missing imports * add necessary imported init funtion * segmenter/seg-l: set resolution to 640 * segmenter/seg-l: fix test size * fix vitjax2mmseg * add README and unittest * fix unittest * add docstring * refactor config and add pretrained link * fix typo * add paper name in readme * change segmenter config names * fix typo in readme * fix typos in readme * fix segmenter typo * fix segmenter typo * delete redundant comma in config files * delete redundant comma in config files * fix convert script * update lateset master version Co-authored-by: MengzhangLI <mcmong@pku.edu.cn> Co-authored-by: Junjun2016 <hejunjun@sjtu.edu.cn> Co-authored-by: Rockey <41846794+RockeyCoss@users.noreply.github.com> Co-authored-by: Miao Zheng <76149310+MeowZheng@users.noreply.github.com>

OpenMMLab-Coodinator · 2023-04-03T06:51:49Z

Hi @rstrudel ！First of all, we want to express our gratitude for your significant PR in the MMSeg project. Your contribution is highly appreciated, and we are grateful for your efforts in helping improve this open-source project during your personal time. We believe that many developers will benefit from your PR.

We would also like to invite you to join our Special Interest Group (SIG) private channel on Discord, where you can share your experiences, ideas, and build connections with like-minded peers. To join the SIG channel, simply message moderator— OpenMMLab on Discord or briefly share your open-source contributions in the #introductions channel and we will assist you. Look forward to seeing you there! Join us ：https://discord.gg/raweFPmdzG

If you are Chinese or have WeChat，welcome to join our community on WeChat. You can add our assistant ：openmmlabwx. Please add "mmsig + Github ID" as a remark when adding friends：）
Thank you again for your contribution❤

rstrudel added 11 commits October 6, 2021 10:53

segmenter: add model

eaef5e8

update

cc8e3f4

readme: update

d609009

config: update

785a9cb

segmenter: update readme

e0c0766

segmenter: update

6bcec05

segmenter: update

5342698

segmenter: update

5570f10

configs: set checkpoint path to pretrain folder

48cb36b

segmenter: modify vit-s/lin, remove data config

b93ad3a

rreadme: update

e8366e3

rstrudel changed the title ~~Dev mmseg~~ [Feature] Support Segmenter Oct 12, 2021

rstrudel added 4 commits October 12, 2021 18:08

configs: transfer from _base_ to segmenter

58f7bec

configs: add 8x1 suffix

9deb2dc

configs: remove redundant lines

6f5ecd5

configs: cleanup

4f5762f

rstrudel mentioned this pull request Oct 14, 2021

[Feature] Support Segmenter #952

Closed

MengzhangLI added the WIP Work in process label Oct 18, 2021

MengzhangLI self-assigned this Oct 18, 2021

Merge branch 'open-mmlab:master' into dev_mmseg

f5cb915

MengzhangLI removed the Merging PR waited for merging label Jan 25, 2022