[Feature] Support BiSeNetV2 #804

MengzhangLI · 2021-08-20T18:05:21Z

Implementation of BiSeNet V2: Bilateral Network with Guided Aggregation for Real-time Semantic Segmentation.
Modified from the BiSeNetV2 PyTorch repository.

Done:

unittest
align inference metric (74.98)
align training metric (from about 73.0 to 74.0).
Totally using FCNHead than changing its middle channels.
align training metric (paper with OHEM is 73.36)
FP16 training results

Todo:

More datasets such as cocostuff
(STOPPED) Changing slightly (remove BN or ReLU in certain layers in BiSeNetV2 backbone) and test its performance.
(STOPPED) Try to use pretrained model from CoinCheung

Notice:
(1) This version of implementation is a little bit different from original version. Here is comparison.

MengzhangLI · 2021-08-20T18:06:53Z

The inference metric is correct of this version.

Just ignore FCNHead1 in this version PR please.

codecov · 2021-08-20T18:22:00Z

Codecov Report

Merging #804 (b2ee8df) into master (4e6f2eb) will increase coverage by 0.49%.
The diff coverage is 98.88%.

❗ Current head b2ee8df differs from pull request most recent head 9c227d7. Consider uploading reports for the commit 9c227d7 to get more accurate results

@@            Coverage Diff             @@
##           master     #804      +/-   ##
==========================================
+ Coverage   89.05%   89.55%   +0.49%     
==========================================
  Files         111      113       +2     
  Lines        6051     6224     +173     
  Branches      969      989      +20     
==========================================
+ Hits         5389     5574     +185     
+ Misses        467      452      -15     
- Partials      195      198       +3

Flag	Coverage Δ
unittests	`89.55% <98.88%> (+0.49%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmseg/models/backbones/swin.py	`87.41% <ø> (ø)`
mmseg/datasets/coco_stuff.py	`87.50% <87.50%> (ø)`
mmseg/models/backbones/bisenetv2.py	`99.28% <99.28%> (ø)`
mmseg/core/seg/sampler/ohem_pixel_sampler.py	`94.87% <100.00%> (+0.27%)`	⬆️
mmseg/datasets/__init__.py	`100.00% <100.00%> (ø)`
mmseg/models/backbones/__init__.py	`100.00% <100.00%> (ø)`
mmseg/models/decode_heads/decode_head.py	`92.22% <100.00%> (+0.86%)`	⬆️
mmseg/models/decode_heads/point_head.py	`96.55% <100.00%> (+0.02%)`	⬆️
mmseg/models/losses/cross_entropy_loss.py	`79.10% <100.00%> (+0.97%)`	⬆️
mmseg/models/losses/dice_loss.py	`100.00% <100.00%> (ø)`
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4e6f2eb...9c227d7. Read the comment docs.

MengzhangLI · 2021-08-23T10:54:05Z

I tried a lot Unittest commit, the reason why I always failed is test Process completed with exit code 137., which means the CPU memory is limited. The solution is making input smaller: from [4, 3, 1024, 2048] to [2, 3, 512, 1024].

Junjun2016 · 2021-08-23T11:22:46Z

I tried a lot Unittest commit, the reason why I always failed is test Process completed with exit code 137., which means the CPU memory is limited. The solution is making input smaller: from [4, 3, 1024, 2048] to [2, 3, 512, 1024].

The input can be smaller for unittests.

MengzhangLI · 2021-08-23T11:52:53Z

The training metric is here:

Note: It is not stable when training from scratch.

Here is its training log:
20210821_060603.log

MengzhangLI · 2021-08-30T17:20:30Z

First time code refactoring done. Here is inference result whose model is above:

python tools/test.py ./configs/bisenet/bisenet_4x4_1024x1024_160k_cityscapes.py ./checkpoints/exp3_fcn/latest_full.pth --eval mIoU

configs/_base_/models/bisenet.py

configs/bisenet/bisenet_4x4_1024x1024_160k_cityscapes.py

MengzhangLI · 2021-08-30T19:23:21Z

The training metric is here:

Note: It is not stable when training from scratch.

Here is its training log:
20210821_060603.log

Just check this log, the cross entropy loss of auxiliary heads all use_sigmoid=True

Junjun2016

Please refine docstring.

mmseg/models/backbones/bisenet.py

Junjun2016 · 2021-08-30T19:53:44Z

The training metric is here:

Note: It is not stable when training from scratch.
Here is its training log:
20210821_060603.log

Just check this log, the cross entropy loss of auxiliary heads all use_sigmoid=True

Which repository or paper did you refer to?

MengzhangLI · 2021-08-31T08:07:49Z

The training metric is here:

Note: It is not stable when training from scratch.
Here is its training log:
20210821_060603.log

Just check this log, the cross entropy loss of auxiliary heads all use_sigmoid=True

Which repository or paper did you refer to?

None of them. I did not care too much about loss in auxiliary head.

Actually paper did not say too much about loss and other repository did not adopt use_sigmoid=True.

configs/bisenetv2/README.md

mmseg/models/backbones/bisenetv2.py

xvjiarui

LGTM except one comment

configs/bisenetv2/README.md

mmseg/models/backbones/bisenetv2.py

Junjun2016

Please resolve some comments.

…into BiSeNetV2

* BiSeNetV2 first commit * BiSeNetV2 unittest * remove pytest * add pytest module * fix ConvModule input name * fix pytest error * fix unittest * refactor * BiSeNetV2 Refactory * fix docstrings and add some small changes * use_sigmoid=False * fix potential bugs about upsampling * Use ConvModule instead * Use ConvModule instead * fix typos * fix typos * fix typos * discard nn.conv2d * discard nn.conv2d * discard nn.conv2d * delete **kwargs * uploading markdown and model * final commit * BiSeNetV2 adding Unittest for its modules * BiSeNetV2 adding Unittest for its modules * BiSeNetV2 adding Unittest for its modules * BiSeNetV2 adding Unittest for its modules * BiSeNetV2 adding Unittest for its modules * BiSeNetV2 adding Unittest for its modules * BiSeNetV2 adding Unittest for its modules * Fix README conflict * Fix unittest problem * Fix unittest problem * BiSeNetV2 * Fixing fps * Fixing typpos * bisenetv2

* add chinese tutorial for new_module. * change some words.

MengzhangLI added 2 commits August 21, 2021 00:53

BiSeNetV2 first commit

cdfbf2a

BiSeNetV2 unittest

5db6f02

remove pytest

7be9a65

MengzhangLI added 4 commits August 21, 2021 02:24

add pytest module

dc195ff

fix ConvModule input name

bfa1ded

fix pytest error

df17af6

fix unittest

9d24dd0

MengzhangLI requested review from clownrat6, Junjun2016 and xiexinch August 23, 2021 11:57

refactor

82acd57

MengzhangLI added the WIP Work in process label Aug 26, 2021

BiSeNetV2 Refactory

96b2489

Junjun2016 reviewed Aug 30, 2021

View reviewed changes

configs/_base_/models/bisenet.py Outdated Show resolved Hide resolved

Junjun2016 reviewed Aug 30, 2021

View reviewed changes

configs/_base_/models/bisenet.py Outdated Show resolved Hide resolved

Junjun2016 reviewed Aug 30, 2021

View reviewed changes

configs/_base_/models/bisenet.py Outdated Show resolved Hide resolved

Junjun2016 reviewed Aug 30, 2021

View reviewed changes

configs/_base_/models/bisenet.py Outdated Show resolved Hide resolved

Junjun2016 reviewed Aug 30, 2021

View reviewed changes

configs/bisenet/bisenet_4x4_1024x1024_160k_cityscapes.py Outdated Show resolved Hide resolved

Junjun2016 reviewed Aug 30, 2021

View reviewed changes

Junjun2016 changed the title ~~[Feature] add BiSeNetV2~~ [Feature] Support BiSeNetV2 Aug 30, 2021

fix docstrings and add some small changes

ee4af5b

MengzhangLI added 4 commits September 17, 2021 18:50

BiSeNetV2 adding Unittest for its modules

20a3bed

BiSeNetV2 adding Unittest for its modules

ea95792

Fix README conflict

52a6d35

Fix README conflict

6e068f4

Junjun2016 reviewed Sep 17, 2021

View reviewed changes

configs/bisenetv2/README.md Outdated Show resolved Hide resolved

MengzhangLI added 2 commits September 18, 2021 01:11

Fix unittest problem

08de8cd

Fix unittest problem

8d0cfe9

xvjiarui reviewed Sep 18, 2021

View reviewed changes

mmseg/models/backbones/bisenetv2.py Show resolved Hide resolved

xvjiarui approved these changes Sep 18, 2021

View reviewed changes