Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor] Refactor transfomer encode layer in ViT and BEiT backbone #1481

Merged
merged 8 commits into from
Apr 23, 2022

Conversation

MeowZheng
Copy link
Collaborator

@MeowZheng MeowZheng commented Apr 17, 2022

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

  1. Remove redundant code
  2. Modularize attention and transformer encode layer

Modification

  1. Add build function for attn and ffn in TransformerEncoderLayer
  2. Add attn_cfg and ffn_cfg in TransformerEncoderLayer
  3. Revise qk_bias to bias in BEiTAttention

Results

Method before refactor inference test after refactor training result after refactor
upernet_beit-base_8x2_640x640_160k_ade20k 53.08 53.08 52.85
upernet_vit-b16_mln_512x512_160k_ade20k 47.76 48.40

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMDet3D.
  4. The documentation has been modified accordingly, like docstring or example tutorials.

@MeowZheng MeowZheng changed the title [Refactor] Refactor transfomer encode layer [Refactor] Refactor transfomer encode layer in ViT and BEiT backbone Apr 17, 2022
@codecov
Copy link

codecov bot commented Apr 17, 2022

Codecov Report

Merging #1481 (8c63256) into master (5b605b0) will decrease coverage by 0.00%.
The diff coverage is 94.73%.

@@            Coverage Diff             @@
##           master    #1481      +/-   ##
==========================================
- Coverage   90.30%   90.30%   -0.01%     
==========================================
  Files         140      140              
  Lines        8335     8345      +10     
  Branches     1400     1403       +3     
==========================================
+ Hits         7527     7536       +9     
- Misses        570      571       +1     
  Partials      238      238              
Flag Coverage Δ
unittests 90.30% <94.73%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmseg/models/backbones/beit.py 91.83% <93.33%> (-0.51%) ⬇️
mmseg/models/backbones/vit.py 90.90% <100.00%> (+0.32%) ⬆️
mmseg/models/losses/accuracy.py 100.00% <0.00%> (ø)
mmseg/models/segmentors/base.py 59.20% <0.00%> (+1.34%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5b605b0...8c63256. Read the comment docs.

@MeowZheng MeowZheng merged commit 239049c into open-mmlab:master Apr 23, 2022
@MeowZheng MeowZheng deleted the refactor-transformer branch April 23, 2022 09:35
ZhimingNJ pushed a commit to AetrexTechnology/mmsegmentation that referenced this pull request Jun 29, 2022
…pen-mmlab#1481)

* [Refactor] Refactor transfomer encoder layer

* fix ffn cfg

* fix drop path cfg

* private init func

* fix qv bias

* fix qkv bias

* fix qkv bias

* wH->Wh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants