Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP]: improvement(ViT): fuse nn.Linear and convert Gather to Crop #442

Closed
wants to merge 14 commits into from
Closed

[WIP]: improvement(ViT): fuse nn.Linear and convert Gather to Crop #442

wants to merge 14 commits into from

Conversation

tpoisonooo
Copy link
Collaborator

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily receiving feedbacks. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Fix review advices from Tencent/ncnn#3730

Modification

  • mmdeploy split nn.Linear into multiple opr when input.shape=[1, x, x]
    • remove ~100 redundant opr in end2end.param (289 --> 195)
  • [WIP] use Crop axis=0 starts=0 ends=1 opr instead of Gather axis=0 indices=[0] in ViT.param

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
  3. If the modification has a dependency on downstream projects of a newer version, this PR should be tested with all supported versions of downstream projects.
  4. The documentation has been modified accordingly, like docstring or example tutorials.

tpoisonooo and others added 4 commits May 7, 2022 16:55
@tpoisonooo
improvement(ViT): fuse nn.Linear and convert Gather to Crop
54cdf8a
@tpoisonooo
improvement(ViT): remove Gather opr
061f404
@grimoire
Limit mmpose requirement (open-mmlab#440)
81a9719
@tpoisonooo @grimoire
docs(docs/zh_cn): update doc and link checker (open-mmlab#418)
5f59b44 
* docs(docs/zh_cn): add doc and link checker

* docs(REAME): update

* docs(docs/zh_cn): update

* docs(benchmark): update table

* docs(zh_cn/benchmark): update link

* CI(docs): update link check

* ci(doc): update checker

* docs(zh_cn): update

* style(ci): remove useless para

* style(ci): update

* docs(zh_cn): update

* docs(benchmark.md): fix mobilnet link error

* docs(docs/zh_cn): add doc and link checker

* docs(REAME): update

* docs(docs/zh_cn): update

* docs(benchmark): update table

* docs(zh_cn/benchmark): update link

* CI(docs): update link check

* ci(doc): update checker

* docs(zh_cn): update

* style(ci): remove useless para

* style(ci): update

* docs(zh_cn): update

* docs(benchmark.md): fix mobilnet link error

* docs(zh_cn/do_regression_test.md): rebase

* docs(docs/zh_cn): add doc and link checker

* Update README_zh-CN.md

* Update README_zh-CN.md

* Update index.rst

* Update check-doc-link.yml

* [Fix] Fix ci (open-mmlab#426)

* fix ci

* add nvidia key

* remote torch

* recover pytorch

* ci(codecov): ignore ci

* docs(zh_cn): add get_started.md

* docs(zh_cn): fix review advice

* docs(readthedocs): update

* docs(zh_CN): update

* docs(zh_CN): revert

* fix(docs): review advices

* fix(docs): review advices

* fix(docs): review

Co-authored-by: q.yao <streetyao@live.com>
@tpoisonooo tpoisonooo changed the title WIP: improvement(ViT): fuse nn.Linear and convert Gather to Crop improvement(ViT): fuse nn.Linear and convert Gather to Crop May 9, 2022
grimoire
grimoire reviewed May 12, 2022
View reviewed changes
mmdeploy/pytorch/ops/linear.py Outdated
backend=Backend.NCNN.value)
def linear__ncnn(ctx, g, input, weight, bias):
"""Support export linear This rewrite enable export Gemm."""
return g.op(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/pytorch/pytorch/blob/c25bdeea26f95d05281cc4309bc400a0eae0b8a4/torch/onnx/symbolic_opset9.py#L4518
Bias could be None.

grimoire
grimoire reviewed May 12, 2022
View reviewed changes
mmdeploy/pytorch/ops/linear.py
def linear__ncnn(ctx, g, input, weight, bias):
"""Support export linear This rewrite enable export Gemm."""
return g.op(
'mmdeploy::Gemm',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove mmdeploy domain? It looks like the same as the official one ONNX Gemm

tpoisonooo and others added 9 commits May 12, 2022 12:22
@tpoisonooo
fix(zh_CN): doc link (open-mmlab#447)
9a4028b 
* docs(zh_CN): fix link missing

* docs(benchmark.md): fix pytorch typo
@RunningLeon
fix import mmrazor error in mmcls test (open-mmlab#467)
6eb83a9
@tpoisonooo @grimoire
refactor(onnx2ncnn): add test case and simplify code (open-mmlab#436)
d04c8dc 
* refactor(onnx2ncnn.cpp): split it to shape_inference, pass and utils

* refactor(onnx2ncnn.cpp): split it to shape_inference, pass and utils

* refactor(onnx2ncnn.cpp): split code

* refactor(net_module.cpp): fix build error

* ci(test_onnx2ncnn.py): add generate model adn run

* ci(onnx2ncnn): add ncnn backend

* ci(test_onnx2ncnn): add converted onnx model`

* ci(onnx2ncnn): fix ncnn tar

* ci(backed-ncnn): simplify dependency install

* ci(onnx2ncnn): fix apt install

* Update backend-ncnn.yml

* Update backend-ncnn.yml

* Update backend-ncnn.yml

* Update backend-ncnn.yml

* Update backend-ncnn.yml

* Update backend-ncnn.yml

* Update backend-ncnn.yml

* Update backend-ncnn.yml

* Update backend-ncnn.yml

* Update backend-ncnn.yml

* Update backend-ncnn.yml

* fix(ci): add include algorithm

* Update build.yml

* parent aa85760
author q.yao <streetyao@live.com> 1651287879 +0800
committer tpoisonooo <khj.application@aliyun.com> 1652169959 +0800

[Fix] Fix ci (open-mmlab#426)

* fix ci

* add nvidia key

* remote torch

* recover pytorch

refactor(onnx2ncnn.cpp): split it to shape_inference, pass and utils

* fix(onnx2ncnn): review

* fix(onnx2ncnn): build error

Co-authored-by: q.yao <streetyao@live.com>
@tpoisonooo
improvement(ViT): fuse nn.Linear and convert Gather to Crop
7e68d48
@tpoisonooo
improvement(ViT): remove Gather opr
adb256c
@tpoisonooo
feat(pytorch/ops): add linear testcase
a2f6903
@tpoisonooo
fix(CI): update linear
c3adba0
@tpoisonooo
fix(linear.py): review advices
6965215
@tpoisonooo
Merge branch 'vit-fuse-linear' of https://github.com/tpoisonooo/mmdeploy
8972093 
 into vit-fuse-linear
@tpoisonooo tpoisonooo changed the title improvement(ViT): fuse nn.Linear and convert Gather to Crop [WIP]: improvement(ViT): fuse nn.Linear and convert Gather to Crop May 16, 2022
@tpoisonooo
Copy link
Collaborator Author

close this PR because of bad git rebase, open a new one #477

@tpoisonooo tpoisonooo closed this May 16, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@grimoire grimoire grimoire left review comments

@RunningLeon RunningLeon Awaiting requested review from RunningLeon

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@tpoisonooo @grimoire @RunningLeon