Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Preliminary implementation on Qualcomm NPU (QNN) backend. #112

Merged
merged 518 commits into from
Aug 9, 2024
Merged

feat: Preliminary implementation on Qualcomm NPU (QNN) backend. #112

merged 518 commits into from
Aug 9, 2024

Conversation

liang1232018
Copy link
Collaborator

A preliminary support for Qualcomm NPU prefilling

  • add a new qnn backend
  • implement llm-specific ops.

Design details referring to this paper: Empowering 1000 tokens/second on-device LLM prefilling with mllm-NPU.

yirongjie and others added 30 commits March 30, 2024 02:41
@yirongjie
feat: add QNN OPT test demo
79f6342
@yirongjie
feat
f1f8401
@yirongjie
fix: rm ls-n
153c026
@yirongjie
fix init Backends
ac89f74
@liang1232018
fix: cpu matmul int8 scale
c3159da 
feat: multi graph backends execute
@liang1232018
refactor: cpu split input execute
3cc4f66
@liang1232018
Merge pull request #21 from liang1232018/develop-QNN-zh
c866437 
Develop qnn zh
@liang1232018
fix : gelu, relu and linear scale bugs.
c5e51aa
@liang1232018
fix : verified no bias model.
c26727d
@liang1232018
fix : verified no layernorm model.
8e7512e
@liang1232018
fix : round quant scale.
f2a45bf
@liang1232018
fix : CPU QNN matmul int8 bugs.
2b6ba59
@liang1232018
feat : support SINT8 in merge output.
2fee710
@liang1232018
fix : update models.
1770912
@chunfenri
fix : update merge output and split input to support residual.
b0e2803
@chunfenri
fix : merge output parameter bugs.
b080672
@chunfenri
feat : suport new Mergeoutput and SplitInput.
60ddcbd
@chunfenri
fix : split input support number params.
0611e31
@chunfenri
feat : support residual and verify correctness.
175f229
@liang1232018
Merge pull request #22 from liang1232018/develop-QNN-CQ
7577a2b 
Develop qnn cq
@liang1232018
refactor: add new executor for qnn pipeline
ff4b49d
@liang1232018
feat: add thread pool in qnn pipeline executor
e4cd0c2
@chunfenri
support all opt models.
6ed9c08
@chunfenri
fix : quantize no clip bug.
018e8bc
@chunfenri
fix : CPU output tensor type bugs.
a517e66
@chunfenri
fix : CPU view output tensor type bugs.
a587a37
@chunfenri
1. support residual models with split and merge. 2. fix FP tensor inp…
54b5ae8 
…ut in QNN will be add 128 bugs by sub 128 in LLaMA add during residual. 3. Set all qnn graph input tenosr as SINT8.
@chunfenri
fix : 4 layer 128 input opt model.
27f7c85
@liang1232018
fix: qnn chunk execute temp result
427b213
@liang1232018
Merge branch 'develop-QNN' into develop-QNN-zh
c407c82
@yirongjie yirongjie self-requested a review August 9, 2024 09:58
@yirongjie yirongjie merged commit 83d65df into main Aug 9, 2024
1 check passed
@yirongjie yirongjie deleted the develop branch August 9, 2024 15:00
@yirongjie yirongjie changed the title Preliminary implementation on Qualcomm NPU (QNN) backend. feat: Preliminary implementation on Qualcomm NPU (QNN) backend. Aug 16, 2024
@yirongjie yirongjie added the good first issue Good for newcomers label Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yirongjie yirongjie yirongjie approved these changes

Assignees
No one assigned
Labels
good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@liang1232018 @yirongjie @chunfenri @oreomaker