Qualcomm AI Engine Direct - Support QNN 2.28

[shewu-quic]

Note that if merged, QNN version less than 2.28 will not support it.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/6811

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 259cbc9 with merge base 7010a11 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[cccclai]

What feature is used from 2.28?

[shewu-quic]

What feature is used from 2.28?

Based on Qnn 2.28 release note, they seem to improve the latency for large models.
Therefore, we could get better performance.
BTW, I found an optimization that has not been updated yet. I will send another PR today so that you can reproduce the current latency with QNN 2.28.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Can you also update this line https://github.com/pytorch/executorch/blob/07c4d0e8624fbeb3b6a387cb9c6ba193512c5de5/shim/xplat/executorch/backends/qualcomm/qnn_version.bzl

[shewu-quic]

Can you also update this line https://github.com/pytorch/executorch/blob/07c4d0e8624fbeb3b6a387cb9c6ba193512c5de5/shim/xplat/executorch/backends/qualcomm/qnn_version.bzl

Fixed

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Do we know if static llama and llama transformer will still work with this version bump? Also we need to figure out a plan to do version bump without bc breaking...

[shewu-quic]

Do we know if static llama and llama transformer will still work with this version bump? Also we need to figure out a plan to do version bump without bc breaking...

Our tests for static llama all are on this version.
It should work for llama transformer based on your CI result.
https://github.com/pytorch/executorch/actions/runs/11906365001/job/33178331953?pr=6811

About bc breaking, I think that we could add some macros around the code which only support on 2.28 to make sure that users could build libqnnexecutorch on older qnn version. Is it acceptable?

[cccclai]

Do we know if static llama and llama transformer will still work with this version bump? Also we need to figure out a plan to do version bump without bc breaking...

Our tests for static llama all are on this version. It should work for llama transformer based on your CI result. https://github.com/pytorch/executorch/actions/runs/11906365001/job/33178331953?pr=6811

About bc breaking, I think that we could add some macros around the code which only support on 2.28 to make sure that users could build libqnnexecutorch on older qnn version. Is it acceptable?

Yeah that sounds good. BC is something we can figure out later. Just hope it can be part of the plan.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Hi sorry for being a bit back and forth here - Context is that the pre-generated .pte file will no longer be compatible. If the work isn't too much, how likely we can make this PR bc compatible? If there is too much work, we can try to merge this.

[shewu-quic]

Hi sorry for being a bit back and forth here - Context is that the pre-generated .pte file will no longer be compatible. If the work isn't too much, how likely we can make this PR bc compatible? If there is too much work, we can try to merge this.

Thank you for your response. I apologize for any confusion. What I meant is that if we merge this PR, libqnnexecutorch will not be able to build with QNN versions earlier than 2.28. However, the pre-generated .pte files should still be supported. The build issue arises in cases like this: QNN_SYSTEM_CONTEXT_BINARY_INFO_VERSION_3 only exists in headers from version 2.28 onwards. If you try to build with version 2.26, you'll encounter an error stating that QNN_SYSTEM_CONTEXT_BINARY_INFO_VERSION_3 cannot be found.

[shewu-quic]

If possible, please go ahead and merge this PR. We will discuss internally whether to add some macros around the code that only supports version 2.28 and later, to ensure that users can build libqnnexecutorch on older QNN versions.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[shewu-quic]

Hi @cccclai,
Thanks for review.
I think that @winskuo-quic has added some macros to ensure that users can build libqnnexecutorch on older QNN versions.
If possible, could you help to merge this PR?

[facebook-github-bot]

@cccclai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[cccclai]

Thanks! Inform the internal team and merging now