Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vit evaluators demo QAT and export to onnx model,but onnx model to snpe-dlc error #37

Open
xiejunyi-joey opened this issue Jul 18, 2023 · 3 comments

Comments

@xiejunyi-joey
Copy link

I used aimet-model-zoo vit demo(vit_quanteval.py),and export the quanted model with json to onnx;And I use snpe-onnx-to-dlc(v2.10),but errors

image

@aiwhz
Copy link

aiwhz commented Jul 18, 2023

@xiejunyi-joey
Have you reproduced the accuracy results with the scripts vit_quanteval.py?
My test results show the quantized model accuracy is not good on imagenet.

2023-07-18 15:01:42,720 - main - INFO - Original model performances
2023-07-18 15:01:42,720 - main - INFO - ===========================
2023-07-18 15:01:42,720 - main - INFO - Original Model | 32-bit Environment | perplexity : 0.8132
2023-07-18 15:01:42,720 - main - INFO - Original Model | 8-bit Environment | perplexity: 0.0016
2023-07-18 15:01:42,720 - main - INFO - Optimized model performances
2023-07-18 15:01:42,721 - main - INFO - ===========================
2023-07-18 15:01:42,721 - main - INFO - Optimized Model | 32-bit Environment | perplexity: 0.8082
2023-07-18 15:01:42,721 - main - INFO - Optimized Model | 8-bit Environment | perplexity: 0.0020

@xiejunyi-joey
Copy link
Author

the original 8bit result is not good
image

@aiwhz
Copy link

aiwhz commented Jul 19, 2023

@xiejunyi-joey I also met the converter issue, hope the later SNPE can fix it.

Regarding the accuracy issue.

  1. The Origin model int8 is not QAT, so it is not good.
  2. After changing some code, I can get the correct results for the Optimized Model INT8. fix the encoding.csv file lacation to the right diractory #38

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants