New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is fastT5 qunatization slower than pytorch dynamic quantization? #72

Open

parikshitsaikia1619 opened this issue Jun 8, 2023 · 0 comments

parikshitsaikia1619 commented Jun 8, 2023 •

edited

Loading

Hello @Ki6an,

I am working on speeding up a finetuned t5-mini batch cpu inference.

On the batch size = 10, sequence length = 300 tokens:

t5-mini inference speed: 3 sec
t5-mini after pytorch built-in dynamic quantization: 2.3 sec
fastT5 after converting to onnx and quantization: 5.9 sec !!

Maybe I am doing something wrong, but after fastT5 it was supposed to be faster right?

pytorch:

fastT5

Collab notebook link:
Link

Please let me know your thoughts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment