[AMP] Turn off accumulation data types for mixed precision pass #8341
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
comaniac
approved these changes
Jun 25, 2021
comaniac
changed the title
|
CI failed due to #8344 |
AndrewZhaoLuo
force-pushed
the
andrewluo-fix-cuda
branch
from
268a49e
to
39cfc8c
Compare
…r now test to mixed precision more tests add internal func call broadcast failures moreee add comment and change lstm unit test to pass on cuda
AndrewZhaoLuo
force-pushed
the
andrewluo-fix-cuda
branch
from
39cfc8c
to
effd073
Compare
ylc
pushed a commit
to ylc/tvm
that referenced
this pull request
Sep 29, 2021
…he#8341) * don't use mixed precision accumulators * turn off fp32 accumulators for now, adjust passing test cases * Add TODO on cuda codegen for failures. Make test case pass on cuda for now test to mixed precision more tests add internal func call broadcast failures moreee add comment and change lstm unit test to pass on cuda * remove debug statements * to mixed precision * rebase main * rtol and atol adjustments * bump up tolerance again * jostle CI
zxy844288792
pushed a commit
to zxy844288792/tvm
that referenced
this pull request
Mar 4, 2022
…he#8341) * don't use mixed precision accumulators * turn off fp32 accumulators for now, adjust passing test cases * Add TODO on cuda codegen for failures. Make test case pass on cuda for now test to mixed precision more tests add internal func call broadcast failures moreee add comment and change lstm unit test to pass on cuda * remove debug statements * to mixed precision * rebase main * rtol and atol adjustments * bump up tolerance again * jostle CI
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
CUDA codegen cannot seem to handle half types super well. Furthermore, mixing half types and floating point also seems to expose additional issues. Furthermore, some schedules which are supposed to support heterogenous outputs do not.
This seems like a problem in codegen not with the mixed precision pass, so for now I am turning off accumulating into FP32 for the mixed precision pass. With this we can tune BERT and YoloV2 with results here: https://docs.google.com/spreadsheets/d/12lgyfuHaRS-X4uG-1iQOV8oAuPpuVAbspcmkOSPRFHQ/edit#gid=0
I will leave the codegen issues for #8294.
I will leave the issues with schedule not supporting output dtypes here #8340