update from main #1
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Clear output of Torch SDPA for masked pieces Since Torch 2.1, the Torch memory-efficient SDPA GPU kernel returns NaN for pieces that are completely masked out. This leads to NaN propagation in the next attention layer, because masked pieces get an attention of zero, but zero times NaN is still NaN. In this we fix this by setting masked tokens to zero to clear out any NaNs. We currently rely on the query dimension of the mask to be singular, but in the future we should probably redesign the `AttentionMask` class to account for the differences between attention masks and causal masks. * black * Update MyPy version to one that supports recent PyTorch * Comment typos and fixes * Add assertion message Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com> * black --------- Co-authored-by: Madeesh Kannan <shadeMe@users.noreply.github.com>
There was a subtle bug where we populate models with parameters that are not leaf nodes because we called `to` on them for device placement. This change fixes this issue and validates that all model parameters are leaf nodes in the model tests.
We added support for TorchScript tracing a while back, so that models can be exported to ONNX. However, the support relies on metaclasses, which breaks with torch.compile in the latest PyTorch versions. However, PyTorch now provides a TorchDynamo-based ONNX exporter: https://pytorch.org/docs/stable/onnx_dynamo.html So it's time to yank TorchScript tracing support and remove all the fragile dataclass/tuple/dict polymorphism.
* Fix `test_rotary_embeddings_against_hf` for latest transformers * xfail test because HfFileSystem is currently broken
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Checklist