Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Allow passing model_args to ST
Details
This fixes #2579.
New models like e5-mistral-7b-instruct use FP16 as the dtype.
However, when loaded using sentence_transformers, they are loaded with FP32. This is because
HF transformers uses FP32 as the default unless torch_dtype='auto' is passed, as mentioned here.
Passing "auto" to model_args does not work because of the below error. Moreover, the SentenceTransformers class does not currently expose a model_args param.
Testing Done
Added a new unit tests that validates the dtype of the loaded model using the embedding tensor
created with it.