Add Phi-3-mini adapter #145
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
msdmkats
approved these changes
May 13, 2024
nailimixaM
added a commit
that referenced
this pull request
Jun 18, 2024
* Update dependencies (#144) * Separate out dependencies for experiments * Raise peft version * Bump transformers * Bump datasets * Update README * Rollback peft to 0.6.0 and make it optional * Use a task metric map in lm_eval runner (#146) * Add Phi-3-mini adapter (#145) * Add Phi-3 adapter * Removed cast. Aligned type with base class. * Add support for llama3 adapter (#147) * Update transformers to 4.41.0 (#150) * Update transformers to latest * Spaces * Point to bug fix commit we want to pick * Update pyproject.toml * update README * update --------- Co-authored-by: Dmitry Kats <dmitrykats@microsoft.com> Co-authored-by: Pashmina Cameron <pcameron@microsoft.com> Co-authored-by: radhikamp99 <47057131+radhikamp99@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds Phi-3-mini support. This requires updating transformers to near latest. There hasn't been a release of transformers package since Phi-3 support was added.
Dependencies:
A commit from Friday in transformers package has an issue so updating transformers package to just before that commit. Once the issue is fixed, we can update to latest git commit of transformers and once there's a release, we can point to the latest package.
Also updating
peft==0.6.2. I have checked the slicing and finetuning for Phi-2 and run the tests. We get nearly the same results as the paperWe can look at renaming the adapters: llama->llama2 and Phi-3->Phi-3-mini in a separate change. I have not tried the Phi-3-mini-128k version! We will need to figure out how to handle that too if we want to.