[CICL] cache flash_attn #223 #215
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Print statements for debugging and initial support for Code Llama * Added multiple print statements for debugging fine tuning * Added support for Code Llama 7b * Depending on the training parameters I set I either get an out of memory GPU error or ValueError(“optimizer got an empty parameter list”) * Code Llama fine-tuning but fails on checkpoint * commenting print statements * updating default config behavior * Begin adding encoding for Code Llama * adding BOS and EOS tokens for Code Llama, model running properly * getting rid of #? * Print statements for debugging and initial support for Code Llama * Added multiple print statements for debugging fine tuning * Added support for Code Llama 7b * Depending on the training parameters I set I either get an out of memory GPU error or ValueError(“optimizer got an empty parameter list”) * Code Llama fine-tuning but fails on checkpoint * commenting print statements * updating default config behavior * Begin adding encoding for Code Llama * adding BOS and EOS tokens for Code Llama, model running properly * getting rid of #?
saving in safe_tensors format
…for some models to set a lora to them)
TOKENIZERS_PARALLELISM=false while finetuning
add inference fixes for codellama
* add deepseek inference and finetuning * no extra kwargs * add deepseek-ai/deepseek-coder-5.7bmqa-base
Revert "move caps to the root" This reverts commit b440517.
* fixes * add extra ENVS to use fork() method inside the container
reymondzzzz
force-pushed
the
v1.2.0-cicl
branch
10 times, most recently
from
5586414
to
354f962
Compare
reymondzzzz
force-pushed
the
v1.2.0-cicl
branch
from
354f962
to
a65ee90
Compare
mitya52
approved these changes
Nov 10, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#223