Broken model output

[kallewoof]

Continuation from huggingface/peft#1813 (comment) to not spam up that pull request.

I will retry training using this repo and see if it makes a difference.

Edit: to answer your question,

Yes, it is for mora_type=2 or 3. Could you provide the settings of MoRA? (Maybe we can discuss this in MoRA repo issue?)

I used the defaults for most of it. Rank=64, learning rate constant 6e-5.

I just now realized you are suggesting mora_type=6 for smaller ranks. Perhaps I should be using that. Maybe the selection (if not specified) can be automated based on r, in case this is a reason for adapter breakdown.

[kongds]

Or can you tell me the training data so I can check this?

[kallewoof]

OK, retrained using this repo and the model is not broken anymore.

Ping @winglian: FYI.

[kallewoof]

After more thorough testing, the model is in fact broken still. I am trying to find a public dataset that exhibits the same behavior. The dataset I am using right now is a mix of completion (prediction) and instruction data, and the model is llama3-8b.

To clarify, it is broken in that it does not produce coherent responses to the context or gravely misunderstands fairly straightforward concepts.

I am also considering doing another test run with mora_type=6 as that seems to be recommended for lower ranks. (I use 64.)

Edited to add: I believe I know what the problem is: with LoRA, I usually train against the base model, and then merge into any of the fine tunes on top of it. This works, possibly because LoRAs do not have as great an effect on the model as MoRA. I am going to retry training on the instruct model directly to see if that solves it.

I will use @winglian's pull request this time, as I believe there is no difference between the two code bases.

[kallewoof]

I trained on and merged with the same instruction model this time, this time using mora_type=6, and the output was better but still broken. This could be a matter of me simply overwhelming the model with too high of a learning rate though, but the MoRA paper did linear 5e-5-7e-5 over several epochs whereas I am doing a single epoch constant lr 6e-5 (right in the middle).

It could also be that MoRA is more powerful and thus more sensitive to issues in the dataset, since it's a mix of pretrain and instruction; with LoRA, it is possible to train on top of an instruction model with a pretrain dataset, and the instruction set remains intact, but it is possible that MoRA "digs deeper" and unravels the instruction aspect.

That said, the output I saw also indicated that the model was gravely confused about what was going on, so I don't think it's only a matter of the instruction format being blown to pieces.

[kongds]

Thank you for sharing your insights and analysis. I haven’t encountered output breaking before, if you could provide similar training data, it would be helpful.

Additionally, if the output breaks, does the loss perform normally compared to LoRA? Regarding the learning rate, the specified rate is used for rank 256 with type 1. The rank and type in MoRA might affect the choice of the learning rate.

I am also a little confused about the current training setting. It seems you previously trained LoRA on the base model and merged it into the instruction model. For MoRA type 6, it seems you trained it on the instruction model with mixed data. I have also tried MoRA with a mix of pre-training and instruction data before, but I haven’t encountered this issue.

[kallewoof]

I tried a training run with purely completion (all instructions removed) on a dozen or so MB worth of stories, and the model exhibited the same broken down behavior.

I am starting to think MoRA training needs to be done quite differently from LoRA training. In your paper, it looks like you did completion (pretraining) training on top of instruction models, but maybe I'm mistaken. My suspicion is that you simply can't do that with MoRA. I'm a little busy with unrelated stuff for a week or two, but once I am able to I will see about getting a dataset that exhibits the behavior I described. (It shouldn't be hard, unless my quite random dataset is unique in ways I don't realize.)

[kongds]

For continual pretraining, we use base models (not instruction models) in our paper. Based on the papers about the continual pretraining of LLMs that I have read, they all seem to use the base model for continual pretraining. Additionally, we only use base models for fine-tuning in other experiments in the paper.

For the current settings, it seems that you want to maintain the instruction-following ability of an instruction model with fine-tuning. MoRA causes the output to break. Given the output before, it seems the model has collapsed during fine-tuning. It would be helpful to check if the training loss is normal. Alternatively, maybe you could change the settings, such as increasing the lora_dropout or decreasing the learning rate.

[Ecocytus]

After more thorough testing, the model is in fact broken still. I am trying to find a public dataset that exhibits the same behavior. The dataset I am using right now is a mix of completion (prediction) and instruction data, and the model is llama3-8b.

To clarify, it is broken in that it does not produce coherent responses to the context or gravely misunderstands fairly straightforward concepts.

I am also considering doing another test run with mora_type=6 as that seems to be recommended for lower ranks. (I use 64.)

Edited to add: I believe I know what the problem is: with LoRA, I usually train against the base model, and then merge into any of the fine tunes on top of it. This works, possibly because LoRAs do not have as great an effect on the model as MoRA. I am going to retry training on the instruct model directly to see if that solves it.

I will use @winglian's pull request this time, as I believe there is no difference between the two code bases.

Maybe it's a matter of encouraging hallucinations, which need a handcrafted "dev" dataset to test the model's capabilities, similar to this work

[kallewoof]

@kongds Thanks for the feedback. I have run out of time to work on this for the near future, but will pick this back up and experiment more as soon as I'm able to.

I'll report back and possibly reopen this if my issues persist. If they do I will provide a dataset for testing on your end.