When trying to merge the finetuned model on merge adapter not working.

[venturaEffect]

Writing here because there is no answer to questions from February asking to solve the same error.

Went to this space also form autotrain as it is not possible to convert into .gguf format once finetuned. And as the admin of autotrain says you have to use the merge adapter space in huggingface. You have to duplicate it and use GPU for it.

Even following the short description just got an error wilth no further explanation. And seen other encountering same problem but no response how to solve it.

And it is crucial as the finetune model needs to be converted into .gguf. Quite common.

Anyone facing the same issue.

[abhishekkrthakur]

hi. ive been answering your concerns and others' concerns consistently. but if you decide not to look at docs, and just bash, thats your choice.

now, for the final time, read what im writing here:

• if YOU train an adapter model, you get only the adapter. NO config.json
• if YOU train an adapter model, with merge_adapters parameter set to true, you get the full model, with config.json
• if you do NOT train an adapter model (set peft=false), you will get the full model, with config.json
• if you have trained an adapter model, and want to merge weights, you can do so on your own (recommended only for adavanced users) using the autotrain tools command:

in the end, its your choice what you want to do and how you want to do it.

transformers supports adapter-only models. autotrain is based on transformers and will only support transformers. it will NEVER convert your trained model to GGUF. if you want to convert your trained models to gguf, use this space: https://huggingface.co/spaces/ggml-org/gguf-my-repo

[venturaEffect]

First of all you didn't answer me before about the solutions. Second, where in the docs can I find all the commands and how to set them up? I have to look others people tutorials to understand how to use it.

First of all I'm not using Mac I'm using WSL for Windows. This is what I found it did the trick on youtube of course without knowing the merge adapters thing you are mentioning here now.

autotrain llm
--train
--model gradientai/Llama-3-8B-Instruct-Gradient-4194k
--data-path XXX/data
--lr 2e-4
--batch-size 2
--epochs 1
--trainer sft
--peft
--project-name blablabla
--username XXX
--push-to-hub
--token $HF_TOKEN

So, my question is where does here the merge adapters come?

Adding --merge_adapters True ??

[abhishekkrthakur]

https://huggingface.co/docs/autotrain/llm_finetuning_params

as i mentioned before, you can still use huggingface tools: inference endpoints, transformers, etc, to run the adapter only model (model without config json).

if you want to do advanced stuff like gguf, then merge the parameters using the param mentioned above during the training or manually after training has finished. adapter way of training is faster and cheaper. merge adapter paramemeter has been provided for advanced users like yourself :)

also, if you have questions, feel free to ask here and ill do my best to fix your problems today itself.

[venturaEffect]

Are there any docs on how to do it manually after training? If I'm right I had to set peft to false and then it would make the trick. So, to make it clear, as the command I provided before just peft to false:

autotrain llm
--train
--model gradientai/Llama-3-8B-Instruct-Gradient-4194k
--data-path Zaesar/emotion_data_combined
--lr 2e-4
--batch-size 2
--epochs 1
--trainer sft
--peft False
--project-name ai-psycho-assistant
--username Zaesar
--push-to-hub
--token $HF_TOKEN

Is this correct?

And again, if it is possible after being trained without setting it to False so there is no config.json file, How do I make it manually? Appreciate.

[abhishekkrthakur]

manually after training: use the command in the screenshot of my first response to this thread.

peft=false: you will get config.json, the model will be a full finetune, requires a lot of memory and might fail if you dont have enough gpu ram.

[venturaEffect]

Is it not possible to use the space to merge? It gives always errors. And even trying it on my computer (4090 GPU) it isn't enough.

Is it possible to make it work properly on the space?

Appreciate.

[abhishekkrthakur]

Do you want to use the space instead?

[venturaEffect]

Yes, if I can finally merge it and ending having a gguf format with the other space.

[abhishekkrthakur]

could you please tell me what the base model is? ill make a custom space to merge

[venturaEffect]

This is very kind from you.

gradientai/Llama-3-8B-Instruct-Gradient-4194k

[venturaEffect]

Hi @abhishekkrthakur , hope you're well. Did you made the custom merge space? Appreciate.

[venturaEffect]

Nevermind I've done it myself.

[abhishekkrthakur]

awesome. im sorry, i only have time later this week to attend to this