Add ability to set lora-base file

[kmilner]

Used the same argument as llama.cpp (--lora-base). I have tested this with GGJT_3 files (using Llama 30B q4_0 with f16 as the lora-base).

[LostRuins]

May I ask what's the purpose of this flag? You can already pick any model to load when launching with --lora. How does this flag help?

[kmilner]

--lora-base is used so that you can apply the LoRA directly to a larger base model (like an f16 model) even if you can't fit it in memory. The LoRA changes weights of the model but it may change them in a way that is rounded differently on a quantized model versus the base model, so in general if you have the f16 model available you should apply the LoRA to that.

From what I understand, llama.cpp lets you load the LoRA against a base model without using much memory because it only needs to check the specific weights in LoRA against the base model and adjust them appropriately in the actual loaded model. But I'm not an expert, that's more of an assumption on my part.

If you load a LoRA with llama.cpp against a quantized model without providing an f16 or f32 base model, llama.cpp generates a warning about this: https://github.com/LostRuins/koboldcpp/blob/concedo/llama.cpp#L2769

[LostRuins]

ah okay I get it. I will review this

[LostRuins]

This is now added as an optional second parameter to the --lora argument. So you can load it like --lora [lora_filename] [base_filename]

[LostRuins]

Available in newest version