Tracking: LoRA

[jon-chuang]

Here are some outstanding issues for LoRA:

• Base implementation (Add LoRA support #820)
• Improve LoRA application time with SIMD (AVX, AVX2) (perf(ggml): tall and skinny GEMM for LoRA: F32 mul_mat([16 X 5120], [16 X 5120]) takes 120ms - 24x slower than expected #956)
• Improve LoRA loading time with MMAP on base model
  • quantizing an MMAPed float16 base model that has had LoRA applied
• Interpolation of weights (start with 1, look into multiple) (feature: linearly interpolating one or multiple LoRA with base model #905)
• Export loaded model to binfile (standalone in CLI with LoRA (--export-lora flag); interactively (?)) (feature: interactively exporting loaded model to binfile #904)
• Investigate extracting LoRA for arbitrary models (see PEFT issue)

[captainzero93]

really desperate to start uing LoRA, however I use GPTQ-4bit-32g.GGML will this be a problem?

[jon-chuang]

So far, we've seen issues with quality on 4 bit base model. That being said, it has produced reasonable output for me some of the time. It is still under investigation.

[bmanturner]

Would this be a good place to request support for multiple lora adapters sharing a similar base model? See here for inspiration: lm-sys/FastChat#1905

[Green-Sky]

Improve LoRA loading time with MMAP on base model

was done here #2095

also not sure this issue is the right one

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.