Confusion about model.save_pretrained("output_dir")

[Air-Spring]

I have used a pretrained model to fine tuning with lora. Which confused me is that the output_save_file including adapter_config.json and adapter_model.bin is different from the tutorial. Except the adapter_config.json and adapter_model.bin, there are some checkpoints directory. Does these checkpoints file influence the load of lora weights? And my adapter_model.bin is just 1kB, it seems as over small compared to official 19M. What factors may cause this issue, is it normal? How to adjust the parameters to make the size of adapter_model.bin bigger?

[ingo-m]

AFAIK the following parameters in LoraConfig should influence the size of adapter_model.bin:

• The r parameter (AFAIK greater r -> larger file size? But I'm also not sure how exactly this works.)
• The bias param (whether bias also gets trained; I suppose if you also train the bias, the filesize should be larger)
• target_modules - which modules of the base model will be affected by LoRA
• Which modules_to_save (as defined in LoraConfig)
• Probably also the lora_alpha parameter in LoraConfig?

And of course which base model you choose will have an effect.

I don't know what the role of the checkpoints is.

Using "bigscience/bloom-560m" as a base model, and the following LoraConfig, I get a adapter_model.bin filesize of 1.03 GB.

LoraConfig(
    r=16,
    lora_alpha=32,
    bias="none",
    modules_to_save=["lm_head"],
)

[ingo-m]

By the way, after finetuning "bigscience/bloom-560m", when I save the finetuned model I do get a plausible model filesize, but saving / loading doesn't work as expected (model does not remember finetuning). #503

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.