Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Confusion about model.save_pretrained("output_dir") #452

Closed
Air-Spring opened this issue May 16, 2023 · 3 comments
Closed

Confusion about model.save_pretrained("output_dir") #452

Air-Spring opened this issue May 16, 2023 · 3 comments

Comments

@Air-Spring
Copy link

I have used a pretrained model to fine tuning with lora. Which confused me is that the output_save_file including adapter_config.json and adapter_model.bin is different from the tutorial. Except the adapter_config.json and adapter_model.bin, there are some checkpoints directory. Does these checkpoints file influence the load of lora weights? And my adapter_model.bin is just 1kB, it seems as over small compared to official 19M. What factors may cause this issue, is it normal? How to adjust the parameters to make the size of adapter_model.bin bigger?

@ingo-m
Copy link

ingo-m commented May 17, 2023

AFAIK the following parameters in LoraConfig should influence the size of adapter_model.bin:

  • The r parameter (AFAIK greater r -> larger file size? But I'm also not sure how exactly this works.)
  • The bias param (whether bias also gets trained; I suppose if you also train the bias, the filesize should be larger)
  • target_modules - which modules of the base model will be affected by LoRA
  • Which modules_to_save (as defined in LoraConfig)
  • Probably also the lora_alpha parameter in LoraConfig?

And of course which base model you choose will have an effect.

I don't know what the role of the checkpoints is.

Using "bigscience/bloom-560m" as a base model, and the following LoraConfig, I get a adapter_model.bin filesize of 1.03 GB.

LoraConfig(
    r=16,
    lora_alpha=32,
    bias="none",
    modules_to_save=["lm_head"],
)

@ingo-m
Copy link

ingo-m commented May 26, 2023

By the way, after finetuning "bigscience/bloom-560m", when I save the finetuned model I do get a plausible model filesize, but saving / loading doesn't work as expected (model does not remember finetuning). #503

@github-actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants