Training Lora on 24gb vram locally

[indianaorz]

Not as much an issue but an fyi. I was getting OOM errors yesterday when trying to train a Lora on my 3090 (24gb vram) and 64gb ddr5.

One issue was the models which convert the dataset to weights (I believe there's 2) were set to local only and that needed to be fixed (set to False), also I had to comment out a line in the beginning which was trying to set the local rank to 0.

So then the training would try to run but I'd run out of vram. I did some research and realized those 2 models were only being used to preprocess the dataset so they didn't need to be used in training.

The solution I did was to build my own preprocess script and then modify the ace dataset class to load the data from the cache and modify the main training script to not use those 2 extra models. Took a while to get everything right but the training is working great now!

In hindsight I probably could have just implemented better memory management in the main training script to make sure the preprocessing models are unloaded from memory before trying to begin training.

I ran it over night and I'm about at step 40k on a 70 song dataset and it's learning it really well with default Lora settings. Not sure how it generalizes yet but I'll test it out when it's done cooking.

Oh also for some reason I had to run in precision = "32-full"

Thanks for all the work y'all have done, this is awesome and it's exactly what I was looking for at exactly the right time!

[DumoeDss]

Wow, that's awesome! We haven't had time to implement these optimizations yet. Looking forward to merging your PR! 🚀

[indianaorz]

I'll see how the training ends up, I haven't done a PR before on github for a public project, but if it turns out well I could do something to provide the code to help out others. I don't think it'll play nicely with the existing code so might have to make some new train scripts if I get some free time after work in the next day or two

[DarkAlchy]

I am still trying to get some info on the format, data, etc... it wants as I have trained FFT, and loras since SD1.4, and used to that, but not sure how to do that for this? Captioning, data, etc...

[woct0rdho]

I'm also trying this. My experience is that currently automatic captioning models like CLAP are pretty bad at describing what we actually want from musics. We can try collecting real user feedbacks from music websites (in China there is NetEase Cloud Music, known for interesting user comments. There is also Bilibili with comments annotated at specific times in the music. For English comments I tried SoundCloud but there are too much spam comments.)

Maybe we can also generate text descriptions from videos (such as movies and MTVs) and align them with the background musics, but I haven't tried this. There are interesting cases like https://search.bilibili.com/all?keyword=%E4%BD%86%E6%98%AF%E7%9C%8B%E5%9B%BE%E8%AF%B4%E8%AF%9D

At this point maybe we can start building a better captioning model.

[indianaorz]

Okay so one more small learning after a day of training, the 70 songs I trained with were all different durations and it seems that the model learned to recreate the song just from the duration of the song and totally ignores the prompts. Going to try another run after padding the files with silence so they're all the same length. Gonna test with a smaller dataset to see if I can get some generalizations. I believe this is likely because of the short duration of the songs I'm training on (about 40s average), and the model probably doesn't have much in that area.

Really impressed by its ability to learn the songs though

[filliptm]

@indianaorz would love these scripts, been trying to get the thing training and I'm just getting OOM with my 48gb card Q.Q

[indianaorz]

@indianaorz would love these scripts, been trying to get the thing training and I'm just getting OOM with my 48gb card Q.Q

If you can wait another 10 hours or so, I'll see if I can push what I have after work

[rkfg]

What is the dataset format exactly? I looked around and the documentation on HF is kinda overwhelming for such a small task. It assumes you load the already existing datasets instead of creating your own from scratch, and even then it seems to be created programmatically with save_to_disk method. At least that's what I found in the various answers. So you're not supposed to manually create these JSONs?

I tried making a JSON file with the structure from README and it's not correct I think. The HF code looks for at least one file, dataset_dict.json, but I'm not sure what should be in it. I think a basic example would be great to have, like we have these 3 songs with these 3 genres (tags) and lyrics, here's how you should organize the files: and the JSON content. Because really it's a small fine tuning task with quite small datasets usually so there's no need to involve sharding/validation/parquet/Amazon S3 (lol) and all that stuff. Sure, it would help if you train on millions of files but for the loras something more community-standard would be better, like for image or video training it's common to use image/txt pairs.

[ChuxiJ]

If your dataset is too small, the model will quickly overfit it. Using a validation set to monitor the training progress is a good habit. Also, if the task is too difficult (far beyond what the model can generate), a small dataset often performs badly. A best practice is to start by training a LoRA closer to the model outputs. For example, in the context of genres like dance music, piano, or guitar solo. This will help you understand how LoRA finetuning works.

[rkfg]

By small I mean around 100 songs maybe. The question is how to start at all, the training instructions skip this very important part. I made plenty of loras before for text2image and text2video models so I know how this works more or less. The other tools like kohya_ss and OneTrainer (that I personally prefer) use a simple image/text pairs, it's easy to make a dataset and test something. But here I'm kinda stuck.

[ChuxiJ]

Data Preparation Process

1. First, prepare your audio files. For example, all of Chopin's piano waltzes.

2. Add default annotations to them, such as Chopin, piano, waltz, and classical music.

3. Write a script to convert the annotations and data into the target format specified in the README instructions. If you don't know how to write it, you can ask an LLM (Large Language Model) to do it. Just tell it your requirements. Roocode is also a good option.

4. Start the training.

[ChuxiJ]

For something like One Trainer, it is generally supported by third-party communities. The official side will create more detailed documentation after releasing the first LoRA model. There are a lot of questions and issues, and since my energy is limited, I may not be able to teach everyone step by step. Therefore, we need those who have successfully trained in the community to create some tutorial documents. Thank you all.