how to train a lora model of an avatar without body via dreambooth

[joey0922]

I wanna train a lora model of an avatar which doesn't have body via dreambooth, but the model often try to generate a body for the avatar which is only a head like a box. How to prevent it? Thanks.

[asomoza]

Hi, it's not clear to me, do you want to train something like this one but with just the faces?

You just need to train it with a lot of examples of what you want, since is more like a concept than a subject, you'll need a lot more images.

[asomoza]

If you repeat the images, which if you don't use augmentation is the same as doing more epochs, will make the model overfit faster.

I don't understand what you're doing though, you're trying to use dreambooth to train the backgrounds and the concept at the same time?

If you don't have that much images you can do training with dreambooth but it will be more rigid and harder to use, when people say you don't need that much images, f you look at the loras they do and how to use them, they always are rigid with the same pose and face, not that much variation and also you can't use them with other loras.

Probably the factor that matters the most for finetuning and lora training is the dataset, if you have a small/bad dataset, there is not that much you can do to train something good.

[joey0922]

I just want to train concept，but the model is much easier to learn the backgrounds of train dataset. It learned to generate backgrounds of train dataset very well when it generate the concept badly. What I want is opposite，generating concept well while the backgrounds are from base model not train dataset.

[asomoza]

With dreambooth if you describe well the background it shouldn't learn it with the rest of the concept, but yeah, this is only if you don't repeat the same background over your dataset, if you do, it will learn it.

There's a function we don't have but onetrainer has it, "Masked Training" which uses masks to train better the subject. I haven't tested it and I don't know how well it works but I've read it's good, maybe you can get better results with it.

[joey0922]

Thanks, I will take a shot if I can. I still have a confusion that why the concept or subject I want to learn is broken or even missing in my generated images. I think if I got this situation, the model I trained is under-fitting? I found it is possible to generate images with concept broken no matter how well the model I trained. What can I do to decrease the ratio of broken concept in generated images?

[asomoza]

it can be by a number of reasons, without actually looking at your code and dataset it's really hard to tell.

I can tell you that what you want to train is not for beginners, most people that have success doing something like this have trained a lot of loras before and learned the in and outs of lora training.

You probably should start with something easier and gradually go for something more complicated, as I see it, this is the order of difficulty for training loras:

1. Human face
2. Human face and body or something like a dog, etc.
3. Similar style to what the model knows (like variations of anime style)
4. Completely new style that the model doesn't know
5. Completely new concept and style that the model doesn't know

You can also use a tested dataset and learn how to caption and how it does affect the training, for example an easy to start one is tuxemon, and if you want a style you can use the vintage ads one.