Training loss is NaN now.

[Strive21]

When will the latest version of the code and data processing code be released?

[SHYuanBest]

Thanks for your interest. Is the training loss NaN at the beginning, what dataset did you use? The latest version of the code may not be released so soon. We will prioritize the release of data processing code and the integration of ConsisID into diffusers.

[Strive21]

Thanks for your interest. Is the training loss NaN at the beginning, what dataset did you use? The latest version of the code may not be released so soon. We will prioritize the release of data processing code and the integration of ConsisID into diffusers.

I downloaded your dataset and processed it appropriately, using CogvideoX-5B-I2V to initialize the weights，which bs is 5 and lr is 3e-7. It has loss in the initial training, but NaN appears after about 500 iterations. Is it because I processed the data wrong? And “fail to detect face using insightface, extract embedding on align face“ occurs during training。

[SHYuanBest]

Oh, I see. This may be a problem with MM-DiT. The training is very unstable because the activation value of the middle layer may be very large, resulting in loss NaN. You can try turning on EMA, gradient accumulation, increasing batchsize, and reducing the learning rate. Another method is to add a regularization term to the output of the middle layer.

[SHYuanBest]

fail to detect face using insightface, extract embedding on align face this warning cannot be avoided because facexlib may not be able to detect the face, and the code will automatically skip this training sample.

[SHYuanBest]

or you can try to train only LoRA instead of all parameters.

Oh, I see. This may be a problem with MM-DiT. The training is very unstable because the activation value of the middle layer may be very large, resulting in loss NaN. You can try turning on EMA, gradient accumulation, increasing batchsize, and reducing the learning rate. Another method is to add a regularization term to the output of the middle layer.

[SHYuanBest]

When will the latest version of the code and data processing code be released?

we have release the data processing code, please refer to here for more details.

[Strive21]

When will the latest version of the code and data processing code be released?

we have release the data processing code, please refer to here for more details.

Thank you！I'll give it a try.

[glimmer16]

Thanks for your interest. Is the training loss NaN at the beginning, what dataset did you use? The latest version of the code may not be released so soon. We will prioritize the release of data processing code and the integration of ConsisID into diffusers.

I downloaded your dataset and processed it appropriately, using CogvideoX-5B-I2V to initialize the weights，which bs is 5 and lr is 3e-7. It has loss in the initial training, but NaN appears after about 500 iterations. Is it because I processed the data wrong? And “fail to detect face using insightface, extract embedding on align face“ occurs during training。

Hi! Have you solved this problem? I meet the same issue and wonder which way to avoid loss NaN.

[SHYuanBest]

Thanks for your interest. Is the training loss NaN at the beginning, what dataset did you use? The latest version of the code may not be released so soon. We will prioritize the release of data processing code and the integration of ConsisID into diffusers.

I downloaded your dataset and processed it appropriately, using CogvideoX-5B-I2V to initialize the weights，which bs is 5 and lr is 3e-7. It has loss in the initial training, but NaN appears after about 500 iterations. Is it because I processed the data wrong? And “fail to detect face using insightface, extract embedding on align face“ occurs during training。

Hi! Have you solved this problem? I meet the same issue and wonder which way to avoid loss NaN.

You may need to construct a higher dataset to continue finetuning ConsisID, or have a larger batch size. Since ConsisID is trained on a higher quality internal dataset, if it continues to be trained on the ConsisID-Preview-Data, it is likely to get worse.

[SHYuanBest]

Or you can load the ckpt of CogVideoX-5B-I2V for training IPT2V from scratch. (Instead of load ConsisID-Preview for continue finetuning.)

[SHYuanBest]

Some solutions can refer to #31.

[Strive21]

Thanks for your interest. Is the training loss NaN at the beginning, what dataset did you use? The latest version of the code may not be released so soon. We will prioritize the release of data processing code and the integration of ConsisID into diffusers.

I downloaded your dataset and processed it appropriately, using CogvideoX-5B-I2V to initialize the weights，which bs is 5 and lr is 3e-7. It has loss in the initial training, but NaN appears after about 500 iterations. Is it because I processed the data wrong? And “fail to detect face using insightface, extract embedding on align face“ occurs during training。

Hi! Have you solved this problem? I meet the same issue and wonder which way to avoid loss NaN.

I try a larger batch size and solve this problem.