multi images as conditions,single control net

[Qiuhao-Wu]

Hello, I met several problems during training my controlnet. I need multi control, just one controlnet, so I concat the conditions along the axis. During training, if I set the dataset just one, the model will converge after 4000 steps. However, when my datasets extend to 100K, my model will not converge after 30K steps, why? My batch size is 4, use 4 v100, so the batchsize is 16.

Can anybody help me?