ControlNet implementation

[bonlime]

Hey @Zeju1997 , first of all thanks for a very interesting paper, haven't seen such a good one for long time.

I think i don't clearly understand what you do in your Controlnet experiments. in the paper you mentioned that ContorlNet conditions are passed through CN encoder and concatenated to SD input, according to the code this is what happens in oft/oft-control/oldm/oft_ldm.py but in original CN the conditions are added as residuals during forward rather that concatenated to SD. Why did you deviated from original CN implementation? Is it only to be able to actually fine-tune SD weights to follow control and be able to apply OFT to this process? (since in original CN implementation i think your method couldn't be applied)