Fix the bug that retains effects of a previous model when switching models during LoRA use.

[hako-mikan]

Description

This fix addresses the issue where switching models during the use of LoRA would cause the influence of the original model to affect the subsequent one. This problem has been reported as several issues, but it remains unresolved due to problems such as non-reproducibility. As discussed in the issues, the impact given during the model switch is very minor, and if the part has the same weight in both the pre- and post-switch models, the generated image does not change and cannot be reproduced.

issues
#13516
#12619
#12872
There may also be reports of this as a separate issue.

Checklist:

• I have read contributing wiki page
• I have performed a self-review of my own code
• My code follows the style guidelines
• My code passes tests

[hako-mikan]

I thought I had set it to the dev, but I made a mistake. My apologies.

[catboxanon]

This was also caught prior in #13178, which results in the diff being empty here since it's the same fix. Gonna go ahead and close this, but feel free to re-open if there's anything more to do.

[miguel234457]

still having this issue as of now, someone can look into this again?