-
Notifications
You must be signed in to change notification settings - Fork 284
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
about feature modulation. #120
Comments
你好!feature modulation指的是ControlNet(右边的黄色部分)的输出以什么样的方式来控制,或者说融入进UNet的feature,最简单的方式就跟ControlNet一样,直接用一个加法,也有更复杂一点的,比如说SUPIR用的scale + shift,PASD用的attention。可以看一下对应的代码。 |
好滴好滴,谢谢您的回复,也就是说只有右边黄色部分follow controlnet的办法只用一个加法融入了unet,只有图中紫色框中的部分是可训练的(1.controlnet,2.zero conv 3.也是zero conv),是这样吗? 以及之前您的code版本中([v1] Tue, 29 Aug 2023 07:11:52)在做inference时提到latent image guidance是关的,我在对应的代码mse_guidance中只在inference.py中发现有这个函数的引用,是不是在train的过程中这个部分latent image guidance也是关的呀? 谢谢您! |
hi, thanks for the excellent work!
I am a little bit confused about the feature modulation part. as the paper says,
3) feature modulation. The previous condition network outputs multi-scale features, which will be used to modulate the intermediate features of the frozen UNet denoiser. Following ControlNet, we only modulate the middle block features and the skipped features through addition operation.
想请您再解释一下这个feature modulation具体是怎么实现的吗?我理解右侧的黄色部分是unet的trainable copy,zero conv部分也是可以训练的,但是我不太明白feature modulation具体是干什么的,想请您帮忙解释一下,谢谢!
The text was updated successfully, but these errors were encountered: