[Community Pipeline] Imagic: Text-Based Real Image Editing with Diffusion Models

[apolinario]

Intro

Community Pipelines are introduced in diffusers==0.4.0 with the idea of allowing the community to quickly add, integrate, and share their custom pipelines on top of diffusers.

You can find a guide about Community Pipelines here. You can also find all the community examples under examples/community/. If you have questions about the Community Pipelines feature, please head to the parent issue.

Idea: Imagic: Text-based Real Image Editing with Diffusion Models

This pipeline aims to implement this paper to Stable Diffusion, allowing for real-world image editing. Example from the paper:

[Alx-AI]

Would love to see this added, notebook implementation here for reference https://github.com/justinpinkney/stable-diffusion/blob/main/notebooks/imagic.ipynb

[asofiaoliveira]

I would like to work on this

[patrickvonplaten]

Awesome @asofiaoliveira !

Feel free to open a PR and to attach it here

[MarkRich]

tried my hand at this here: #958 please let me know if there's any comments!

[asofiaoliveira]

I guess I'll leave it to @MarkRich then 😅

[0xdevalias]

FYI, it looks like that PR has been merged now:

• Add imagic to community pipelines #958

And the implementation is available here:

• https://github.com/huggingface/diffusers/tree/main/examples/community#imagic-stable-diffusion

Can this issue be closed now?

[njucckevin]

FYI

Is there someone try the effect with this code? I failed to achieve the effect in the paper (for example, let a doy playing with a toy) with the Imagic Stable Diffusion.

[askerlee]

@0xdevalias It seems there is some issue with the implementation? In train(), the text embedding is optimized first. But prior to that, unet and text_encoder are set to disable BP:

    self.unet.requires_grad_(False)
    self.text_encoder.requires_grad_(False)

Does this mean there won't be valid gradients back propagated from the loss to the text embedding?
I'm not very sure. Thanks.

[0xdevalias]

@askerlee I don't know anything about the implementation, nor really used it. I just noticed the PR and figured I'd link it here

[ShaoTengLiu]

@njucckevin I also get wrong results.
For example, here are the results for "A photo of a bird spreading wings":

Can anyone give me some hints?

[BoyuanJiang]

@njucckevin I also get wrong results. For example, here are the results for "A photo of a bird spreading wings":

Can anyone give me some hints?

I also cannot reproduce the result in the paper

[ghost]

Hey guys,
here is what I have:

[ghost]

I am using stable diffusion to replicate their result on imagen(non open source) and used 500 optimisation steps and 1000 fine-tuning steps. The best result come from the last picture where we have lambda coefficient of 1. Not quite sure why the change comes so late.

[tasinislam21]

@njucckevin I also get wrong results. For example, here are the results for "A photo of a bird spreading wings":

Can anyone give me some hints?

I am getting the exact same problem. What is the solution to this?

[tasinislam21]

FYI

Is there someone try the effect with this code? I failed to achieve the effect in the paper (for example, let a doy playing with a toy) with the Imagic Stable Diffusion.

I am also facing this problem. When I set anything below 1 for lambda coefficient, I get an image that is the same as the input image. If I change the coefficient by more than 1 then I get an image of a random bird spreading its wing.

[shwetabhardwaj44]

I am using stable diffusion to replicate their result on imagen(non open source) and used 500 optimisation steps and 1000 fine-tuning steps. The best result come from the last picture where we have lambda coefficient of 1. Not quite sure why the change comes so late.

Hi @Kathy-Peng I am also following the same config and my code is written as below. Could you kindly confirm if there is any missing step in this. In my results, the edited image also doesn't change much even at lambda coefficient = 1.1

model_id = "CompVis/stable-diffusion-v1-4"
pipe = DiffusionPipeline.from_pretrained(
                          model_id,
                          cache_dir=CACHE_DIR,
                          safety_checker=None,
                          use_auth_token=True,
                          custom_pipeline="imagic_stable_diffusion",
                          scheduler = DDIMScheduler(\
                                      beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear",\
                                      clip_sample=False, set_alpha_to_one=False)
                          )
pipe.to("cuda")
generator = torch.Generator("cuda").manual_seed(0)

alphas = [0.8, 0.9, 1, 1.1, 1.2]
guidance_scale = [6.5, 7.0, 7.5, 8.0, 8.5, 9.5]
# curr_prompt  is given
# curr_image is loaded

res = pipe.train(curr_prompt, image=curr_image, generator=generator)
reconstructed_image = res.images[0]

## Once the pipeline is trained, run inference with different alphas and text guidance scales.
for alpha in alphas:
        for text_guide in guidance_scale:                                                                             
                edited_image  = pipe(num_inference_steps=50, alpha=alpha, guidance_scale=text_guide).images[0]

[HashmatShadab]

Was anyone able to reproduce the results?

[witcherofresearch]

@HashmatShadab @njucckevin @BoyuanJiang @ShaoTengLiu @1702609 @shwetabhardwaj44 Please check out my new paper Forgedit, which is much faster than Imagic and edting results are way much better than Imagic with Stable Diffusion. https://github.com/witcherofresearch/Forgedit/
Here is a few examples on the editing results of Forgedit. For example, for the target prompt 'A photo of a bird spreading wings.' and original image

using the DreamBoothForgedit, we could get