Does RePaintPipeline work with Stable diffusion?

[FBehrad]

Hello,
First of all, thank you so much for adding RePaintPipeline. This pipeline works much better than stable diffusion inpainting when I use DDPMs (such as ddpm-ema-bedroom-256 and ddpm-bedroom-256) . However, when I use CompVis/stable-diffusion-v1-4, some bugs appear. For example, I get the following error:

TypeError: set_timesteps() takes from 2 to 3 positional arguments but 5 were given

I wonder is it possible to use stable diffusion as its generator?
Here, is my code:

scheduler = LMSDiscreteScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", num_train_timesteps=1000)

pipe = RePaintPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", scheduler=scheduler) 
pipe = pipe.to("cuda")

generator = torch.Generator(device="cuda").manual_seed(0)
output = pipe(
    original_image=img,
    mask_image=msk,
    num_inference_steps=250,
    eta=0.0,
    jump_length=10,  
    jump_n_sample=10,
    generator=generator,
)
inpainted_image = output.images[0]

[Revist]

Hi @FBehrad,

The thing is you should supply repaint scheduler to the pipeline not LMSDescrete. In the pr #974 there was discussion about making repaint usable with all schedulers, but long story short for now it works only with DDIM and (because DDIM generalizes DDPM) DDPM.

Take a look here https://github.com/huggingface/diffusers/blob/main/tests/pipelines/repaint/test_repaint.py

[patrickvonplaten]

cc @anton-l

[FBehrad]

Hi @FBehrad,

The thing is you should supply repaint scheduler to the pipeline not LMSDescrete. In the pr #974 there was discussion about making repaint usable with all schedulers, but long story short for now it works only with DDIM and (because DDIM generalizes DDPM) DDPM.

Take a look here https://github.com/huggingface/diffusers/blob/main/tests/pipelines/repaint/test_repaint.py

Thank you for your quick response.

[patrickvonplaten]

Gently ping @anton-l

[anton-l]

To expand on @Revist's answer a bit: the RePaint pipeline only works for image diffusion, as opposed to latent diffusion (as in Stable Diffision). Combining the two would be an awesome community pipeline project! Opened an issue for it here: #1333

[Randolph-zeng]

Incase anyone here is interested in the progress of this feature request, there is a community contribution PR going on right now, but still I believe we might need some help to figure out why it is not working as expected, any help would be appreciated : ) #1333 (comment)
#1602