feat: add repaint #974

Revist · 2022-10-25T10:38:47Z

Added repaint pipeline based on https://arxiv.org/pdf/2201.09865.pdf @anton-l

In tests i was not sure where you take numbers to compare against from so I run the pipeline and pasted what it produces for me:
[0.14537135, 0.10728511, 0.08822048, 0.15828621, 0.11806837, 0.11007798, 0.15231332, 0.1214554, 0.15475643]

HuggingFaceDocBuilderDev · 2022-10-25T10:42:17Z

The documentation is not available anymore as the PR was closed or merged.

anton-l

Hi @Revist! Excited to see a working RePaint port, thank you so much for working on it!

Overall the implementation is super nice, just left a couple of suggestions to bring it closer to the Stable Diffusion inpainting API (which got added after my original repaint PR).

Now I wonder if it's possible to move the noising & masking logic from RePaintScheduler to RePaintPipeline, so that it can be used with other schedulers like PNDM or K-LMS? By using just step() and add_noise(), similar to the unsupervised inpainting pipeline for Stable Diffusion: https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint_legacy.py#L46
What do you think, would that be possible to do here?

tests/pipelines/repaint/test_repaint.py

src/diffusers/pipelines/repaint/pipeline_repaint.py

src/diffusers/schedulers/scheduling_repaint.py

Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

Revist · 2022-10-28T14:02:15Z

Hi @anton-l !

Was doing some reading as i was not that familiar with these, but it seems that although theoretically that should be possible (after all RePaint just repeats some steps) in practice you have some problems like here:
https://github.com/huggingface/diffusers/blob/main/src/diffusers/schedulers/scheduling_pndm.py#L209
where the step depends on some counter and not only on timestep itself. To include PNDM first thing would be to remove such dependency.
Then one could build repaint timesteps and ask with step() and add_noise() to go forward and backward, where how to go forward would be recognized by timestep value and scheduler name.

Revist · 2022-10-28T15:25:48Z

Reading further in to make a step with PNDM you do not need a single model_output at single time but at t, t+delta/2 and t+delta (from different inputs in fact so we have 4 passes) so that changes logic here:
https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/pndm/pipeline_pndm.py#L85
where you would like to give a single model_output to step, you would have to pass model (unet) and image (x_t) instead.
You skipped this problem by creating virtual steps in this scheduler which are not really steps, then using this counter. But i would say the problems arise from this (and makes code complex imo).
In other words it ought to be true that if self.timesteps contains some step then this is really a step in time and not some placeholder for making calculations.

Revist · 2022-11-02T10:34:09Z

@anton-l I am not sure if you agree with the above comments. If so I could take a look at how to change the code to the PNDM (and possibly other) schedulers however that seems like significant changes across code.

anton-l · 2022-11-02T13:24:31Z

@Revist I've tried refactoring it a bit and completely share your concerns now, let's leave the scheduler as is for now!

Pinging @patrickvonplaten @patil-suraj for a second review, as this is a new pipeline and scheduler :)

src/diffusers/pipelines/repaint/pipeline_repaint.py

patrickvonplaten · 2022-11-02T17:47:31Z

tests/pipelines/repaint/test_repaint.py

+            "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/"
+            "repaint/celeba_hq_256_result.png"
+        )
+        expected_image = np.array(expected_image, dtype=np.float32) / 255.0


Ok for now, but let's make sure to pass to numpy image tests in the very near future

Will fetch the numpy images for all integration tests in a bulk PR 👍

patrickvonplaten

Looks good to me ! Before merging let's please make sure to add it to the docs as well and maybe reach out to the original author here?

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

anton-l · 2022-11-02T21:53:56Z

Added the docs for the pipeline: https://moon-ci-docs.huggingface.co/docs/diffusers/pr_974/en/api/pipelines/repaint
And the scheduler: https://moon-ci-docs.huggingface.co/docs/diffusers/pr_974/en/api/schedulers#diffusers.RePaintScheduler

Gently pinging the RePaint author @andreas128 for an optional review and any comments, in case out email thread gets lost 🤗

anton-l · 2022-11-03T14:48:26Z

@Revist the pipeline will be included in today's diffusers==0.7.0 release! 🥳

If you're interested, maybe we could work together on a Spaces demo similar to SD-inpainting https://huggingface.co/spaces/runwayml/stable-diffusion-inpainting to promote the pipeline further, let me know what you think!
Also would be interesting to try integrating OpenAI's improved-diffusion models to completely reproduce the RePaint paper 🔥

Revist · 2022-11-04T09:54:47Z

@anton-l Sounds interesting. What models would you like to integrate from improved diffusion? I was thinking of taking a look into #894 as this may be of use to our group in the near future. Maybe we could speak on some other communicator?

anton-l · 2022-11-17T17:02:21Z

@Revist ping me ( @anton-l ) on the HF discord: https://discord.gg/G7tWnz98XR
We have a diffusers core-contributors channel there, where we can discuss everything further :)

* feat: add repaint * fix: fix quality check with `make fix-copies` * fix: remove old unnecessary arg * chore: change default to DDPM (looks better in experiments) * ".to(device)" changed to "device=" Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> * make generator device-specific Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> * make generator device-specific and change shape Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> * fix: add preprocessing for image and mask Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> * fix: update test Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> * Update src/diffusers/pipelines/repaint/pipeline_repaint.py Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * Add docs and examples * Fix toctree Co-authored-by: fja <fja@zurich.ibm.com> Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Anton Lozhkov <anton@huggingface.co>

feat: add repaint

a664721

anton-l self-requested a review October 25, 2022 10:43

fja and others added 4 commits October 25, 2022 12:48

fix: fix quality check with make fix-copies

1f49481

fix: remove old unnecessary arg

9e56ee8

Merge branch 'main' into fja_add_repaint

4729b92

chore: change default to DDPM (looks better in experiments)

52df0ba

anton-l reviewed Oct 26, 2022

View reviewed changes

Revist and others added 5 commits October 27, 2022 12:41

".to(device)" changed to "device="

8a54f4c

Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

make generator device-specific

9c6a4c0

Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

make generator device-specific and change shape

cceb2c1

Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

fix: add preprocessing for image and mask

74e5e3d

Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

fix: update test

e1a28a3

Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

anton-l requested review from patrickvonplaten and patil-suraj November 2, 2022 13:24

patrickvonplaten reviewed Nov 2, 2022

View reviewed changes

src/diffusers/pipelines/repaint/pipeline_repaint.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Nov 2, 2022

View reviewed changes

patrickvonplaten approved these changes Nov 2, 2022

View reviewed changes

Update src/diffusers/pipelines/repaint/pipeline_repaint.py

36dfa24

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

anton-l mentioned this pull request Nov 2, 2022

[Community, WIP] Add RePaint inpainting pipeline #243

Closed

anton-l added 3 commits November 2, 2022 21:06

Merge branch 'main' into fja_add_repaint

0c58054

Add docs and examples

f13f5da

Fix toctree

6217e3d

patrickvonplaten approved these changes Nov 3, 2022

View reviewed changes

Merge branch 'main' into fja_add_repaint

7ea0dc7

anton-l merged commit d38c804 into huggingface:main Nov 3, 2022

Revist mentioned this pull request Nov 9, 2022

Does RePaintPipeline work with Stable diffusion? #1213

Closed

Revist deleted the fja_add_repaint branch November 11, 2022 13:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add repaint #974

feat: add repaint #974

Revist commented Oct 25, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Oct 25, 2022 •

edited

Loading

anton-l left a comment

Revist commented Oct 28, 2022

Revist commented Oct 28, 2022 •

edited

Loading

Revist commented Nov 2, 2022

anton-l commented Nov 2, 2022

patrickvonplaten Nov 2, 2022

anton-l Nov 2, 2022

patrickvonplaten left a comment

anton-l commented Nov 2, 2022 •

edited

Loading

anton-l commented Nov 3, 2022

Revist commented Nov 4, 2022

anton-l commented Nov 17, 2022

feat: add repaint #974

feat: add repaint #974

Conversation

Revist commented Oct 25, 2022 • edited Loading

HuggingFaceDocBuilderDev commented Oct 25, 2022 • edited Loading

anton-l left a comment

Choose a reason for hiding this comment

Revist commented Oct 28, 2022

Revist commented Oct 28, 2022 • edited Loading

Revist commented Nov 2, 2022

anton-l commented Nov 2, 2022

patrickvonplaten Nov 2, 2022

Choose a reason for hiding this comment

anton-l Nov 2, 2022

Choose a reason for hiding this comment

patrickvonplaten left a comment

Choose a reason for hiding this comment

anton-l commented Nov 2, 2022 • edited Loading

anton-l commented Nov 3, 2022

Revist commented Nov 4, 2022

anton-l commented Nov 17, 2022

Revist commented Oct 25, 2022 •

edited

Loading

HuggingFaceDocBuilderDev commented Oct 25, 2022 •

edited

Loading

Revist commented Oct 28, 2022 •

edited

Loading

anton-l commented Nov 2, 2022 •

edited

Loading