-
Notifications
You must be signed in to change notification settings - Fork 3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Hackathon No.106 add Stable Diffusion Pipeline Attend and excite (#5296)
* add attend_and_excite * rewrite AttendExciteCrossAttnProcessor by flatten * check code style * add record * modify sample code in README * move markdown file * remove md file * remove unusing code, replace paddle.equal by paddle.equal_all
- Loading branch information
Showing
7 changed files
with
1,115 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # Stable Diffusion | ||
|
|
||
| ## Using `StableDiffusionAttendAndExcitePipeline` with `PNDMScheduler` | ||
|
|
||
| Given a pre-trained text-to-image diffusion model (e.g., Stable Diffusion) the method, Attend-and-Excite, guides the generative model to modify the cross-attention values during the image synthesis process to generate images that more faithfully depict the input text prompt. | ||
|
|
||
| You can run this pipeline as so: | ||
|
|
||
| ```python | ||
| from pathlib import Path | ||
| import paddle | ||
| from ppdiffusers import StableDiffusionAttendAndExcitePipeline, PNDMScheduler | ||
|
|
||
|
|
||
| scheduler = PNDMScheduler.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="scheduler") | ||
| pipe = StableDiffusionAttendAndExcitePipeline.from_pretrained( | ||
| "CompVis/stable-diffusion-v1-4", | ||
| scheduler=scheduler, | ||
| ) | ||
|
|
||
| seed = 123 | ||
| prompt = "A playful kitten chasing a butterfly in a wildflower meadow" | ||
| token_indices = [3,6,10] | ||
|
|
||
| generator = paddle.Generator().manual_seed(seed) | ||
| image = pipe( | ||
| prompt=prompt, | ||
| token_indices=token_indices, | ||
| generator=generator, | ||
| ).images[0] | ||
|
|
||
| # save | ||
| output_dir = Path("output_pd") | ||
| prompt_output_path = output_dir / prompt | ||
| prompt_output_path.mkdir(exist_ok=True, parents=True) | ||
| image.save(prompt_output_path / f'{seed}.png') | ||
|
|
||
| ``` | ||
|
|
||
| And the above script running in V100-32GB generates the following image: | ||
|
|
||
| <center> | ||
| <img src="https://user-images.githubusercontent.com/40912707/226089491-0f3f66c2-3c88-4518-9ee4-d77debd50e9e.png"> | ||
| </center> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| # Stable Diffusion | ||
|
|
||
| ## 使用 `StableDiffusionAttendAndExcitePipeline` 和 `PNDMScheduler` | ||
|
|
||
| 给定一个预先训练好的文本到图像的扩散模型（例如Stable Diffusion），方法`Attend-and-Excite`能引导生成模型在图像生成过程中修改交叉注意力的数值，使得生成图片更忠实于输入文本的提示。 | ||
|
|
||
| 使用该pipeline的示例代码如下 | ||
|
|
||
| ```python | ||
|
|
||
| from pathlib import Path | ||
| import paddle | ||
| from ppdiffusers import StableDiffusionAttendAndExcitePipeline, PNDMScheduler | ||
|
|
||
|
|
||
| scheduler = PNDMScheduler.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="scheduler") | ||
| pipe = StableDiffusionAttendAndExcitePipeline.from_pretrained( | ||
| "CompVis/stable-diffusion-v1-4", | ||
| scheduler=scheduler, | ||
| ) | ||
|
|
||
| seed = 123 | ||
| prompt = "A playful kitten chasing a butterfly in a wildflower meadow" | ||
| token_indices = [3,6,10] | ||
|
|
||
| generator = paddle.Generator().manual_seed(seed) | ||
| image = pipe( | ||
| prompt=prompt, | ||
| token_indices=token_indices, | ||
| generator=generator, | ||
| ).images[0] | ||
|
|
||
| # save | ||
| output_dir = Path("output_pd") | ||
| prompt_output_path = output_dir / prompt | ||
| prompt_output_path.mkdir(exist_ok=True, parents=True) | ||
| image.save(prompt_output_path / f'{seed}.png') | ||
|
|
||
| ``` | ||
|
|
||
| 在V100-32GB显卡运行上述代码生成结果如下： | ||
|
|
||
| <center> | ||
| <img src="https://user-images.githubusercontent.com/40912707/226089491-0f3f66c2-3c88-4518-9ee4-d77debd50e9e.png"> | ||
| </center> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
|
|
||
|
|
||
| ## 示例prompt | ||
|
|
||
| 以[Hugging Face Spaces](https://huggingface.co/spaces/AttendAndExcite/Attend-and-Excite)选用的prompt和seed为例展示对齐效果如下表：`image_error`一栏分别展示了ppdiffusers和diffusers生成的图片及其绝对误差和相对误差在像素点的分布；`latents_error`一栏展示了模型在去噪过程中每一步生成的潜变量的最大绝对误差（蓝色线条）和最大相对误差（橙色线条）。 | ||
|
|
||
| | prompt | seed | image_error | latents_error | | ||
| | :----------------------------------------------------------- | :--: | ------------------------------------------------------------ | ------------------------------------------------------------ | | ||
| | A grizzly bear catching a salmon in a crystal clear river surrounded by a forest | 123 |  |  | | ||
| | A horse and a dog | 123 |  | ! | | ||
| | A mouse and a red car | 2098 |  |  | | ||
| | A painting of an elephant with glasses | 123 |  |  | | ||
| | A playful kitten chasing a butterfly in a wildflower meadow | 123 |  |  | | ||
| | A pod of dolphins leaping out of the water in an ocean with a ship on the background | 123 |  |  | | ||
|
|
||
|
|
||
|
|
||
|
|
||
|
|
||
| ## 更多的prompt | ||
|
|
||
|
|
||
| 按照论文的prompt构造方法尝试更多的prompt和seed对比ppdiffuser和diffusers的表现。 | ||
|
|
||
| 三种prompt模板分别如下，加粗的token表示使用注意力的token： | ||
|
|
||
| * a **{animal}** and a **{animal}** | ||
| * a **{animal}** and a {color} **{object}** | ||
| * a {colorA} **{objectA}** and a {colorB} **{objectB}** | ||
|
|
||
| 其中用于填充的单词如下表 | ||
|
|
||
| | category | words | | ||
| | -------- | ------------------------------------------------------------ | | ||
| | animals | cat, dog, bird, bear, lion, horse, elephant, monkey, frog, turtle, rabbit, mouse | | ||
| | objects | backpack, glasses, crown, suitcase, chair, balloon, bow, car, bowl, bench, clock, apple | | ||
| | colors | red, orange, yellow, green, blue, purple, pink, brown, gray, black, white | | ||
|
|
||
| 对每一个prompt再随机生成5个seed，对比结果绘制散点图如下，其中每个点代表该prompt和seed的设置下在该step上的误差，颜色越亮则表示散点越聚集： | ||
|
|
||
| | 绝对误差 | 相对误差 | | ||
| | -------------------------- | -------------------------- | | ||
| |  |  | | ||
|
|
||
| 在测试的105组prompt和seed中，有13个prompt和seed的组合生成的图片有较大差别，记录如下： | ||
|
|
||
| | prompt | seed | image_error | latents_error | | ||
| | :------------------------------------- | ---- | ------------------------------------------------------------ | ----------------------------------------------------------- | | ||
| | a dog and a pink crown | 1930 |  |  | | ||
| | a mouse and a yellow apple | 28 |  |  | | ||
| | a dog and a pink crown | 1285 |  |  | | ||
| | a mouse and a yellow apple | 60 |  | | | ||
| | a mouse and a yellow apple | 49 |  |  | | ||
| | a white chair and a green chair | 933 |  |  | | ||
| | a dog and a black bow | 148 |  |  | | ||
| | A horse and a dog | 27 |  | | | ||
| | a horse and a gray bow | 1753 | ! |  | | ||
| | A painting of an elephant with glasses | 1860 |  || | ||
| | a horse and a gray bow | 1597 |  | | | ||
| | a dog and a pink crown | 1743 | |  | | ||
| | a white chair and a green chair | 890 | |  | | ||
|
|
||
|
|
||
|
|
||
| #### 由于存在梯度，同样的prompt和seed设置下多次运行依旧存在误差 | ||
|
|
||
| | diffusers | ppdiffusers | | ||
| | ------------------------------------------------------------ | ------------------------------------------------------------ | | ||
| |  |  | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.