-
Notifications
You must be signed in to change notification settings - Fork 5.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add InstructPix2Pix pipeline (#2040)
* being pix2pix * ifx * cfg image_latents * fix some docstr * fix * fix * hack * fix * Apply suggestions from code review Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * add comments to explain the hack * move __call__ to the top * doc * remove height and width * remove depreications * fix doc str * quality * fast tests * chnage model id * fast tests * fix test * address Pedro's comments * copyright * Simple doc page. * Apply suggestions from code review * style * Remove import * address some review comments * Apply suggestions from code review Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> * style Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
- Loading branch information
1 parent
3ecbbd6
commit e5ff755
Showing
9 changed files
with
1,085 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,72 @@ | ||
<!--Copyright 2023 The HuggingFace Team. All rights reserved. | ||
|
||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with | ||
the License. You may obtain a copy of the License at | ||
|
||
http://www.apache.org/licenses/LICENSE-2.0 | ||
|
||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on | ||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the | ||
specific language governing permissions and limitations under the License. | ||
--> | ||
|
||
# InstructPix2Pix: Learning to Follow Image Editing Instructions | ||
|
||
## Overview | ||
|
||
[InstructPix2Pix: Learning to Follow Image Editing Instructions](https://arxiv.org/abs/2211.09800) by Tim Brooks, Aleksander Holynski and Alexei A. Efros. | ||
|
||
The abstract of the paper is the following: | ||
|
||
*We propose a method for editing images from human instructions: given an input image and a written instruction that tells the model what to do, our model follows these instructions to edit the image. To obtain training data for this problem, we combine the knowledge of two large pretrained models -- a language model (GPT-3) and a text-to-image model (Stable Diffusion) -- to generate a large dataset of image editing examples. Our conditional diffusion model, InstructPix2Pix, is trained on our generated data, and generalizes to real images and user-written instructions at inference time. Since it performs edits in the forward pass and does not require per example fine-tuning or inversion, our model edits images quickly, in a matter of seconds. We show compelling editing results for a diverse collection of input images and written instructions.* | ||
|
||
Resources: | ||
|
||
* [Project Page](https://www.timothybrooks.com/instruct-pix2pix). | ||
* [Paper](https://arxiv.org/abs/2211.09800). | ||
* [Original Code](https://github.com/timothybrooks/instruct-pix2pix). | ||
* [Demo](https://huggingface.co/spaces/timbrooks/instruct-pix2pix). | ||
|
||
|
||
## Available Pipelines: | ||
|
||
| Pipeline | Tasks | Demo | ||
|---|---|:---:| | ||
| [StableDiffusionInstructPix2PixPipeline](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_instruct_pix2pix.py) | *Text-Based Image Editing* | [🤗 Space](https://huggingface.co/spaces/timbrooks/instruct-pix2pix) | | ||
|
||
<!-- TODO: add Colab --> | ||
|
||
## Usage example | ||
|
||
```python | ||
import PIL | ||
import requests | ||
import torch | ||
from diffusers import StableDiffusionInstructPix2PixPipeline | ||
|
||
model_id = "timbrooks/instruct-pix2pix" | ||
pipe = StableDiffusionInstructPix2PixPipeline.from_pretrained( | ||
model_id, torch_dtype=torch.float16, revision="fp16", safety_checker=None | ||
).to("cuda") | ||
|
||
url = "https://raw.githubusercontent.com/timothybrooks/instruct-pix2pix/main/imgs/example.jpg" | ||
|
||
|
||
def download_image(url): | ||
image = PIL.Image.open(requests.get(url, stream=True).raw) | ||
image = PIL.ImageOps.exif_transpose(image) | ||
image = image.convert("RGB") | ||
return image | ||
|
||
|
||
image = download_image(url) | ||
|
||
prompt = "turn him into a cyborg" | ||
images = pipe(prompt, image=image, num_inference_steps=10, guidance_scale=1.1, image_guidance_scale=1).images | ||
images[0].save("david_cyborg.png") | ||
``` | ||
|
||
## StableDiffusionInstructPix2PixPipeline | ||
[[autodoc]] StableDiffusionInstructPix2PixPipeline | ||
- __call__ | ||
- all |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.