Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FLUX image-to-image and inpainting #6798

Merged
merged 20 commits into from
Sep 2, 2024
Merged

Add FLUX image-to-image and inpainting #6798

merged 20 commits into from
Sep 2, 2024

Conversation

RyanJDick
Copy link
Collaborator

@RyanJDick RyanJDick commented Aug 30, 2024

Summary

This PR adds support for Image-to-Image and inpainting workflows with the FLUX model.

Full changelog:

  • Split out FLUX VAE Encode and FLUX VAE Decode nodes
  • Renamed FLUX Text-to-Image node to FLUX Denoise (since it now supports image-to-image too). This is a workflow-breaking change.
  • Added support for FLUX image-to-image via the Latents param on the FLUX denoising node.
  • Added support for FLUX masked inpainting via the Denoise Mask param on the FLUX denoising node.
  • Added "Denoise Start" and "Denoise End" params to the "FLUX Denoise" node.
  • Updated the "FLUX Text to Image" default workflow.
  • Added a "FLUX Image to Image" default workflow.

Example

FLUX inpainting workflow
image

Input image
image

Mask
image

Output image
image

Callouts for reviewers:

  • I renamed FLUXTextToImageInvocation -> FLUXDenoisingInvocation. This is, of course, a breaking change. It feels like the right move and now is the right time to do it. Any objection?
  • I added new FLUX VAE Encode and FLUX VAE Decode nodes. Alternatively, I could have tried to match these names to the corresponding SD nodes (e.g. FLUX Image to Latents, FLUX Latents to Image). Personally, I prefer the current names, but want to hear other opinions.

Usage notes:

  • With the default dev timestep scheduler, the image structure is largely determined in the first ~3 steps. A consequence of this is that the denoise_start parameter provides limited 'granularity' of control. This will likely be improved in the future as we add more scheduler options. In the meantime, you will likely want to use small values for denoise_start (e.g. 0.03) to start denoising on step ~1-4 out of ~30.
  • Currently, there is no 'noise' parameter on the FLUX Denoise node, so the denoise_end parameter has limited utility. This will be added in the future.

QA Instructions

Test the following workflows:

  • Vanilla FLUX text-to-image behaviour is unchanged
  • Image-to-image with FLUX dev, no mask
  • Image-to-image with FLUX dev, with mask
  • Image-to-image with FLUX schnell, no mask (smoke test, not expected to work well)

Merge Plan

No special instructions.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)

@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files services PRs that change app services frontend PRs that change frontend files python-tests PRs that change python tests labels Aug 30, 2024
Copy link
Member

@hipsterusername hipsterusername left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only consideration here is inconsistency between name of VAE between flux and SD nodes. I think the simpler name is probably the best top level name, but it should be discoverable under encode

Copy link
Member

@hipsterusername hipsterusername left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will approve in case the desire is to address this later

Copy link
Collaborator

@brandonrising brandonrising left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than that comment lgtm! It's somewhat tangential to the work here so it could be updated in a separate PR

@RyanJDick
Copy link
Collaborator Author

The only consideration here is inconsistency between name of VAE between flux and SD nodes. I think the simpler name is probably the best top level name, but it should be discoverable under encode

Renamed 'FLUX VAE Encode'/'FLUX VAE Decode' to 'FLUX Image to Latents'/'FLUX Latents to Image' in b43da46

@RyanJDick RyanJDick merged commit 24065ec into main Sep 2, 2024
14 checks passed
@RyanJDick RyanJDick deleted the ryan/flux-split-vae branch September 2, 2024 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend PRs that change backend files frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests services PRs that change app services
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants