Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FLUX CFG support #7152

Merged
merged 4 commits into from
Oct 22, 2024
Merged

FLUX CFG support #7152

merged 4 commits into from
Oct 22, 2024

Conversation

RyanJDick
Copy link
Collaborator

@RyanJDick RyanJDick commented Oct 18, 2024

Summary

Add support for Classifier-Free Guidance with FLUX.

  • Using CFG doubles the time for the denoising process. Running both the positive and negative conditioning in a single batch is left for future work, because most users are already VRAM-constrained (this would probably be faster at the cost of higher peak VRAM).
  • Negative text conditioning is optional and only required if cfg_scale != 1.0
  • CFG is skipped if cfg_scale == 1.0 (i.e. no compute overhead in this case)
  • cfg_scale_start_step and cfg_scale_end_step can be used to easily control the range of steps that CFG is applied for.
  • CFG is a prerequisite for IP-Adapter support.

Example

Positive Caption: Professional photography of a luxury hotel in the Nevada desert
CFG: 1.0
image

Positive Caption: Professional photography of a luxury hotel in the Nevada desert
Negative Caption: Swimming pool
CFG: 2.0
Same seed
image

QA Instructions

  • Test interactions with ControlNet
  • Verify that peak RAM/VRAM utilization has not increased significantly
  • Test that CFG is skipped when cfg_scale == 1.0
  • Test that negative text conditioning can be omitted when cfg_scale == 1.0
  • Test that a clear error message is returned when negative text conditioning is omitted when cfg_scale != 1.0
  • Test that the negative text prompt gets applied when cfg_scale >1.0
  • Test that a collection of cfg_scale values can be provided for per-step control.
  • Test that cfg_scale_start_step and cfg_scale_end_step control the range of steps that CFG is applied

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)

@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files labels Oct 18, 2024
@psychedelicious
Copy link
Collaborator

How does CFG interact with guidance? Should both of these be exposed separately in the linear UI?

@github-actions github-actions bot added the python-tests PRs that change python tests label Oct 21, 2024
@RyanJDick RyanJDick mentioned this pull request Oct 21, 2024
13 tasks
@RyanJDick RyanJDick marked this pull request as ready for review October 22, 2024 20:45
@RyanJDick RyanJDick enabled auto-merge October 22, 2024 20:51
@RyanJDick RyanJDick merged commit c466d50 into main Oct 22, 2024
14 checks passed
@RyanJDick RyanJDick deleted the ryan/flux-cfg branch October 22, 2024 21:09
psychedelicious added a commit that referenced this pull request Oct 22, 2024
## Summary

This PR adds support for the XLabs IP-Adapter
(https://huggingface.co/XLabs-AI/flux-ip-adapter) in workflows. Linear
UI integration is coming in a follow-up PR. The XLabs IP-Adapter can be
installed in the Starter Models tab.

Usage tips:

- Use a `cfg_scale` value of 2.0 to 4.0
- Start with an IP-Adatper weight of ~0.6 and adjust from there.
- Set `cfg_scale_start_step = 1`
- Set `cfg_scale_end_step` to roughly the halfway point (it's
unnecessary to apply CFG to all steps, and this will improve processing
time).

Sample workflow:
<img width="976" alt="image"
src="https://github.com/user-attachments/assets/4627b459-7e5a-4703-80e7-f7575c5fce19">

Result:

![image](https://github.com/user-attachments/assets/220b6a4c-69c6-447f-8df6-8aa6a56f3b3f)

## Related Issues / Discussions

Prerequisite: #7152

## Remaining TODO:

- [ ] Update default workflows.

## QA Instructions

- [x] Test basic happy path
- [x] Test with multiple IP-Adapters (it runs, but results aren't great)
- [ ] ~Test with multiple images to a single IP-Adapter~ (this is not
supported for now)
- [ ] Test automatic runtime installation of CLIP-L, CLIP-H, and CLIP-G
image encoder models if they are not already installed.
- [ ] Test starter model installation of the XLabs FLUX IP-Adapter
- [ ] Test SD and SDXL IP-Adapters for regression.
- [ ] Check peak memory utilization.

## Merge Plan

- [ ] Merge #7152 
- [ ] Change target branch to main

## Checklist

- [x] _The PR has a short but descriptive title, suitable for a
changelog_
- [x] _Tests added / updated (if applicable)_
- [ ] _Documentation added / updated (if applicable)_
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend PRs that change backend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants