Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FLUX XLabs IP-Adapter Support #7157

Merged
merged 32 commits into from
Oct 22, 2024
Merged

Conversation

RyanJDick
Copy link
Collaborator

@RyanJDick RyanJDick commented Oct 21, 2024

Summary

This PR adds support for the XLabs IP-Adapter (https://huggingface.co/XLabs-AI/flux-ip-adapter) in workflows. Linear UI integration is coming in a follow-up PR. The XLabs IP-Adapter can be installed in the Starter Models tab.

Usage tips:

  • Use a cfg_scale value of 2.0 to 4.0
  • Start with an IP-Adatper weight of ~0.6 and adjust from there.
  • Set cfg_scale_start_step = 1
  • Set cfg_scale_end_step to roughly the halfway point (it's unnecessary to apply CFG to all steps, and this will improve processing time).

Sample workflow:
image

Result:
image

Related Issues / Discussions

Prerequisite: #7152

Remaining TODO:

  • Update default workflows.

QA Instructions

  • Test basic happy path
  • Test with multiple IP-Adapters (it runs, but results aren't great)
  • Test with multiple images to a single IP-Adapter (this is not supported for now)
  • Test automatic runtime installation of CLIP-L, CLIP-H, and CLIP-G image encoder models if they are not already installed.
  • Test starter model installation of the XLabs FLUX IP-Adapter
  • Test SD and SDXL IP-Adapters for regression.
  • Check peak memory utilization.

Merge Plan

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • Documentation added / updated (if applicable)

…-Adapter. Need to revisit how to clean this all up long term.
@github-actions github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files python-tests PRs that change python tests labels Oct 21, 2024
@github-actions github-actions bot added the frontend PRs that change frontend files label Oct 22, 2024
Base automatically changed from ryan/flux-cfg to main October 22, 2024 21:09
RyanJDick and others added 5 commits October 22, 2024 21:17
This masked an issue w/ the CLIP Vision model. Issue is now handled in reducer/graph builder.
FLUX IP Adapter only works with RGB. Did the same for non-FLUX to be safe & consistent, though I don't think it's strictly necessary.
@psychedelicious psychedelicious merged commit aff6ad0 into main Oct 22, 2024
14 checks passed
@psychedelicious psychedelicious deleted the ryan/flux-ip-adapter-cfg-2 branch October 22, 2024 22:57
@RyanJDick RyanJDick mentioned this pull request Nov 18, 2024
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backend PRs that change backend files frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants