Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enable specifying schedulers for text-to-image and image-to-image #132

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

ad-astra-video
Copy link
Collaborator

@ad-astra-video ad-astra-video commented Jul 28, 2024

This PR adds ability to set schedulers using a json string in the request for t2i and i2i pipeline. name of the scheduler can be a preset or a class name for the scheduler from diffusers.

There are presets to make setting the main schedulers easier from https://huggingface.co/docs/diffusers/en/api/schedulers/overview.

Scheduler is created by mapping the preset name to the diffusers scheduler class using importlib so the schedulers are not imported until needed.
If a scheduler fails to be added the pipeline raises an error. If no scheduler is specified the default scheduler is applied.

performance testing
loading no scheduler (reassigns default scheduler)
2024-07-28 01:12:00,677 - app.pipelines.text_to_image - INFO - setting pipeline scheduler to:
2024-07-28 01:12:00,677 - app.pipelines.text_to_image - INFO - loading scheduler took 0.0002524852752685547s

loading DPM++ 2M scheduler from preset on RealVisXL_V4.0 model
2024-07-28 01:13:44,851 - app.pipelines.text_to_image - INFO - setting pipeline scheduler to: {"name":"DPM++ 2M","args":{}}
2024-07-28 01:13:44,858 - app.pipelines.text_to_image - INFO - loading scheduler took 0.00660252571105957s

loading DPM2 scheduler from preset on RealVisXL_V4.0 model
2024-07-28 01:10:39,576 - app.pipelines.text_to_image - INFO - setting pipeline scheduler to: {"name":"DPM2","args":{}}
2024-07-28 01:10:39,817 - app.pipelines.text_to_image - INFO - loading scheduler took 0.24109697341918945s

loading Euler flow match for SD3, setting shift arg.
2024-07-28 01:42:36,853 - app.pipelines.text_to_image - INFO - setting pipeline scheduler to: {"name":"Euler flow match","args":{"shift":"3"}}
2024-07-28 01:42:36,856 - app.pipelines.text_to_image - INFO - loading scheduler took 0.002814054489135742s

format of json string in http request:
{ "name": "Euler", "args": {} }

@ad-astra-video ad-astra-video changed the title enable specifying schedulers for text-to-image enable specifying schedulers for text-to-image and image-to-image Sep 4, 2024
@ad-astra-video
Copy link
Collaborator Author

Updated this to 0.2.0 release

@ad-astra-video
Copy link
Collaborator Author

ad-astra-video commented Nov 5, 2024

@rickstaa I rebased this and tested. I used RealVisXL_4.0 to test setting scheduler and observed small changes in image holding the seed constant

Example requests using /docs endpoint:

#see attached example output
{
  "model_id": "",
  "loras": "",
  "prompt": "A pair of clear hands looking through a transparent glass Christmas ball , hyper-realistic, minimalist, futuristic background with cute Christmas decorations like Santa Claus, a snowman, and snowflakes, 8k",
  "height": 1024,
  "width": 1024,
  "guidance_scale": 7.5,
  "negative_prompt": "face asymmetry, eyes asymmetry, deformed eyes, open mouth",
  "safety_check": true,
  "seed": 0,
  "num_inference_steps": 28,
  "num_images_per_prompt": 1,
  "scheduler": ""
}
#see attached example output
{
  "model_id": "",
  "loras": "",
  "prompt": "A pair of clear hands looking through a transparent glass Christmas ball , hyper-realistic, minimalist, futuristic background with cute Christmas decorations like Santa Claus, a snowman, and snowflakes, 8k",
  "height": 1024,
  "width": 1024,
  "guidance_scale": 7.5,
  "negative_prompt": "face asymmetry, eyes asymmetry, deformed eyes, open mouth",
  "safety_check": true,
  "seed": 0,
  "num_inference_steps": 28,
  "num_images_per_prompt": 1,
  "scheduler": "{\"name\":\"DPM++ 2M SDE Karras\", \"args\":{}}"
}

DEFAULT
default
DPM++ SDE KARRAS
DPM++_SDE_KARRAS

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant