Add sd3 pipeline and models #96
Conversation
There was a problem hiding this comment.
Looks great to me, thanks a lot for the great contribution @ZachNagengast 🙌 Curious about your thoughts on a few questions.
| isa = XCRemoteSwiftPackageReference; | ||
| repositoryURL = "https://github.com/apple/ml-stable-diffusion"; | ||
| repositoryURL = "https://github.com/argmaxinc/ml-stable-diffusion.git"; |
There was a problem hiding this comment.
Any feedback on apple/ml-stable-diffusion#329?
There was a problem hiding this comment.
Apple appears to still be reviewing but I'll be happy to update when that is merged - this is just for testing and review purposes here.
Diffusion/Common/ModelInfo.swift
Outdated
| static let sd3 = ModelInfo( | ||
| modelId: "argmaxinc/coreml-stable-diffusion-3-medium", | ||
| modelVersion: "SD3 medium (512, macOS)", | ||
| supportsEncoder: false, | ||
| quantized: false, | ||
| isSD3: true | ||
| ) |
There was a problem hiding this comment.
This model does not have a split einsum package; however, some Macs default to using the ANE with split einsum attention. What do you think we could do to override compute unit selection in this case, and force it to always be GPU?
There was a problem hiding this comment.
I took the inverse approach and only enabled it for .all and .gpu https://github.com/huggingface/swift-coreml-diffusers/pull/96/files#diff-eb8807a52606483b97fb37ae59fb50afcd9e0504aa9381d40727807badf31233R111. If user selects ANE it will show a pipeline error because the zip wont be found. We could make that a little cleaner though with a descriptive error, if you'd prefer.
There was a problem hiding this comment.
A better error sounds good if we can. I think it's ok to stop there unless the first-time launch for new users defaults to split einsum + sd3 on some Macs.
There was a problem hiding this comment.
I'm adding this to the selector so that folks will know when it is available or not. This way it will not fail and delete the model cache, since it will never attempt to download it in the first place.
Not supported:
Supported:
It also will auto switch back to GPU if selecting a model where the previous compute unit was ANE.
There was a problem hiding this comment.
Is there a way to convert the SDXL and SD3 model into Split_Einsum? I have a MBP M2 and an iPad Pro M4 both with 16GB Ram and on both the ANE is a lot faster than CPU/GPU! For now I am using the 768x768 Split_Einsum SDXL model for iOS and I am wondering why there are no Split_Einsum versions of those bigger models?
|
Worth noting that SD3 is having some issue on macOS 15 with memory & speed, will try to drill down on this issue before the the OS comes out. |
…t-coreml-diffusers into sd3-app-support
There was a problem hiding this comment.
Very cool, @ZachNagengast, thanks a lot for taking care of all these details!
This PR supports SD3 models via apple/ml-stable-diffusion#329
It includes two new models exported via DiffusionKit
Note:
It also includes UserDefaults memory for both prompt and generation config.
More quantizations and speedups to come, so stay tuned!