Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ControlNet Annotators to ONNX #52

Open
jdp8 opened this issue Dec 31, 2023 · 0 comments
Open

ControlNet Annotators to ONNX #52

jdp8 opened this issue Dec 31, 2023 · 0 comments

Comments

@jdp8
Copy link

jdp8 commented Dec 31, 2023

At the moment I am able to use Canny Edge Detection, Pose Estimation and Semantic Segmentation (all in JavaScript thanks to OpenCV.js) in diffusers.js for the annotation (pre-processing) of images that are used as input for various ControlNets. These are the only ones that I've managed to implement so far but there are more Annotator models such as Scribble, HED, M-LSD, etc that I have not found a JavaScript implementation for.

It would be useful to have these models in ONNX format so they can be called the same way as an UNET, VAE or other model. A script that can convert the Annotator models to ONNX format or the models directly in ONNX format so they can be used in diffusers.js would be an ideal solution if possible.

I believe the original implementation of the Annotator models is controlnet_aux.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant