Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add controlnet module #79

Merged
merged 4 commits into from
Jan 18, 2024
Merged

Add controlnet module #79

merged 4 commits into from
Jan 18, 2024

Conversation

xiaoweiw-nv
Copy link
Collaborator

The controlnet module uses a TensorRT-optimized controlnet implementation.

Currently, the prompt is read from a file by the text_module and passed to the controlnet_module, the graph looks like:

decode --> controlnet_module ---> encode
      |            ^
      |            |
      -------> text_module

To sync the prompt with the image, the decoder output is also sent to the text_module, once the decoder outputs EOF, the text_module will also generate EOF. However, in the controlnet_module, the propmt_queue is always empty.

To run the pipeline:

python test_controlnet.py

The input image in case you need it:
bird

@xiaoweiw-nv xiaoweiw-nv requested a review from sfeiwong December 12, 2023 03:44
a show case to make the multistream of frame and
prompt text work together in the demo
@xiaoweiw-nv
Copy link
Collaborator Author

Thanks @sfeiwong ! I will merge your modification into the pipeline and push another commit that contains everything.

@xiaoweiw-nv
Copy link
Collaborator Author

@sfeiwong New code pushed, empty queue problem solved, please review.

@sfeiwong sfeiwong merged commit 651252f into master Jan 18, 2024
1 check passed
@sfeiwong sfeiwong deleted the controlnet_module branch January 18, 2024 07:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants