Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support to pass image embeddings to the WAN I2V pipeline. #11175

Merged
merged 9 commits into from
Apr 8, 2025

Conversation

goiri
Copy link
Contributor

@goiri goiri commented Mar 30, 2025

What does this PR do?

Allows computing the image embeddings externally and use them.
It also fixes the function documentation.

Before submitting

Who can review?

Pipeline changes: @yiyixuxu and @asomoza

It allows computing the image embeddings externally and use them.
@goiri goiri changed the title Add support to pass image embeddings to the pipeline. Add support to pass image embeddings to the WAN I2V pipeline. Mar 30, 2025
@hlky
Copy link
Member

hlky commented Mar 31, 2025

@bot /style

Copy link
Contributor

Style fixes have been applied. View the workflow run here.

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Co-authored-by: hlky <hlky@hlky.ac>
@yiyixuxu
Copy link
Collaborator

yiyixuxu commented Apr 2, 2025

ohh test failures aer relevant, can we look into them?

@goiri
Copy link
Contributor Author

goiri commented Apr 2, 2025

ohh test failures aer relevant, can we look into them?

Fixing the following errors:

FAILED tests/pipelines/wan/test_wan_image_to_video.py::WanImageToVideoPipelineFastTests::test_callback_cfg - ValueError: Cannot forward both `image`: <PIL.Image.Image image mode=RGB size=16x16 at 0x7FB4AA18F9A0> and `image_embeds`: ['latents', 'prompt_embeds', 'negative_prompt_embeds']. Please make sure to only forward one of the two.
FAILED tests/pipelines/wan/test_wan_image_to_video.py::WanImageToVideoPipelineFastTests::test_callback_inputs - ValueError: Cannot forward both `image`: <PIL.Image.Image image mode=RGB size=16x16 at 0x7FB4DA2F4BE0> and `image_embeds`: ['latents']. Please make sure to only forward one of the two.
FAILED tests/pipelines/wan/test_wan_image_to_video.py::WanImageToVideoPipelineFastTests::test_cfg - ValueError: Cannot forward both `image`: <PIL.Image.Image image mode=RGB size=16x16 at 0x7FB4AAC4A3E0> and `image_embeds`: ['latents']. Please make sure to only forward one of the two.
FAILED tests/pipelines/wan/test_wan_image_to_video.py::WanImageToVideoPipelineFastTests::test_dict_tuple_outputs_equivalent - ValueError: Cannot forward both `image`: <PIL.Image.Image image mode=RGB size=16x16 at 0x7FB4AA17DC00> and `image_embeds`: ['latents']. Please make sure to only forward one of the two.
FAILED tests/pipelines/wan/test_wan_image_to_video.py::WanImageToVideoPipelineFastTests::test_encode_prompt_works_in_isolation - ValueError: Cannot forward both `image`: <PIL.Image.Image image mode=RGB size=16x16 at 0x7FB4AA0D5330> and `image_embeds`: ['latents']. Please make sure to only forward one of the two.
FAILED tests/pipelines/wan/test_wan_image_to_video.py::WanImageToVideoPipelineFastTests::test_inference - ValueError: Cannot forward both `image`: <PIL.Image.Image image mode=RGB size=16x16 at 0x7FB4AA7A2D40> and `image_embeds`: ['latents']. Please make sure to only forward one of the two.
FAILED tests/pipelines/wan/test_wan_image_to_video.py::WanImageToVideoPipelineFastTests::test_inference_batch_consistent - ValueError: Cannot forward both `image`: <PIL.Image.Image image mode=RGB size=16x16 at 0x7FB490136CE0> and `image_embeds`: ['latents']. Please make sure to only forward one of the two.
FAILED tests/pipelines/wan/test_wan_image_to_video.py::WanImageToVideoPipelineFastTests::test_save_load_local - ValueError: Cannot forward both `image`: <PIL.Image.Image image mode=RGB size=16x16 at 0x7FB4DA2F5150> and `image_embeds`: ['latents']. Please make sure to only forward one of the two.
FAILED tests/pipelines/wan/test_wan_image_to_video.py::WanImageToVideoPipelineFastTests::test_save_load_optional_components - ValueError: Cannot forward both `image`: <PIL.Image.Image image mode=RGB size=16x16 at 0x7FB4AAA19150> and `image_embeds`: ['latents']. Please make sure to only forward one of the two.

goiri added 2 commits April 2, 2025 13:32
The arguments are passed unnamed and callback gets passed as image_embeds.
@yiyixuxu yiyixuxu merged commit 841504b into huggingface:main Apr 8, 2025
11 of 12 checks passed
@yiyixuxu
Copy link
Collaborator

yiyixuxu commented Apr 8, 2025

merged, thanks for the PR!

@goiri
Copy link
Contributor Author

goiri commented Apr 8, 2025

Thank you very much for the reviews!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants