Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support SVD dynamic shape[feat] #564

Merged
merged 3 commits into from
Jan 26, 2024
Merged

Support SVD dynamic shape[feat] #564

merged 3 commits into from
Jan 26, 2024

Conversation

lixiang007666
Copy link
Contributor

@lixiang007666 lixiang007666 commented Jan 23, 2024

This PR is done:

issue: https://github.com/siliconflow/sd-team/issues/240

  • SVD acceleration supports dynamic resolution switching.
  • Add example.

Run:

export ONEFLOW_RUN_GRAPH_BY_VM="1"
python3 examples/image_to_video.py --variant fp16 --output-video vm.mp4 --decode-chunk-size 4 --run_multiple_resolutions true

Output:

Loading pipeline components...: 100%|█████████████████████████████████████████████████████| 5/5 [00:00<00:00,  7.50it/s]
warmup:
100%|███████████████████████████████████████████████████████████████████████████████████| 25/25 [02:29<00:00,  5.99s/it]
infer:
100%|███████████████████████████████████████████████████████████████████████████████████| 25/25 [00:28<00:00,  1.15s/it]
test dynamic resolution switch:
100%|███████████████████████████████████████████████████████████████████████████████████| 25/25 [00:28<00:00,  1.15s/it]
image

@@ -11,6 +11,9 @@
from diffusers.models.transformer_2d import Transformer2DModel
if diffusers_version >= version.parse("0.24.00"):
from diffusers.models.resnet import SpatioTemporalResBlock
from diffusers.models.transformer_temporal import TransformerSpatioTemporalModel
from diffusers.models.attention import TemporalBasicTransformerBlock
from diffusers.models.unet_spatio_temporal_condition import UNetSpatioTemporalConditionModel

if diffusers_version >= version.parse("0.25.00"):
from diffusers.models.autoencoders.autoencoder_kl_temporal_decoder import TemporalDecoder
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

需要试下对 diffusers 0.25 的兼容性

Copy link
Contributor Author

@lixiang007666 lixiang007666 Jan 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

在 0.24 和 0.25 上测试过了。


hidden_states = self.norm(hidden_states)
inner_dim = hidden_states.shape[1]
# hidden_states = hidden_states.permute(0, 2, 3, 1).reshape(batch_frames, height * width, inner_dim)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

都默认加个注释以方便后面维护: Rewrite for onediff dynamic shape

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@lixiang007666 lixiang007666 requested a review from hjchen2 January 23, 2024 11:06
@strint
Copy link
Collaborator

strint commented Jan 23, 2024

另外现在的 ci 是还没跑 svd 的测试,其中一个主要原因是 CI 机器的显存太小?

@lixiang007666
Copy link
Contributor Author

另外现在的 ci 是还没跑 svd 的测试,其中一个主要原因是 CI 机器的显存太小?

是的

@hjchen2 hjchen2 merged commit 07184c5 into main Jan 26, 2024
5 checks passed
@hjchen2 hjchen2 deleted the support_svd_dynamic_shape branch January 26, 2024 14:56
@yh8899
Copy link

yh8899 commented Feb 21, 2024

@lixiang007666 我看这里面有用到unflatten这个算子,我在oneflow里面没有找到这个算子的实现?请问你用的是哪个版本的one flow呢?

@strint
Copy link
Collaborator

strint commented Feb 21, 2024

@lixiang007666 我看这里面有用到unflatten这个算子,我在oneflow里面没有找到这个算子的实现?请问你用的是哪个版本的one flow呢?

这个算子是在一个 oneflow 内部扩展版本中实现的,还没开放。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants