support_deepcache_svd_pipeline #514
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
clackhan
commented
Jan 11, 2024
•
clackhan
commented
- A100
| PyTorch | Torch-Compile | Oneflow | OneDiff-DeepCache | |
|---|---|---|---|---|
| 576 x 1024, 25 frames, decode chunk size 5 | 50.930s | 43.376s | 31.933s | 18.486s |
onediff_diffusers_extensions/diffusers_extensions/deep_cache/__init__.py
Outdated
Show resolved
Hide resolved
onediff_diffusers_extensions/diffusers_extensions/deep_cache/models/pipeline_utils.py
Outdated
Show resolved
Hide resolved
strint
reviewed
Jan 11, 2024
| @@ -0,0 +1,257 @@ | |||
| # Run with ONEFLOW_RUN_GRAPH_BY_VM=1 to get faster | |||
| MODEL = 'stabilityai/stable-video-diffusion-img2vid-xt' | |||
There was a problem hiding this comment.
这个例子我们放到适配 diffusers 的文件夹（https://github.com/siliconflow/onediff/tree/main/onediff_diffusers_extensions）下，建一个 video 文件夹，然后添一个 video 的readme ，介绍下使用方式和性能。
There was a problem hiding this comment.
example 后面拿来测试 compile 。和 diffusers 适配相关的后面都放 https://github.com/siliconflow/onediff/tree/main/onediff_diffusers_extensions
|
I'm testing this pipeline although the inference seems to work, it fails before returning the output (so I suspect it might be VAE related?) when using the OneDiff's Enterprise edition (VM Mode=1) Logs |
|
If I stop compiling Looking into underlying tensors ( |
You can test reduce the |
This script has already limited the fp16 range to avoid overflow by setting |
|
Thanks for the hint, setting |
|
isidentical
reviewed
Jan 12, 2024
| def __init__( | ||
| self, | ||
| vae: AutoencoderKLTemporalDecoder, | ||
| image_encoder: CLIPVisionModelWithProjection, |
There was a problem hiding this comment.
Seems like this component fails to get saved, FYI:
2024-01-12 01:11:46.164 [stderr ] File "/root/.cache/isolate/virtualenv/5371bcd257691477725a108bc5f62c9ccb52f53db92ecf3cddd4ceb4111f24d0/lib/python3.11/site-packages/onediff/infer_compiler/with_oneflow_compile.py", line 406, in save_graph
2024-01-12 01:11:46.164 [stderr ] flow.save(state_dict, file_path)
2024-01-12 01:11:46.164 [stderr ] File "/root/.cache/isolate/virtualenv/5371bcd257691477725a108bc5f62c9ccb52f53db92ecf3cddd4ceb4111f24d0/lib/python3.11/site-packages/oneflow/framework/check_point_v2.py", line 720, in save
2024-01-12 01:11:46.164 [stderr ] pickled_bytes = pickle.dumps(obj)
2024-01-12 01:11:46.164 [stderr ] ^^^^^^^^^^^^^^^^^
2024-01-12 01:11:46.164 [stderr ] _pickle.PicklingError: Can't pickle <class 'transformers.models.clip.modeling_clip.CLIPVisionModelOutput'>: it's not the same object as transformers.models.clip.modeling_clip.CLIPVisionModelOutput
There was a problem hiding this comment.
Compilation of image_encoder can be skipped, we have not tested its graph save.
…iliconflow/onediff into support_deepcache_svd_pipeline
|
Another bug occurs when I try two different image resolutions back to back, but not sure if it is an deepcache problem or a generic SVD problem (ONEFLOW_RUN_GRAPH_BY_VM=1) (and this takes a really long time to complete) |
The method for dynamic switch resolution in SVD is currently under development. |
|
we have been playing with this for a while and it seems to work really well 💯 |
have you tried compile fast_unet? we forgot to do this in the previous example. compiling fast_unet will make the pipeline faster. |
|
yep! we are compiling unet, fast_unet and vae.decoder (the rest seem to be problematic for save/load cases and didn't provide enough speed up to warrant investigating why) |
strint
approved these changes
Jan 19, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.