VRAM does not clear after video generation so goes OOM after generating 2-3 videos

[sdes21]

Hey, I’m using the image to video model. I can generate the lowest resolution videos with a 24GB vram but with each video generation, the VRAM resources keep going up till it goes out of memory.

Is there a way to clear the memory after a video is generated?

[yunkchen]

Could you show your whole log, which contains each video generation.

[sdes21]

Ok will try again and show the logs. I’m using the L4 gpu from Google Colab Pro.

[sdes21]

A matching Triton is not available, some optimizations will not be enabled
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/xformers/init.py", line 57, in _is_triton_available
import triton # noqa
ModuleNotFoundError: No module named 'triton'
2024-11-20 02:01:37.278124: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable TF_ENABLE_ONEDNN_OPTS=0.
2024-11-20 02:01:37.732939: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-11-20 02:01:37.943587: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-11-20 02:01:38.001004: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-11-20 02:01:38.339352: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-20 02:01:40.220905: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
/usr/local/lib/python3.10/dist-packages/gradio/components/dropdown.py:188: UserWarning: The value passed into gr.Dropdown() is not in the list of choices. Please update the list of choices to include: none or set allow_custom_value=True.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/gradio/utils.py:1002: UserWarning: Expected 2 arguments for function <function ui..upload_generation_method at 0x79620c475e10>, received 1.
warnings.warn(
/usr/local/lib/python3.10/dist-packages/gradio/utils.py:1006: UserWarning: Expected at least 2 arguments for function <function ui..upload_generation_method at 0x79620c475e10>, received 1.
warnings.warn(
Running on local URL: http://0.0.0.0:7860/
Running on public URL: https://8c02f0e679cdfaaa3a.gradio.live/

This share link expires in 72 hours. For free permanent hosting and GPU upgrades, run gradio deploy from Terminal to deploy to Spaces (https://huggingface.co/spaces)
Update diffusion transformer

missing keys: 0;

unexpected keys: 0;

[] []
loaded 3D transformer's pretrained weights from /content/drive/MyDrive/EasyAnimate/models/Diffusion_Transformer/EasyAnimateV5-12b-zh-InP/transformer ...

missing keys: 0;

unexpected keys: 0;

[]

All Parameters: 11797.69248 M

attn1 Parameters: 1812.54144 M

Loading checkpoint shards: 100% 2/2 [02:17<00:00, 68.58s/it]
The config attributes {'snr_shift_scale': 1.0} were passed to EulerDiscreteScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
Loading pipeline components...: 100% 7/7 [00:00<00:00, 234.55it/s]
You have disabled the safety checker for <class 'easyanimate.pipeline.pipeline_easyanimate_multi_text_encoder_inpaint.EasyAnimatePipeline_Multi_Text_Encoder_Inpaint'> by passing safety_checker=None. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at huggingface/diffusers#254 .
Update diffusion transformer done
The config attributes {'snr_shift_scale': 1.0} were passed to EulerDiscreteScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
We strongly recommend passing in an attention_mask since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked.
You may ignore this warning if your pad_token_id (0) is identical to the bos_token_id (0), eos_token_id (2), or the sep_token_id (None), and your input is not padded.
/content/drive/MyDrive/EasyAnimate/easyanimate/pipeline/pipeline_easyanimate_multi_text_encoder_inpaint.py:1053: FutureWarning: Accessing config attribute enable_clip_in_inpaint directly via 'EasyAnimateTransformer3DModel' object attribute is deprecated. Please access 'enable_clip_in_inpaint' over 'EasyAnimateTransformer3DModel's config object instead, e.g. 'unet.config.enable_clip_in_inpaint'.
elif clip_image is None and num_channels_transformer != num_channels_latents and self.transformer.enable_clip_in_inpaint:
100% 30/30 [05:03<00:00, 10.11s/it]
The config attributes {'snr_shift_scale': 1.0} were passed to EulerDiscreteScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
/content/drive/MyDrive/EasyAnimate/easyanimate/pipeline/pipeline_easyanimate_multi_text_encoder_inpaint.py:1053: FutureWarning: Accessing config attribute enable_clip_in_inpaint directly via 'EasyAnimateTransformer3DModel' object attribute is deprecated. Please access 'enable_clip_in_inpaint' over 'EasyAnimateTransformer3DModel's config object instead, e.g. 'unet.config.enable_clip_in_inpaint'.
elif clip_image is None and num_channels_transformer != num_channels_latents and self.transformer.enable_clip_in_inpaint:
100% 30/30 [05:04<00:00, 10.17s/it]
The config attributes {'snr_shift_scale': 1.0} were passed to EulerDiscreteScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
/content/drive/MyDrive/EasyAnimate/easyanimate/pipeline/pipeline_easyanimate_multi_text_encoder_inpaint.py:1053: FutureWarning: Accessing config attribute enable_clip_in_inpaint directly via 'EasyAnimateTransformer3DModel' object attribute is deprecated. Please access 'enable_clip_in_inpaint' over 'EasyAnimateTransformer3DModel's config object instead, e.g. 'unet.config.enable_clip_in_inpaint'.
elif clip_image is None and num_channels_transformer != num_channels_latents and self.transformer.enable_clip_in_inpaint:
9% 3/35 [00:37<06:37, 12.43s/it]

After this step the UI shows and error and goes OOM.

[bubbliiiing]

It should be solved in #158.

[sdes21]

New model runs fine! Thanks. Closing issue.