[BUG]Memory Issue when generating images for the second time #602
Comments
|
give full logs from start |
|
I do not know if the issues are related in a sense that it's a memory leak, but my second generation of pictures is significantly slower as well. Here is a full log of first and second generation back to back: For good measure I've done it a second time and just reopened the run.bat and started with a different prompt: GPU Mem usage after this and closing fooocus: 3,8/16 GB This is reproducible for me. |
Here is the full log. To create a public link, set |
|
see if fixed in 2.1.38 |
Got fixed in 2.1.49 |
When I generate images first time with one image prompt, everything works fine.
However, at the second generation, the GPU memory run out.
Here is the error
`Preparation time: 19.46 seconds
loading new
ERROR diffusion_model.output_blocks.0.1.transformer_blocks.4.ff.net.0.proj.weight CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0; 22.41 GiB total capacity; 21.52 GiB already allocated; 11.69 MiB free; 22.21 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
ERROR diffusion_model.output_blocks.0.1.transformer_blocks.5.attn1.to_v.weight CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 22.41 GiB total capacity; 21.56 GiB already allocated; 11.69 MiB free; 22.21 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
ERROR diffusion_model.output_blocks.0.1.transformer_blocks.5.attn1.to_out.0.weight CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 22.41 GiB total capacity; 21.56 GiB already allocated; 11.69 MiB free; 22.21 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Traceback (most recent call last):
File "D:\Repos\Fooocus\modules\async_worker.py", line 551, in worker
handler(task)
File "D:\Repos\Fooocus\venv\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\Repos\Fooocus\venv\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "D:\Repos\Fooocus\modules\async_worker.py", line 460, in handler
comfy.model_management.load_models_gpu([pipeline.final_unet])
File "D:\Repos\Fooocus\repositories\ComfyUI-from-StabilityAI-Official\comfy\model_management.py", line 397, in load_models_gpu
cur_loaded_model = loaded_model.model_load(lowvram_model_memory)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Repos\Fooocus\repositories\ComfyUI-from-StabilityAI-Official\comfy\model_management.py", line 286, in model_load
raise e
File "D:\Repos\Fooocus\repositories\ComfyUI-from-StabilityAI-Official\comfy\model_management.py", line 282, in model_load
self.real_model = self.model.patch_model(device_to=patch_model_to) #TODO: do something with loras and offloading to CPU
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Repos\Fooocus\repositories\ComfyUI-from-StabilityAI-Official\comfy\model_patcher.py", line 161, in patch_model
temp_weight = comfy.model_management.cast_to_device(weight, device_to, torch.float32, copy=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\Repos\Fooocus\repositories\ComfyUI-from-StabilityAI-Official\comfy\model_management.py", line 498, in cast_to_device
return tensor.to(device, copy=copy).to(dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0; 22.41 GiB total capacity; 21.56 GiB already allocated; 11.69 MiB free; 22.21 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Total time: 209.24 seconds
Keyboard interruption in main thread... closing server.
Process finished with exit code -1
`
At my second attempt to track this error, I add an endpoint. When I clicked generate and it meet the endpoint, I found 6 models have been loaded into memory. May be this is the issue?
Thanks for help~
The text was updated successfully, but these errors were encountered: