Docker compose: Disk space and other issues

[trapf]

System

• Arch Linux, x86
• docker-compose
• nvidia-container-toolkit
• cuda 12.6
• RTX 4070 Ti

Reproduction

1. clone repository (commit e872353)
2. execute sudo docker compose up -d
3. open http://localhost:7865/ or http://127.0.0.1:7865/
4. leave default parameters, click Generate button

The Issue

• unable to generate samples

Docker Logs

ace-step  | 
ace-step  | ==========
ace-step  | == CUDA ==
ace-step  | ==========
ace-step  | 
ace-step  | CUDA Version 12.6.0
ace-step  | 
ace-step  | Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
ace-step  | 
ace-step  | This container image and its contents are governed by the NVIDIA Deep Learning Container License.
ace-step  | By pulling and using the container, you accept the terms and conditions of this license:
ace-step  | https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
ace-step  | 
ace-step  | A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
ace-step  | 
ace-step  | * Running on local URL:  http://0.0.0.0:7865
ace-step  | 
ace-step  | To create a public link, set `share=True` in `launch()`.
ace-step  | 2025-05-13 07:10:31.337 | WARNING  | acestep.pipeline_ace_step:__call__:1613 - Checkpoint not loaded, loading checkpoint...
ace-step  | 2025-05-13 07:10:31.338 | INFO     | acestep.pipeline_ace_step:load_checkpoint:157 - Download models from Hugging Face: ACE-Step/ACE-Step-v1-3.5B, cache to: /home/appuser/.cache/ace-step/checkpoints
ace-step  | /opt/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py:799: UserWarning: Not enough free disk space to download the file. The expected file size is: 6611.42 MB. The target location /home/appuser/.cache/ace-step/checkpoints/models--ACE-Step--ACE-Step-v1-3.5B/blobs only has 1311.58 MB free disk space.
ace-step  |   warnings.warn(
ace-step  | Traceback (most recent call last):
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 468, in http_get
ace-step  |     hf_transfer.download(
ace-step  | Exception: Failed too many failures in parallel (3): Request: error sending request for url (https://cdn-lfs-us-1.hf.co/repos/cd/9a/cd9acef1a73c791752bc8518b5af176197237d3403ba695cd46566af21d88362/e810f16728d8a2e0d1b9c3a907aac8c9a427ce38edbd890cb3dce5ff92da5aad?response-content-disposition=inline%3B+filename*%3DUTF-8%27%27diffusion_pytorch_model.safetensors%3B+filename%3D%22diffusion_pytorch_model.safetensors%22%3B&Expires=1747123832&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc0NzEyMzgzMn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2RuLWxmcy11cy0xLmhmLmNvL3JlcG9zL2NkLzlhL2NkOWFjZWYxYTczYzc5MTc1MmJjODUxOGI1YWYxNzYxOTcyMzdkMzQwM2JhNjk1Y2Q0NjU2NmFmMjFkODgzNjIvZTgxMGYxNjcyOGQ4YTJlMGQxYjljM2E5MDdhYWM4YzlhNDI3Y2UzOGVkYmQ4OTBjYjNkY2U1ZmY5MmRhNWFhZD9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPSoifV19&Signature=j1sbRtB91SjXkR8WMfcufJechws%7E4Vk8LBCWLLIjFCDfvyy-pDwJg0kZS465jFReRXbWOkDuXz2oF%7Eauf72L3fsNvPsS2rB0TBoW08GID-G2JjgI%7E7bQBS37C0gOZjEjR8TM9WPnQGKDew4TaPfyfmq1Sp1%7E5lH%7Ea7ZMGuG3m%7ECA12d8JQC-quaw%7E07Ae385FxXJBYtJway0UEjiub2K4GMaxMJx2ZcnII8ErF--O4LbqDuPU7aUnCUq7%7EgE6uHS1SnUWhzYrr7guv7bobjjmdw%7EPlFWOjcaHU-a09F1tkWNY00SBOC6TD9iXkFgb9Ud0d6vqJ57L%7E5rRoIx7wzUzA__&Key-Pair-Id=K24J24Z295AEI9) (no permits available)
ace-step  | 
ace-step  | The above exception was the direct cause of the following exception:
ace-step  | 
ace-step  | Traceback (most recent call last):
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/gradio/queueing.py", line 625, in process_events
ace-step  |     response = await route_utils.call_process_api(
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/gradio/route_utils.py", line 322, in call_process_api
ace-step  |     output = await app.get_blocks().process_api(
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/gradio/blocks.py", line 2137, in process_api
ace-step  |     result = await self.call_function(
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1663, in call_function
ace-step  |     prediction = await anyio.to_thread.run_sync(  # type: ignore
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
ace-step  |     return await get_async_backend().run_sync_in_worker_thread(
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
ace-step  |     return await future
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 967, in run
ace-step  |     result = context.run(func, *args)
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/gradio/utils.py", line 890, in wrapper
ace-step  |     response = f(*args, **kwargs)
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/acestep/pipeline_ace_step.py", line 1617, in __call__
ace-step  |     self.load_checkpoint(self.checkpoint_dir)
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/acestep/pipeline_ace_step.py", line 158, in load_checkpoint
ace-step  |     checkpoint_dir_models = snapshot_download(REPO_ID, cache_dir=checkpoint_dir)
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
ace-step  |     return fn(*args, **kwargs)
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 295, in snapshot_download
ace-step  |     _inner_hf_hub_download(file)
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 271, in _inner_hf_hub_download
ace-step  |     return hf_hub_download(
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
ace-step  |     return fn(*args, **kwargs)
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1008, in hf_hub_download
ace-step  |     return _hf_hub_download_to_cache_dir(
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1159, in _hf_hub_download_to_cache_dir
ace-step  |     _download_to_tmp_and_move(
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1723, in _download_to_tmp_and_move
ace-step  |     http_get(
ace-step  |   File "/opt/venv/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 479, in http_get
ace-step  |     raise RuntimeError(
ace-step  | RuntimeError: An error occurred while downloading using `hf_transfer`. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.

[ChuxiJ]

The Docker deployment hasn't been fully tested. If you solve this problem, you are welcome to submit a PR to us.

[Dannyzen]

This may be a local disk space issue. Can you confirm there is enough space on the partition in which docker stores its images? docker info | grep "docker root" should tell you where that is.

[trapf]

This may be a local disk space issue. Can you confirm there is enough space on the partition in which docker stores its images? docker info | grep "docker root" should tell you where that is.

Seems like docker is located on the root partition of the Arch installation, when installed via the pacman package manager. This in itself is not great, since the root partition is usually capped to a low amount of disk space, but that did not cause the issue.
I believe part of the issue is that the volumes defined in the docker-compose.yml are not relative paths. This makes it so the volumes are stored on the root partition, as well. Downloading the model causes the root partition to fill up and eventually have no more space.
This is not great and potentially a big problem for installations like mine. A solution is to remove defining the volumes at the bottom of the yml file, and replacing the volumes inside the docker service with this:

services:
  &name ace-step:
    build:
      context: https://github.com/ace-step/ACE-Step.git
      dockerfile: Dockerfile
    container_name: *name
    hostname: *name
    stop_grace_period: 2s
    ports:
      - "7865:7865"
    
    volumes:
      - ./appuser:/home/appuser
      - ./app-root:/app

I have to note that I am not that experienced with writing Dockerfiles and docker-compose.ymls myself, but for my case, it sorted it out and created the volumes in the repository root instead. However, I had to move the acestep directory into ./app-root. Maybe replacing ./app-root with ./acestep would suffice, but I haven't tested that yet.

[Dannyzen]

Thanks for raising. As many projects have relative directory assignments and it is reasonable to expect the behavior of the webui to ouput files into a ./outputs path, (similar to stable-diffusion) I've made this change in #171