[Core] Wrong detection of the space available.

[Bye-legumes]

What happened + What you expected to happen

When we run ray in container, ray will have log looks like this

That means the capacity is detected from the host machine (1000GB), while the available gets from the docker file. So although I have 32GB available as it believe the disk is almost full 95%, it still has misleading log.

Versions / Dependencies

latest.

Reproduction script

build:
      context: .
      network: host
      dockerfile: Dockerfile
    volumes:
      # With this you can use docker outside (bmc host docker engine) of docker (devcontainer)
      # Forwards the local Docker socket to the container.
      # - /var/run/docker.sock:/var/run/docker-host.sock

      # Update this to wherever you want VS Code to mount the folder of your project and change in devcontainer.json to auto open it
      - ../:/workspace:consistent
      - type: tmpfs
        target: /workspace/tmp/
        tmpfs:
          size: 16G  # Size in bytes, 15GB
    # allow usage of gpus
    deploy:
      resources:
        #reservations:
        #  devices:
        #    - driver: nvidia
        #      count: all
        #      capabilities: [gpu]
        limits:
          cpus: "10.0"
          memory: 32G

    # shared mem used by Ray otherwise it will use /tmp that reduce performance
    shm_size: '15gb' # default 96/3

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[rkooo567]

based on my search, it seems like docker doesn't isolate disk space https://www.google.com/search?q=docker+isolate+disk+space%3F&oq=docker+isolate+disk+space%3F&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigATIHCAMQIRifBTIHCAQQIRifBTIHCAUQIRifBTIHCAYQIRifBTIHCAcQIRifBTIHCAgQIRifBTIHCAkQIRifBdIBCDMzNzBqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8.

How did you set 32GB disk space limit on your container? also what's the command you use to get 32GB disk space? (is it df -h)

[Bye-legumes]

based on my search, it seems like docker doesn't isolate disk space https://www.google.com/search?q=docker+isolate+disk+space%3F&oq=docker+isolate+disk+space%3F&gs_lcrp=EgZjaHJvbWUyBggAEEUYOTIHCAEQIRigATIHCAIQIRigATIHCAMQIRifBTIHCAQQIRifBTIHCAUQIRifBTIHCAYQIRifBTIHCAcQIRifBTIHCAgQIRifBTIHCAkQIRifBdIBCDMzNzBqMGo3qAIAsAIA&sourceid=chrome&ie=UTF-8.

How did you set 32GB disk space limit on your container? also what's the command you use to get 32GB disk space? (is it df -h)

maybe I set wrong mount disk. Sry for confusion.

[Bye-legumes]

close

[Liquidmasl]

I am running in this issue as well I believe.

as available space it shows my 4tb, but the available is... a lot less, even though i have a lot of space left, so ray stops while it should not.

I dont set any memory limit in the docker compose. and I set ray _memory to 800gb
shm is set in the docker compose to 200gb

I dont understand whats happening
(raylet) [2024-08-21 13:11:24,500 E 254 284] (raylet) file_system_monitor.cc:111: /tmp/ray/session_2024-08-21_12-57-11_901253_74 is over 95% full, available space: 147223207936; capacity: 3936290357248. Object creation will fail if spilling is required.

So where does it get available space from? and where does he get total memory from?