[Object Spilling] Plasma store uses more shared memory than object_store_memory sometimes when spilling

[rkooo567]

What is the problem?

It looks like sometimes (when there's a huge memory pressure), plasma store uses more memory than object_store_memory, which causes the SIGBUS. For example, I ran one stressful spilling workload, and I received SIGBUS while my /dev/shm size was 120GB and the object store memory limit was 80GB. When I killed the raylet, all memory was freed.

Reproduction (REQUIRED)

Please provide a short code snippet (less than 50 lines if possible) that can be copy-pasted to reproduce the issue. The snippet should have no external library dependencies (i.e., use fake or mock data / environments):

No reproduction now, but it should be relatively easy to create the one.

• I have verified my script runs in a clean environment and reproduces the issue.
• I have verified the issue also occurs with the latest wheels.

[ericl]

I can reproduce this pretty easily by writing big files to /dev/shm (cat /dev/zero > /dev/shm/file) and then trying to put more objects in Ray. I think the underlying issue is we are not catching a failed allocation correctly, trigger SIGBUS somewhere in plasma's memory allocator (cc @suquark ).

[ericl]

It seems the issue here is pretty fundamental to the way we are using mmap. We are creating a file with unallocated pages in /dev/shm (to avoid immediately using memory). However, this means the application can get SIGBUS at any point if here are no more pages allocatable in /dev/shm.

Possible alternatives include:

• dynamically growing the file in /dev/shm
• zeroing out all new allocated pages in plasma to catch SIGBUS

[ericl]

@rkooo567 can you try using the flag? If it works, I think we can add this to our documentation.

[rkooo567]

The issue wasn't reproducible always. But I can try with 100GB shuffle just to see how slow it is when initiated. I can @clarkzinzow can also verify it with his Uber workload.

[ericl]

FYI the workaround is to set RAY_PREALLOCATE_PLASMA_MEMORY=1; available in nightly builds (introduced in https://github.com/ray-project/ray/pull/15669/files)

[stephanie-wang]

#15951 fixes this issue, but it can cause ObjectStoreFullErrors instead, since the underlying issue of fragmentation is still there. @ericl is working on a potential fix in #16097, which uses /tmp as a fallback when plasma runs out of shared memory.