rustc: sane handling for failed memory allocation

[metux]

There should be a sane handling of failed memory allocation (eg. when virtual memory exhausted),
at least issue a proper error, instead of just segfaulting.

[sfackler]

"sane" is a pretty vague term - you're going to have to be more specific. How would a "proper error" be issued? What would the form of that error be?

[metux]

Something like "out of memory while ...", maybe "failed to allocate more pages", etc.
Just segfaulting isn't actually informative :p

[sfackler]

It's not going to segfault, it's going to SIGILL, and it'll print fatal runtime error: allocator memory exhausted before doing that:

$ cat src/lib.rs
#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
        let _v = vec![1; 1024 * 1024 * 1024 * 1024 * 1024];
    }
}
$ cargo test
   Compiling bar v0.1.0 (file:///Users/sfackler/bar)
    Finished dev [unoptimized + debuginfo] target(s) in 0.62 secs
     Running target/debug/deps/bar-4ac15025f612bc06

running 1 test
fatal runtime error: allocator memory exhausted

[metux]

I've observed segfaults w/o any further messages while compiling Servo, right after vsize reached 4G limit.
Exactly what you'd expect from an failed mmap()-based allocation w/o proper error handling.

[sfackler]

You may want to look at core dumps to see where those segfaults are coming from.

[olson-dan]

I'd like for it to panic but I suppose that may not be possible. I'm currently running on Windows where exceeding memory doesn't appear to do anything except print the message "fatal runtime error: allocator memory exhausted" and continue running until the OS kills it. I'm guessing this may be because SIGILL isn't a thing on Windows.

Notably when spawning multiple threads with rayon I see the "fatal runtime error: allocator memory exhausted" message printed from each thread. The process does not appear to be stopped at the point of the failure so I am not sure how best to debug the issue.

Edit: After looking at implementation it seems to boil down to llvm::trap which I guess would be caught on Windows if a debugger was open. Panic would be best, though I understand it being hard to do.

[steveklabnik]

Triage: these semantics are currently on purpose, and adjustments to them involve a bunch of various RFCs. I don't think keeping this open in addition to those is particularly useful, so closing.

[SimonSapin]

Some standard library collections have a try_reserve method to grow them while handling allocation errors. They return a Result. However these methods are still unstable at this point: #48043

Most methods like Vec::push do not return a Result. Instead, on allocation error they call std::alloc::handle_alloc_error which runs a configurable hook, then aborts the process. The default hook prints a message that includes the size of the failed allocation request. The hook can be replaced with std::alloc::set_alloc_error_hook… but that API is also unstable for now: #51245

We also have an accepted RFC for having some unspecified mechanism to turn the process abort into a panic. This is tracked at #43596. It’s possible this will end up resolved as “you can call set_alloc_error_hook with a function that panics”.

Exactly what you'd expect from an failed mmap()-based allocation w/o proper error handling.

As far as I know none of the above applies when mmap is called directly. I don’t know what proper error handling for mmap looks like. (libc’s implementation of malloc might call mmap, but it should return a null pointer on failure.)