OOM is silent beyond aborting

[alexcrichton]

Right now when we hit OOM we simply invoke abort(), an LLVM intrinsic for an undefined instruction to kill the program. This led me to having to run an unoptimized rustc in GDB for 4+ hours just to figure out the backtrace pointed at alloc::heap::reallocate.

We should make OOM easier to debug by at least printing a message. Note, however, that the current OOM strategy was done to allow LLVM to optimize allocations, so we need to balance our priorities with that.

[brson]

We need an abort function somewhere so people stop calling that intrinsic.

[huonw]

Note, however, that the current OOM strategy was done to allow LLVM to optimize allocations, so we need to balance our priorities with that.

I would think that a function like

#[cold]
#[inline(never)]
fn fail_alloc() -> ! {
   rterrln!("failed allocation: aborting");
   abort()
}

would be nearly as optimisable as abort (especially if we could mark it nothrow/readnone etc. somehow).

[thestinger]

It just needs to be extra sure that there's no allocation happening as part of the printing.

[tpickett66]

This seems like a good easy(ish) ticket to get started learning on but wanted to make sure it wasn't already in the works before I spent too much time on it.

In the event it hasn't been worked on yet I was thinking something like this might work in place of the existing oom function

/// Common out-of-memory routine
#[cold]
#[inline(never)]
static OOM_MESSAGE = "failed allocation: aborting\n"
pub fn oom() -> ! {
    // FIXME(#14674): This really needs to do something other than just abort
    //                here, but any printing done must be *guaranteed* to not
    //                allocate.

    unsafe {    
        with_task_stdout(|io| {
            io.write(OOM_MESSAGE.as_bytes())
        })
        core::intrinsics::abort()
    }
}

[tpickett66]

After looking into with_task_stdout this approach won't work since it may try to alloc at least one additional memory segment.

[eddyb]

@tpickett66 Look at how rtprint does it, it just goes straight to write.

[codyps]

I'm not familiar with how rust is currently handling oom internally, but for some cases it doesn't seem like we necessarily need to be careful (at all) about allocating more memory.

Example: Vec::with_capacity(larger_number_than_all_memory).

In that case, we actually have plenty of memory, we just have an erroneous allocation that fails. In this particular case, it might be possible to just panic!() (and have us get all the nice things that come with panicing, like an optional backtrace).

To avoid adding heuristics to detect this case, it might make sense to just have oom try to panic first, but detect recursion and in that event execute a simpler non-allocating bailout.

The above would actually be fairly helpful even if we keep the plain call to abort().

[eddyb]

@jmesmon I believe it's possible to panic with a static allocation, if the handler checks for that case (and doesn't treat the pointer as a Box<T>).
Then you're left with allocations in destructors, which are rare (I don't recall a single one, actually. unless it's a panicking destructor).
Even if there are spurious allocations, they may still not hit the OOM condition - basically what you're saying, AFAICT.

Oh, I see @alexcrichton mentions optimizing allocations.
That makes it trickier, if nounwind actually results in significant optimizations around allocations.
There's still the choice to invoke the backtrace machinery and then abort, if you have a hook for that in liballoc.
cc @dotdash

[codyps]

@eddyb my point is that in some OOM cases we aren't actually out-of-memory.

In those cases, further allocations will still succeed even though a bogus allocation attempt was issued.

[eddyb]

@jmesmon yes, but @alexcrichton's point was about the possibility of unwinding interfering with optimizations, and causing a lot more landing pads to be generated.

[alexcrichton]

Closing in favor of #27335, I think that's a good solution to take.

[bluss]

This is a bug even if the solution has been redirected to the RFC process. I want to track this.

[steveklabnik]

Then it should be tracked in the RFC repo. We don't keep bugs open for things that go through RFCs.

[bluss]

We do keep bugs here though? Is this really just a new feature?

[ghost]

Seems like this discussion might be better in rust-lang/rfcs#1398