Abort on some large allocation requests, Panic on other

[bluss]

Compare these two:

let v = vec![0u8; !0];
let u = vec![0u16; !0];

We request a vector with 18446744073709551615 elements.

• For u8 we receive out of memory (null) from the allocator, and call alloc::oom, which aborts the program: application terminated abnormally with signal 4 (Illegal instruction)
• For u16, we get an assertion: thread '<main>' panicked at 'capacity overflow', ../src/libcore/option.rs:330

This is inconsistent. Why don't we abort on both of these cases? We abort on too large allocation requests, so those that are even larger could abort too?

[arielb1]

@bluss

For the record, your code is the equivalent of

fn main() { let v = ::std::vec::from_elem(0u16, !0); }

In the u16 case, the amount of memory that is supposed to be allocated is 2*(2^64 - 1), which exceeds the size of usize. That is checked for rather explicitly in Vec::with_capacity. Maybe we should abort instead of panicking in that case too.

[bluss]

Additional info: already !0 bytes is more than we can safely allow allocating (due to known fact #22104). We should treat that as capacity overflow, but decline to do so, because it will abort anyway.

[steveklabnik]

/cc @rust-lang/compiler , I think?

[Gankra]

This is a @rust-lang/libs problem.

I am of the opinion that aborts should only occur on problems that unwinding can't address. That is, if you overflow the capacity we have to check for that, and we can panic right away. Everything will be cleaned up and the world will be happy.

However if an allocation makes it past our checks and then fails in the allocator, we have historically conservatively assumed that you are legitimately OOM and abort; unwinding can lead to arbitrary other allocations which is presumably bad to do during OOM.

In practice, the platforms we support basically can't OOM naturally (by which I mean, you're asking for the last few bits of free memory), so I reckon if you ever managed to trigger an OOM it's because you managed to request All The Memory, in which case there's plenty of slack space for unwinding. If it's a legit OOM then we'll double panic and crash anyway.

However #14674 mentions that aborting on OOM enables LLVM to better optimize around allocations. I don't know the details.

Edit: Regardless, I do not want to add any kind of "heuristic checks" for "will probably OOM" or "was probably a real OOM".

[Aatch]

I think that, ultimately, abort vs. unwind behaviour should be left up to the implementation, but the standard library should abort on failed allocations. This particular example is just a red herring. It's only because u8 is a single byte that it doesn't trigger any kind of arithmetic overflow behaviour. It's an interesting quirk, but doesn't really reflect any real inconsistent behaviour.

[Gankra]

Also to be completely clear to those not super familiar with the issue: the "odd" behaviour described in this issue is completely by design, and not a mistake. Rather this issue is positing that the behaviour should be changed to be more consistent.

[bluss]

Of course. I don't think it's completely by design, it's patched up to be this way. For example, if we know a capacity of > isize::MAX is always too large, why does that abort and not panic?

[Gankra]

@bluss any time we detect this, I believe we panic. Only alloc(..) == null aborts to my knowledge: https://github.com/rust-lang/rust/blob/master/src/liballoc/raw_vec.rs

We simply don't even check in some cases where we know any degeneracies will be caught by the allocator (e.g. when doubling capacity on 64-bit).

[Gankra]

Note that checking for > isize::MAX on 64-bit is almost surely useless since your system will OOM you far before that point using normal growth. If you explicit ask for > isize::MAX then you're basically arbitrarily "lucky". It's almost inconsistent to check, in that regard.

[nagisa]

checking for > isize::MAX on 64-bit is almost surely useless since your system will OOM you far before that point using normal growth.

That’s what they said about 32 bits timestamps and addressing too.

[Gankra]

@nagisa Today this is simply a hard hardware issue: you only get 48 bits of the address space.

[nagisa]

Yes, but it is always useful to be aware of future prospects.

[Gankra]

@nagisa I have thankfully architected the current design to make such an upgrade trivial. In fact, all you have to do is delete some code!

https://github.com/rust-lang/rust/blob/master/src/liballoc/raw_vec.rs#L445-L453

[steveklabnik]

So is this not a bug?

[Gankra]

It's not an accident, at least.

On Tue, Aug 4, 2015 at 2:54 PM, Steve Klabnik notifications@github.com
wrote:

So is this not a bug?

—
Reply to this email directly or view it on GitHub
#26951 (comment).