Investigate how LLVM deals with huge stack frames

[arielb1]

According to @thestinger, LLVM has UB when dealing with stack frames that have size of the order of magnitude of the address space. Investigate this and add the needed fixes.

This is related to #17913 and not dealt with in #18041.

[steveklabnik]

Triage: I'm not aware of anyone having done any investigation since.

[bstrie]

Is any of this actionable? Is the UB documented somewhere, or described by a bug report? Given that this bug is going on three years old, it's also possible that this isn't relevant anymore. Can someone demonstrate this with a test case? If not, I'd consider closing this.

[bstrie]

It's been three years since this was filed and we still have no clue what this is referring to, no reproduction, and no way to know if this hasn't already been long since fixed on LLVM's side. Closing this, but feel free to reopen if anyone can find a relevant LLVM bug (though #45839 (comment) has set the precedent of "It doesn't make sense to have a bug for every LLVM bug", so use your discretion).

[sunfishcode]

I'm not aware of any specific undefined behavior, though there could well be bugs. Here's an experiment:

pub fn bar(foo: &Fn(&[u8], &[u8], &[u8], &[u8], &[u8], &[u8])) {
    let a = [0u8; 0x7fff_ffff];
    let b = [0u8; 0x7fff_ffff];
    let c = [0u8; 0x7fff_ffff];
    let d = [0u8; 0x7fff_ffff];
    let e = [0u8; 0x7fff_ffff];
    let f = [0u8; 0x7fff_ffff];
    foo(&a, &b, &c, &d, &e, &f);
}

Compile with rustc --crate-type rlib --emit asm -O with a 32-bit x86 target gets this prologue:

        pushl   %ebp
        pushl   %ebx
        pushl   %edi
        pushl   %esi
        movl    $12884901884, %eax
        calll   __rust_probestack
        subl    %eax, %esp

12884901884 is the size we need, but this isn't valid 32-bit x86. llvm-mc silently wraps it, which is also what happens when LLVM uses it in integrated-as mode. GAS also wraps it, though at least gives a warning:

t.s:21: Warning: 00000002fffffffc shortened to 00000000fffffffc

I've now filed https://bugs.llvm.org/show_bug.cgi?id=35345 concerning this.

Since this code is entirely pathological, it seems reasonable to have compilers just reject it. I assume the fix for the above should be to have LLVM use report_fatal_error. I don't know when that bug will be fixed though, so it may be desirable to issue an error based on a conservative approximation before LLVM as well.

If Rust ever has the ability to emit dynamic-sized allocas, that may introduce other concerns not covered here.