Shims ICE when a scalar argument isn't actually a scalar

[RalfJung]

For instance, consider:

extern "C" {
    fn pipe(pipefd: [i32; 2]) -> i32;
}

fn main() {
    let mut fds: [i32; 2] = [0; 2]; 
    assert_eq!(unsafe { pipe(fds) }, 0); 
}

I think to fix this properly we need to completely re-do the way we handle shim arguments: every OpTy needs to be transmuted to the right type before we do anything else with it. And we can't just just call OpTy::transmute as that can't be used for arbitrary transmute, e.g. changing ABIs can fail spectacularly.

Once this is consistently done everywhere, we can also remove the ScalarSizeMismatch error from the interpreter. This was added because otherwise ICEs can be triggered by using an incorrect scalar type in the signature of a function that has a Miri shim, but as we have seen, ICEs can still be triggered -- and once we use our own hard-coded types everywhere, ScalarSizeMismatch can no longer be triggered at all.

We probably want a macro that helps with that.

[RalfJung]

I think we want a new function like get_arg or so that preprocesses the OpTy. Given an op: OpTy and ty: Ty, we do a check like

• let layout = layout_of(ty);
• Check layout.abi.eq_up_to_validity(&op.layout.abi). If that is false, the argument type is invalid -> raise UB.
• Return op.transmute(layout).

[RalfJung]

Similar ICEs can occur when the return type of the function is wrong -- we end up calling write_scalar with a scalar of the wrong size. This either fails to initialize a part of the result (if the scalar is smaller than the return place), or ICEs since we are exceeding the bounds of our AllocRefMut (if the scalar is bigger than the return place).

If it is a ScalarPair, we can even end up writing things at wrong offsets. (But AllocRefMut will at least keep us inside the bounds of the place.)