Shims ICE when a scalar argument isn't actually a scalar

[RalfJung]

For instance, consider:

extern "C" {
    fn pipe(pipefd: [i32; 2]) -> i32;
}

fn main() {
    let mut fds: [i32; 2] = [0; 2]; 
    assert_eq!(unsafe { pipe(fds) }, 0); 
}

I think to fix this properly we need to completely re-do the way we handle shim arguments: every OpTy needs to be transmuted to the right type before we do anything else with it. And we can't just just call OpTy::transmute as that can't be used for arbitrary transmute, e.g. changing ABIs can fail spectacularly.

Once this is consistently done everywhere, we can also remove the ScalarSizeMismatch error from the interpreter. This was added because otherwise ICEs can be triggered by using an incorrect scalar type in the signature of a function that has a Miri shim, but as we have seen, ICEs can still be triggered -- and once we use our own hard-coded types everywhere, ScalarSizeMismatch can no longer be triggered at all.

We probably want a macro that helps with that.

[RalfJung]

I think we want a new function like get_arg or so that preprocesses the OpTy. Given an op: OpTy and ty: Ty, we do a check like

• let layout = layout_of(ty);
• Check layout.abi.eq_up_to_validity(&op.layout.abi). If that is false, the argument type is invalid -> raise UB.
• Return op.transmute(layout).

[RalfJung]

Similar ICEs can occur when the return type of the function is wrong -- we end up calling write_scalar with a scalar of the wrong size. This either fails to initialize a part of the result (if the scalar is smaller than the return place), or ICEs since we are exceeding the bounds of our AllocRefMut (if the scalar is bigger than the return place).

If it is a ScalarPair, we can even end up writing things at wrong offsets. (But AllocRefMut will at least keep us inside the bounds of the place.)

[geetanshjuneja]

Can I look into this task?

[RalfJung]

Definitely! However, this will require some design work. As mentioned in the issue description, we don't have a clear plan yet for how to best fix this issue.

[RalfJung]

With #4185 having landed, there's a whole lot of low-hanging fruit here that is well-suited for first contributors: pick a family of related shim implementations, and convert them from check_shim to check_shim_abi. For now, skip any shims that take or return any non-trivial types (e.g. structs or really anything except for integers and pointers). We have exactly one example of using check_shim_abi that you can follow:

miri/src/shims/unix/foreign_items.rs

Lines 203 to 210 in 74f4d67

	let [fd] = this.check_shim_abi(
	link_name,
	abi,
	ExternAbi::C { unwind: false },
	[this.tcx.types.i32],
	this.tcx.types.i32,
	args,
	)?;

The main work here is figuring out the argument and return types for all those functions.

[geetanshjuneja]

Hi, sorry I forgot about this. I'll start working on this asap.

[RalfJung]

No rush, and you don't have to do it alone. :) This is easily something where multiple people can get their first Miri contribution.