add checks to the `transmute` intrinsic to only allow "safe" transmutes

[oli-obk]

examples of unsafe transmutes:

• 32-bit values that aren't valid unicode indices casted to char
• &T -> &mut T
• null value to a NonZero type

examples of transmutes that should be something else (warn?)

• integer -> *const T (cast)
• &T -> *const T (cast/coercion) (linted by clippy)
• *const T -> &T (should be a deref) (linted by clippy)

[solson]

integer -> *const T (cast)

This is actually really common (e.g. std::ptr::null() does this).

&T -> *const T (cast/coercion)
*const T -> &T (should be a deref)

I agree but this is out of scope of Miri. rustc itself should lint these, IMHO. (Normally I would say clippy, but I think these are important enough for rustc.)

[oli-obk]

This is actually really common (e.g. std::ptr::null() does this).

std::ptr::null uses a cast

[solson]

@oli-obk Oops, sorry, somehow I misunderstood what you meant even though it was clear. I should have been sleeping rather than replying. :)

Yeah, those transmutes should be linted against, too, ideally by rustc.

[cramertj]

| 32-bit values that aren't valid unicode indices casted to char

Shouldn't they be using std::char::from_u32 or std::char::from_u32_unchecked?

[bjorn3]

Transmuting a struct with integer at some place to a struct with pointer at that place should be a error.

[oli-obk]

Transmuting a struct with integer at some place to a struct with pointer at that place should be a error.

that means that 0 as *const i32 should be an error, too, since the equivalent transmute<usize, *const i32>(0) would be an error. We'll catch such errors at the use site, if they are errors.

Ignore that. We should definitely error at transmutes, but allow pointer casts until dereferenced

[cramertj]

Not sure how hard it would be to check this, but it should also be an error to transmute to an enum with an invalid discriminant.

[solson]

This discussion is going out of bounds in terms of what makes sense to implement inside of Miri. Miri already handles when you do invalid transmutes and then do something that depends on the result being valid (like matching over an enum), it just doesn't error out at transmute time.

Like I said before, this is maybe something that could be brought up as a lint for rustc, but it's not actionable for Miri.

[oli-obk]

Miri can detect transmutes that are UB because of the value. Everything that is UB because of the types should be a lint.

[solson]

When is a transmute ever UB just by itself? The UB generally comes from the invalid value being used later (and it might not be).

Additionally, it doesn't make sense to special-case transmute. The same effects come from derefing pointers (which might have been casted) and possibly other means. In all these cases Miri is already set up to deal with invalid values that might arise.

[oli-obk]

right. it would probably be better to include a full tracing mode instead of doing something like this.