We're not detecting all kinds of invalid transmute for function arguments/return values

[RalfJung]

This code should be UB, but is accepted by Miri:

#![allow(internal_features)]
#![feature(core_intrinsics, custom_mir)]

use std::intrinsics::mir::*;
use std::ptr;

// This function supposedly returns a char, but actually returns something invalid
// in a way that never materializes a bad char value.
#[custom_mir(dialect = "runtime", phase = "optimized")]
fn f() -> char {
    mir! {
        {
            let tmp = ptr::addr_of_mut!(RET);
            let ptr = tmp as *mut u32;
            *ptr = u32::MAX;
            Return()
        }
    }
}

fn main() {
    let f: fn() -> u32 = unsafe { std::mem::transmute(f as fn() -> char) };
    // There's a char-to-u32 transmute happening here
    f();
}

The return here is a char-to-u32 transmute, and we are only checking the target type, not the source type. We do have a test for checking the target type.

Something similar can happen for an argument:

#![allow(internal_features)]
#![feature(core_intrinsics, custom_mir)]

use std::intrinsics::mir::*;
use std::ptr;

fn f(_c: u32) {}

// Call that function in a bad way, with an invalid char, but without
// ever materializing this as a char value outside the call itself.
#[custom_mir(dialect = "runtime", phase = "optimized")]
fn call(f: fn(char)) {
    mir! {
        let res: ();
        {
            let c = u32::MAX;
            let tmp = ptr::addr_of!(c);
            let ptr = tmp as *const char;
            // The call site now is a char-to-u32 transmute.
            Call(res, retblock, f(*ptr))
        }
        retblock = {
            Return()
        }
    }
}

fn main() {
    let f: fn(char) = unsafe { std::mem::transmute(f as fn(u32)) };
    call(f);
}

Again the source type of the argument transmute is not checked. We do have a test for checking the target type.