[strict provenance] Rationalize "Oxford Casts"

[Gankra]

This issue is part of the Strict Provenance Experiment - #95228

Oxford Casts are those that fly in the face of Harvard Architectures, and let you convert between data pointers and function pointers. In my current understanding, this affects both AVR and WASM.

For WASM, function pointers are actually indices into a ~vtable somewhere. This seems to mostly work fine and it's just a matter of "ask bad questions, get bad answers"

For AVR it causes Actual Compiler Problems that the stdlib seems to be actively routing around:

rust/library/core/src/ptr/mod.rs

Lines 1390 to 1395 in 9280445

	// HACK: The intermediate cast as usize is required for AVR
	// so that the address space of the source function pointer
	// is preserved in the final function pointer.
	//
	// https://github.com/avr-rust/rust/issues/143
	fmt::Pointer::fmt(&(*self as usize as *const ()), f)

I do not know what "resolving" this looks like. It could involve any of:

• just documenting the current state of affairs
• developing a "coherent" model for what these operations do
• fixing the compiler issues with AVR
• forbidding oxford casts on some or all targets
• introducing new APIs to "do it right"

I can't really help any more than this, because this is all outside my areas of expertese/motivation.

[Gankra]

Oh also, it might be useful/important to make it easier to talk about "some" function pointer. That came up a lot in places with oxford casts.

The problematic code above is already flailing at the classic Tuple Problem:

rust/library/core/src/ptr/mod.rs

Lines 1746 to 1758 in bb5c437

	fnptr_impls_args! {}
	fnptr_impls_args! { A }
	fnptr_impls_args! { A, B }
	fnptr_impls_args! { A, B, C }
	fnptr_impls_args! { A, B, C, D }
	fnptr_impls_args! { A, B, C, D, E }
	fnptr_impls_args! { A, B, C, D, E, F }
	fnptr_impls_args! { A, B, C, D, E, F, G }
	fnptr_impls_args! { A, B, C, D, E, F, G, H }
	fnptr_impls_args! { A, B, C, D, E, F, G, H, I }
	fnptr_impls_args! { A, B, C, D, E, F, G, H, I, J }
	fnptr_impls_args! { A, B, C, D, E, F, G, H, I, J, K }
	fnptr_impls_args! { A, B, C, D, E, F, G, H, I, J, K, L }

Too tired to dig up the links but there is also:

• Code in fmt that was trying to compare the address of a function with an "opaque" ABI to one with a more specific one to check if fmt is being invoked with a specific special callback (usize formatter?).

• Code in linux sys/os that seems to be implementing dlopen and therefore is using ⚠️transmute_copy⚠️ on a usize to 🙀an arbitrary generic T🙀 to convert from the "opaque" dlopen result to the expected function pointer type.

I sketched out an OpaqueFnPtr type type in sptr but without lang support it has to do the transmute_copy nightmare I describe above so it's unclear if it's at all useful.

[Gankra]

Oh wait nice, I forgot I actually originally "fixed" all these cases and then got skittish about it, but left the "revert" as a separate commit precisely so it would be easier to see all the places where we do "interesting" stuff with function pointers:

b608df8

[RalfJung]

fixing the compiler issues with AVR

Arguably it is a bug that rustc generates fn ptrs and raw ptrs into different address spaces but then fails to emit addrspacecasts between them when lowering casts. This is really not something that library code should have to work around.

[leo60228]

I think I mentioned this on Twitter, but just as a note, the specific libcore cast mentioned just gets converted back to a usize by fmt::Pointer::fmt.