Add `-Zrandomize-layout` flag to better detect code that rely on unspecified behavior related to memory layout

[marmeladema]

In order to better detect code that rely on unspecified behavior related to memory layout, it could be useful to have a -Zrandomize-layout flag that add some padding bytes at the start of all #[repr(rust)] structs and unions. It should also add different amounts of padding to different fields.

This should help detecting invalid transmutes, or invalid unions.

Zulip thread: https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/Transmute.20safety.20question

[RalfJung]

I think this could be very useful indeed... maybe @eddyb has some ideas for how to best implement this.

[eddyb]

You can make this deterministic by using the def_path_hash for a given definition (it's an equivalent of DefId that's stable across incremental recompilation) - you might not even need to seed a PRNG with it if all you need is one value!

We can also do a limited version of this by default, but since it won't change the size, it would only fail at runtime #38550 (comment)

[eddyb]

On Zulip @RalfJung pointed out that I didn't explain why you would want to make this deterministic (or why that's the main difficulty), so here's 3 reasons why this has to be deterministic:

• unsizing coercions mean that generic types need to have the same field order across instantiations, when the field types match (ignoring the field that would be unsized) - so we can reorder based on alignment, but not randomize differently each time
• layouts have to be the same for the same type across sibling crates (which do not know about eachother)
• introducing non-determinism into the query system is unsound at least for incremental recompilation (where you will get miscompiles, i.e. potentially UB in safe code, if you have things like order-dependency or outright RNG, as the incremental dependency tracking will not know to invalidate queries that would now produce different results)

There's two ways to get deterministic per-type-definition "seeds":

• generate a random number when compiling the definition (and use that one value for shuffling the layout of every instance in any crate) - this is still troublesome with incremental, you would have to add it explicitly as an incremental input, in order to be sound, and it will cause a lot of invalidation
• use the def_path_hash (which I mentioned in the previous comment), which is effectively the TypeId hash corresponding to a DefId, and rely on its hash properties be pseudo-random (it should change with compiler releases and crate versions specified in the Cargo.toml, so at least we should get some variety - though maybe not as much on nightly sadly)

You might probably also want a CLI flag to e.g. XOR in a fixed seed as well, even with the latter method, but it would have to be embedded in the type definition (so that it's correct cross-crate without requiring the same flag to passed in order to avoid miscompilation), so it's as much effort as both.

[petrochenkov]

Randomizing the layout for testing purposes was being suggested starting from the original struct layout RFC (https://github.com/rust-lang/rfcs/blob/master/text/0079-undefined-struct-layout.md#alternatives), it's unfortunate that it's still not implemented.

I've found one similar existing issue for this - #38550.

[elichai]

I think the "easiest" thing that should also give the highest reward is adding uninit bytes at the front of every repr(Rust) struct (obviously under some -Zpad-layout)
because this will:

1. Break compilation for direct transmutes (different sizes).
2. Will break anything that assumes specific size.
3. Will make every direct read from a pointer UB because it will read uninitialized values, which Miri should catch.

we can then extend to:
A. Randomize the fields order.
B. Mess with alignment (can probably only be increased and not decreased).
C. Add uninit bytes at the middle/end? (not sure if this achieves anything)

[marmeladema]

I would be glad to help implementing a first iteration of those ideas namely padding bytes at the start of repr(rust) struct. But I am not knowledgeable enough about those parts of the compiler.
Would someone be available to mentor me?

[eddyb]

@marmeladema It's pretty much all in rustc_middle::ty::layout. Wherever field offsets are computed (univariant_uninterned), you can, for example, add the final alignment to all of the field offsets - this will increase the size of the whole type but keep all fields correctly aligned (and it's slightly simpler than changing field order, which has to be done similarly to how we sort fields by alignment).
You should be able to PM me if you have more questions.

If you rely on the same flag to be passed across all the crates, or if you record what the choices were when a certain crate was contained (you could look at how -Z symbol-mangling-version is handled), you should be able to avoid problems with cross-crate consistency, which as per #77316 (comment) was my main worry.