project-safe-transmute

[nikomatsakis]

Summary

Figuring out how we can make safe ways to transmute.

Links

• Repository rust-lang/project-safe-transmute
• Zulip stream #project-safe-transmute

[nikomatsakis]

• 2020-04-16: current plans are working towards a “much reduced” proposal:
  • what is the minimal trait (or set of traits) to allow some of this to live in libs
  • still in brainstorming state
• Repo with marker traits
  • Some set of marker traits (similar to Copy wrt implementations) → enable operations
    • Opt in (for backwards compatibility, in particular around marker traits exposing internal implementation details unintentionally in a semver-breaking way)
  • Partial progress on determining marker traits.

[nikomatsakis]

• 2020-06-01: Crate posted to the ecosystem experimenting with some of the traits discussed before
  • Trying to figure out if that is covering our needs and whether it can/should transition to compiler
  • Recent experiment used const generics to figure out things like “does a structure have alignment requirements” without requiring special compiler support — so maybe const generics can be used instead to enable?

[nikomatsakis]

2020-06-22:

• We have a scheduled meeting that includes a lot of details with the current status and, possibly, another write-up being prepared.

[nikomatsakis]

2020-07-06:

• We had the design meeting and will publish the minutes.
• We are looking for what are some "minimal steps" lang can take to unblock progress here, with the goal of taking the typic-approach of having libraries experimenting with traits and systems for defining safe transmute.
• One suggestion was to introduce some marker traits that different libraries can use, though this is more of a library concern. Another option might to include some intrinsic const fn or other function for determining when types are "equivalent" etc.
• Next step is to figure out what those steps are and prepare an RFC for them.

[nikomatsakis]

2020-07-20:

• Gearing up to actually post RFC based on the results from previous discussion. Still iterating.

[nikomatsakis]

2020-08-03:

• Draft RFC available

[rylev]

The draft RFC posted above is ready for review. We'd love to have final feedback before posting to the RFCs repo.

The RFC is quite long, but a large part of it is motivation. If you're already convinced that safe transmute opens up a large amount of features for moving currently unsafe code to safe, then you can skip that section.

We'd primarily like to review the overall path forward and whether the mechanisms proposed seem like a reasonable path. Additionally, there is a "MVP" proposal which aims to be a first step in implementing the wider vision. Please let us know if that seems reasonable too.

[nikomatsakis]

2020-08-31:

There has been a lot of iteration and there is now a draft RFC that is ready for lang-team review. There is a "TL;DR" summary that explains the intentions and plan. The full RFC is extremely detailed and almost tutorial-esque. Now at the point where it's time to read through it and decide whether to go forward.

Lang team recommends opening the RFC on the rfcs repo.

[rylev]

The RFC has been posted. 🎉 A big shout out to @jswrenn for the really well crafted RFC.

Besides general feedback, I think it would be particularly interesting to get feedback on the MVP proposal as well as the stability guarantee mechanism which so far seems to be the most controversial aspect of the RFC.

[jswrenn]

I've also written some supplementary text on why there's a stability problem in the first place: https://jack.wrenn.fyi/blog/transmutation-stability/

[joshtriplett]

We discussed this in the lang team meeting today.

@rust-lang/lang folks should look at the RFC and post comments. In particular, in addition to posting issues, please post comments if you think part or all of it looks good, so we can help set direction.

[nikomatsakis]

Current status is that there is a lot of active discussion (primarily in Zulip, with summaries posted to the thread) about the precise mechanism in the RFC, with the goal of avoiding accidental transmutes between structs that couldn't have been constructed (because of non-public fields). Coming towards agreement on the what 'minimal' looks like. The "MVP" has been slimmed down from the original proposal as a result.

[nikomatsakis]

Update 2020-09-21:

Current status is that there is a pending RFC that lang team folks out to be reading (and others!).

rust-lang/rfcs#2981

The RFC was recently updated based on some Zulip feedback. Might be useful to have a design meeting or other discussion about what has changed since the last round of RFC. @joshtriplett to reach out to group to see about producing a summary, as a start.

[rylev]

Update 2020-10-01:

The RFC feedback has settled with no new feedback in the last 6 days. @jswrenn has addressed most of the feedback, and he is currently working on modeling the transmutability rules to formalize them in order to aid in implementation as well as ensure everyone understands exactly what is being proposed. With regards to @joshtriplett's request for a summary on changes to constructability and a subsequent further trimming down of the MVP (which is the only large scale change since the RFC was posted), @jswrenn wrote the a summary

There is a slight concern that we might not have yet received enough feedback from members of the lang and compiler teams. The RFC is quite long so this is perhaps to be expected, but I just wanted to raise the point.

[nikomatsakis]

Update 2020-10-12

@rylev, @joshtriplett -- we discussed this a bit during 2020-10-12 and had a few questions arise in the meeting (will also post to the RFC):

• Niko reports that he read the RFC and was surprised to see things like Here! macro, have to ponder that.
• One question that we were unsure about was whether Here! was meant to guarantee:
  • permitting transmute within a crate even if semver stability is not guaranteed (because you are in same compilation unit)
  • or privacy (i.e., permitting transmute iff you have private access to the fields of a type)
  • or both (or some other reasons)
  • but perhaps we also just need to re-read RFC
• We may want to schedule another design meeting to dig in synchronously, but for now folks still need to read async

@Mark-Simulacrum noted that there was some prior discussion on Zulip.

[rylev]

cc @jswrenn on the above ^^

I think the design has settled enough that another sync meeting would be really helpful. Is there a date we can set that gives folks a reasonable amount of time to go over the RFC again but ensures that we make progress on getting this RFC to the next step?

[jswrenn]

• One question that we were unsure about was whether Here! was meant to guarantee: [...]

See here: rust-lang/rfcs#2981 (comment)

---

I think the design has settled enough that another sync meeting would be really helpful.

Agree completely!

[nikomatsakis]

Update and summary of conversation:

• Thread is pretty active, some debate about what the "minimum surface area" truly is.
• The Here! macro is one center of controversy, follow up to NeglectStability.
• As an implementation detail it might make sense, but not clear that Here! is the right point in the design space overall.
• Follow-up design meeting makes sense -- needs an issue filed with some agenda, @scottmcm will take point on that.

[nikomatsakis]

Update 2020-10-26:

• Would still like a follow-up design meeting, but haven't gotten organized yet.

[rylev]

Update 2020-11-02:

The controversial idea of a generic Scope parameter (and associated Here! macro) was dropped in favor of a more conventional Muckable trait which mechanically mirrors the Copy trait. Types that implement this trait state that their layouts are a part of their stable API and changing the layout constitutes a breaking API change.

This mechanism offers less fine grained control (when compared to the previous mechanism) over declaring that a type can be stably transmuted to, but it was argued that existing patterns (such as the new type pattern) can be used if the user desires more fine grained control. Additionally, other mechanisms can be added in future RFCs that would allow users to stably permit certain transmute operations for types that don't implement Muckable.

The NeglectStability mechanism (which was previously noted as controversial) is still part of the design.

[scottmcm]

Jack as written up a new summary of the proposal to have it available in one place: https://hackmd.io/@jswrenn/SJmHwgdov

[jswrenn]

Update 2021-04-06:

We opened an MCP for a transmutability lang item that will serve as a foundation for further experiments in building abstractions over transmutability. I also closed the initial RFC. We'll revisit the issue of public-facing, stable APIs after we have some time to experiment. The MCP is in final comment period, and I'm looking forward to getting started on implementation. While I think I have a good sense of how to programmatically test for transmutability, I'm sure I'll have plenty of questions about how to integrate it into the compiler once I get down to it.

[jswrenn]

Update 2021-06-29:

No significant progress since the last update due to hardware failure and dissertation. I started on implementation last week, but could use some mentoring.

[jswrenn]

Update 2021-12-01:

Implementation work has accelerated thanks to @oli-obk's mentoring! I have a branch implementing a stub of the transmutability trait, just to pin down exactly where this mechanism will fit into trait selection. Meanwhile, @ezrosent did a literature review, recognized some similarities between transmutability and bisimulation, and is working on formulating an algorithm for checking transmutability.

[jswrenn]

Update 2021-12-24:

rust-lang/rust#92268 provides an initial, incomplete implementation of MCP 411: Lang Item for Transmutability!