better documentation of reborrowing

[soulwa]

tl;dr: Rust documentation is unclear on reborrowing, should explain details in the Reference

As it stands, there is only one mention of "reborrowing" in the Reference, in this section on raw pointers. This only refers to reborrowing pointers, and not references, which is important in understanding how Rust handles mutable references (often implicitly reborrowing them instead of moving them). There is also some documentation on implicit borrowing, but again, this is sparse and does not cover reborrowing in the context of mutable references.

A key example of the implicit borrowing which confused me is the following:

fn main() {
   	let mut x = vec![1, 2, 3];
   	let y = &mut x;
   	let z: &mut Vec<i32> = y;
   	z.push(4);
   	y.push(5);
}

This code compiles fine, and x, as expected, becomes [1, 2, 3, 4, 5]. Initially, this code was unclear to me, but I was able to understand how it worked learning about both reborrowing and non lexical lifetimes. However, along the way I became familiar with how little documentation there is on the former. Questions have come up on the topic frequently, like this Reddit post from 2016, this post from 2018 on the Rust forums, more Reddit posts from 2018 and 2019, and in this forum post from January, which I found extremely helpful. A few of the discussions in these posts reference sources to reborrowing, but the only official documentation which was available is an earlier version of the Rustonomicon. There are some instances of the author of The Rust Programming Language, such as here, here, and in rust-lang/rust#25899, mentioning adding reborrowing to the new book, but it seems that has been abandoned since: see rust-lang/book#2144.

This concept is counter-intuitive to beginners, as it seems to violate the idea that there can only be one mutable reference to an object in a given scope. It also raises frequent forum questions, which become an effective way to get answers, but only due to the lack of documentation. Because of this, I think that some proper documentation on reborrowing is necessary. It would explain some of the quirks of this system, as choosing whether or not to reborrow is a decision made by the compiler, and can lead to some strange compilation errors without understanding this concept. For example,

let mut x = vec![1, 2, 3];
let y = &mut x;
let z = y;

will move y into z, but substituting the last line for

let z: &mut _ = y;

will reborrow y, since the compiler has to guarantee that z is &mut T. y can be freely used after the second case, but not at all after the first case. Rust has also made some progress with reborrowing, as issues like reborrowing in match arms which failed to compile in the past now work fine, without the need to force a move. The compiler can infer when it is necessary to move a &mut instead of reborrow it, but the logic behind this decision should be made clear to the programmer so that there is a better understanding of the final product.

It seems like there was some effort to document this topic (through non-lexical lifetimes) in #290. This documentation is unfinished, potentially due to the in-progress nature of non-lexical lifetimes, but the feature is stable, which means it could accompany a discussion of reborrowing. I think reborrowing is an important enough topic to get covered in some official documentation, and if not the book, then the reference should expand upon this technique.

[zancas]

It seems to me that a particularly important example of the type expression-annotated version of reborrowing occurs when the identifier being bound is declared as a function parameter, e.g.:

fn receive_refmut(y: &mut String) {
    dbg!(&y);
    y.push('*');
}
fn main() {
    let x = &mut String::from("x");
    receive_refmut(x);
    dbg!(x);
}

Since this issue is, As Far As I Can Tell, the best reference on the topic, I am adding my 0.02 ⓩ here.

[Subsentient]

Ugh, is there any documentation for this anywhere in the Reference? If so, can someone post a link?
A lack of explanation for language syntax is a very serious problem imho, and deeply underscores the need of Rust standardization.

[tlyu]

There's a lot of helpful information here https://rust-lang.github.io/rfcs/2094-nll.html#reborrow-constraints but NLL are apparently not formally stabilized yet, even though their implementation has been in stable for years. Hopefully we can add some of that material to the Reference anyway?

[tlyu]

This is one of the shortest examples I could write to illustrate the difference between a reborrow and a forbidden immutable borrow of a mutably borrowed value:

#![allow(unused)]
fn reborrow() {
    let mut x = 0;
    let mx = &mut x;
    let rx = &*mx; // reborrow
    println!("{:?}", rx); // this would implicitly reborrow if given `mx` instead
    *mx = 2; // use here so the lifetime includes the creation of `rx`
}
fn immutable_borrow() {
    let mut x = 0;
    let mx = &mut x;
    let rx = &x; // error: cannot borrow `x` as immutable because it is also borrowed as mutable
    // println!("{:?}", rx); // doesn't even need to be used to cause a compile error!
    *mx = 2; // use here so the lifetime includes the creation of `rx`
}

(playground)

compiler output:

error[E0502]: cannot borrow `x` as immutable because it is also borrowed as mutable
  --> src/lib.rs:12:14
   |
11 |     let mx = &mut x;
   |              ------ mutable borrow occurs here
12 |     let rx = &x; // error: cannot borrow `x` as immutable because it is also borrowed as mutable
   |              ^^ immutable borrow occurs here
13 |     // println!("{:?}", rx); // doesn't even need to be used to cause a compile error!
14 |     *mx = 2; // use here so the lifetime includes the creation of `rx`
   |     ------- mutable borrow later used here

[crlf0710]

Reborrow is actually covered as one specific kind of "coercion", and is already covered in the reference. It's indeed nice to expand and explain more clearly.

[Subsentient]

Reborrow is actually covered as one specific kind of "coercion", and is already covered in the reference. It's indeed nice to expand and explain more clearly.

A systems language like Rust should probably have all scenarios that cause implicit reborrows clearly defined. Right now, to find that out, I'd literally have to read the compiler source code. My first language was C, and the standard document to explain behavior was absolutely invaluable. I regret to say that learning Rust, occasionally the only way to find the info I need is some random guy on Stack Overflow. There are things that I want to know that I can't look up, where my mind just goes "well I bet the rustc guys don't even know either". I would really like to see this situation improved. Every other language I use, even tiny Lua, has adequate documentation there. Rust is the clear outlier.

Right now, the only way to find out what is legal Rust is by what the compiler accepts, which changes from time to time without warning.

[tlyu]

Reborrow is actually covered as one specific kind of "coercion", and is already covered in the reference. It's indeed nice to expand and explain more clearly.

I guess I could look at it sideways and kind of read the type coercions section like that… but I think it's missing quite a few things, like how should I expect coercion to work with non-Copy types? (which leads into how it would work on mutable references, which are not Copy)

Also, reborrowing adds lifetime constraints that amount to extending the original borrow as well as temporarily invalidating the reference being reborrowed. (I think it counts as a "borrow" of the reborrowed reference, even though the reborrow isn't a pointer to the reborrowed reference?) Nothing in the Reference explains how that works.

(edit: also, as @soulwa mentioned above, the only place any form of the word "reborrow" occurs (that I could find) is in the subsection about raw pointers. I could find nothing about reborrowing mutable references as another kind of reference.)

[RalfJung]

but I think it's missing quite a few things, like how should I expect coercion to work with non-Copy types?

The term Copy does not appear on https://doc.rust-lang.org/nightly/reference/type-coercions.html. Why would you think coercions work any different for Copy types than they do for non-Copy types?
They work exactly the same way in both cases.

[tlyu]

but I think it's missing quite a few things, like how should I expect coercion to work with non-Copy types?

The term Copy does not appear on https://doc.rust-lang.org/nightly/reference/type-coercions.html. Why would you think coercions work any different for Copy types than they do for non-Copy types?
They work exactly the same way in both cases.

I think with non-Copy types there's the issue of permanently losing information if the coercion is lossy, and I think there's no guarantee of coercions being non-lossy. (Though I guess this is more of a possibly non-obvious adverse consequence and not necessarily "how coercions work".)

(edit: Thinking about this some more, the non-Copy nature of mutable references makes reborrowing a necessity for their ergonomic use. This might be obvious in retrospect, but a beginner shouldn't have to deduce the existence of reborrows from the existence of &mut T -> &T coercion, the non-Copy nature of mutable references, and the observed fact that mutable references (sometimes) continue to be usable after passing them to functions that take shared references.)

[tlyu]

A systems language like Rust should probably have all scenarios that cause implicit reborrows clearly defined.

I think implicit borrows are fairly well defined. It might not be well-defined how they can be reborrows, and the consequences of that.

Right now, to find that out, I'd literally have to read the compiler source code. My first language was C, and the standard document to explain behavior was absolutely invaluable. I regret to say that learning Rust, occasionally the only way to find the info I need is some random guy on Stack Overflow. There are things that I want to know that I can't look up, where my mind just goes "well I bet the rustc guys don't even know either". I would really like to see this situation improved. Every other language I use, even tiny Lua, has adequate documentation there. Rust is the clear outlier.

I share a lot of the same frustrations. I see lots of features getting stabilized with inadequate updates to documentation. It seems to be a pattern. I know it's exciting to implement new stuff. It also makes Rust less approachable to beginners if it contains a lot of "magic" that isn't easily discovered or explained. Sometimes it's only explained on a Stack Overflow thread, or some blog post. If I'm lucky, I can find an RFC with a tracking issue, and maybe the tracking issue will be closed with an unchecked task linking to another issue, untouched for years, for updating the documentation.

[Subsentient]

It also makes Rust less approachable to beginners if it contains a lot of "magic" that isn't easily discovered or explained.
Sometimes it's only explained on a Stack Overflow thread, or some blog post. If I'm lucky, I can find an RFC with a tracking issue, and maybe the tracking issue will be closed with an unchecked task linking to another issue, untouched for years, for updating the documentation.

I'm glad to see I'm not entirely insane. :^)
Rust has often infuriated me in this respect, I've had more strong emotions drawn out of me by Rust than any language I can remember, and I've used a lot.
To me, my mantra is "understand, then use", and it has been since I started programming.
Rust is essentially forcing me to break that habit, by forcing me to accept a great deal of random magic with no explanation or documentation, and to say that sits poorly with me is an understatement. Take C++, it's a homemade bomb of a language, but at the end of the day, every behavior is documented, every conversion pattern laid out in excruciating detail, every coercion handled by the standard document. I continue to use Rust because I find the memory safety of the language invaluable, but sadly it is still in my mind a very immature language, in every meaning of the word.

This is why I believe an ISO standard for Rust is absolutely imperative, even if the reference implementation continues to advance. Rust has made it abundantly clear that it needs some outside force or pact to constrain it from becoming a pile of "magic" and unicorn farts, much like Go has been since its inception.

[tlyu]

One place where Rust's magic might be adequately documented is method call coercions: the Reference has a fairly complete list of transformation of the receiver types, and any surprising behavior that I've discovered in that area has been logically consistent with those rules, in retrospect. I also think that some of the less expected consequences of that magic could stand to use a little more highlighting.

[dlukes]

Rust has often infuriated me in this respect, I've had more strong emotions drawn out of me by Rust than any language I can remember, and I've used a lot.

I can relate to that :) Rust has taught me a lot, and I love it for that. But it has also made some of these learnings incredibly hard-won, because you have to tear through that magic-coated facade.

It's a double-edged sword -- on the one hand, without these tricks that improve ergonomics, Rust would probably be much more tedious to use day-to-day. On the other, when so much is hidden under the surface, you can easily build inaccurate generalizations that come to bite you when things go awry. Because at that point, you can't be blissfully oblivious of what's going on under the hood anymore, because the diagnostics that you get are often only actionable if you have at least a vague idea of what your code actually boils down to.

Humans are great at learning from patterns, but the only patterns you can learn from are patterns that you actually see. So my wish is not really for an exhaustive spec (although it certainly wouldn't hurt), but rather that Rust would emphasize and ship with tools that teach you to peek under the magic veil. Something like cargo-inspect for desugaring, but more robust and built in. Maybe a tool to visualize materialized lifetimes. Maybe optional hints from LSP servers pointing out places where potentially tricky magic is going on.

One good example of the double-edged sword dynamics is type inference: not having to type the types is great, but that also means not seeing them, which is less so. This has been especially confusing for me in the case of reference types -- with auto-deref thrown into the mix, I've learnt to fix issues by trial and error and/or through a conversation with the compiler, but intuition on how to get these situations right on the first try has been much slower to build. I hope it'll get better once I finally figure out how to enable rust-analyzer's inlay type-hints in Neovim, which feels like a great solution to this problem -- you don't have to type the types, but you can still see them.

Maybe this would be a good theme for the 2024 edition -- lifting the veil :)

[madsmtm]

Not sure if this is the right place to note this, but something that confused me is how a method call reborrows, while using fully qualified syntax doesn't.

The documentation on method-call expressions does say "When looking up a method call, the receiver may be automatically dereferenced or borrowed in order to call a method", but to me it's not immediately obvious that that includes a reborrow.

Example (playground link).

struct X;

trait Foo: Sized {
    fn foo(self) {}
}

impl Foo for &mut X {}

fn main() {
    let x: &mut X = &mut X;
    x.foo();
    x.foo(); // Reborrows x

    let x: &mut X = &mut X;
    Foo::foo(x);
    Foo::foo(x); // Doesn't reborrow x, fails to compile
}

[Subsentient]

something that confused me is how a method call reborrows, while using fully qualified syntax doesn't.

I'm afraid it's unlikely that documentation alone will fix this. From what I can see personally, Rust designed these borrow features for convenience, with no regard to consistency of behavior or ease of documentability -- I suspect that's why this issue has had no progress -- there is nothing useful to document other than compiler magic that works in some cases and not in others.

I fear a reckoning will come for Rustaceans where Rust will be forced to break a huge swath of codebases while it fixes language inconsistencies in a way that makes sense to document. I only hope that the Rust developers actually learn from it. Consistency first.

Perhaps the core team's hatred of completely independent alternative rustc implementations partly derives from the fact that Rust is poorly documented and the syntax is in places determined by compiler internals rather than an actual codified language dialect.

I think it's really, really bad when a language relies on compiler logic that even the devs don't know about instead of trying to conform to some kind of syntax or specification, and I'm still appalled at the lack of progress in correcting it. It's a difficult issue, to be sure, but it's a self-inflicted wound that should have been foreseen. Rust must now bathe in its own shame, acknowledge the mistakes, and correct them. Unfortunately, users do and probably will continue to pay for these oversights.

[SteveLauC]

One year ago, I found the weird behavior of this 'reborrow' stuff, I googled a lot and found this issue.
And now, I am stuck with match ergonomics. Just like one year ago, I don't know where to seek help, some random stack overflow guy will help if I get lucky.

Sometimes it's only explained on a Stack Overflow thread, or some blog post. If I'm lucky, I can find an RFC with a tracking issue, and maybe the tracking issue will be closed with an unchecked task linking to another issue, untouched for years, for updating the documentation.
I think it's really, really bad when a language relies on compiler logic that even the devs don't know about instead of trying to conform to some kind of syntax or specification

I can not agree more, they are so true🤕

[vagueanxiety]

To give some perspective, I started learning Rust a month ago (and it's been a great experience!), and I got confused by this re-borrow logic. After a bit of searching and reading stackoverflow (1, 2), I arrived here.

I was confused mostly because: 1) reborrowing of mutable reference is often implicit 2) why doesn't this kind of reborrowing violate the "no multiple mutable references" rule in the book. Luckily, the stackoverflow posts I found explained both quite well.

Even as a Rust newbie, I can relate to the comments above about compiler magic and language consistency. For example, another thing I found confusing while learning iterators was autoderef and autoref in operators, which is already being worked on.

Not sure if this is constructive, but I thought it might help other people and move this issue a little bit forward.

[Subsentient]

Even as a Rust newbie, I can relate to the comments above about compiler magic and language consistency.
Not sure if this is constructive, but I thought it might help other people and move this issue a little bit forward.

Concurring voices are helpful in drawing attention to the issue, in hopes that the attention will lead to a greater desire to fix this.
This is a serious consistency issue. This issue should not be sitting unaddressed for this long. I don't think I'm alone here. I want to see it fixed, because for all its flaws, I do like Rust, and I want to see it actually rise to the level of usefulness as C and C++. That won't happen as long as Rust can't even document, much less standardize, its own syntax and semantics.
If you need to use Stack Overflow to find crucial language information, you have already failed. What's worse, the inconsistent manifestation of this magic means that no Stack Overflow answers are likely to contain a full, accurate picture of real-world rustc behavior. This needs to be fixed.

My suggestion is:

1. Update rustc to apply this magic everywhere it is possible to automatically detect it is needed. (it is easier to widen permitted code than to break existing code)
2. Determine the rules and process clearly and comprehensively that rustc uses for reborrow magic.
3. Document this extensively.

[santhosh-tekuri]

fn main() {
    let mut arr = [10, 7, 8, 9, 1, 5];
    let r1 = &mut arr;
    let r2: &mut [i32; 6] = r1;
    test(r1);
    test(r2);
}

fn test(r: &mut [i32; 6]) {}

gives following error:

error[E0499]: cannot borrow `*r1` as mutable more than once at a time
 --> src/main.rs:5:10
  |
4 |     let r2: &mut [i32; 6] = r1;
  |                             -- first mutable borrow occurs here
5 |     test(r1);
  |          ^^ second mutable borrow occurs here
6 |     test(r2);
  |          -- first borrow later used here

For more information about this error, try `rustc --explain E0499`.

as a beginner, the code seems that I am borrowing arr as mutable 3 times.
the compiler says line 4 is first borrow, but it seems for me second borrow.

[gabyx]

I also started learning Rust a month ago, and can only really agree to all what is said here. Rust is really great with its unique compile-time reference counting mechanics.

The reborrowing stuff really quite confused me to the gratest and even by reading this issue I feel a bit sad that the Rust development kept slacking about these details.

I understand that documenting difficult language features is a hard abstraction problem when condensing into easier words for beginners and even experts. Explaining is hard. Even harder if the details are only clear to some compiler developers by looking at rustc. Also its hard not going to end up in a language-lawyer-bible as in the C++ community where I come from which is probably the opposite what would benefit the community. I feel that the tools at hand in VS Code rust-analyzer etc already are absolutely great tools to help in understanding what the compiler does. I can only dream of a tool which will let me inspect the lifetimes and other stuff like reborrowing or conversions etc.
This is probably of much greater value then strive for direct standardization.

Documenting a langauge is essentially an educational task in computer science and should be treated exactly that way. So looking at the Rust book and all hard work which has been done: This manual is by far better than anything I've come across when learning C++ long time ago with the C++ reference at hand, because the Rust book is a learning manual. It is already quite great and should be extended to include such weird topics. Standardization is great, but probably easier when the stabilization part is a bit over.

[Kwarrtz]

I appreciate all the general discussion on inconsistency and lack of standardization in Rust, but I feel like it may have distracted somewhat from the fact that this issue was opened almost three years ago and a feature which is core to the everyday use of the language and a common point of confusion for beginners is still not documented anywhere in any of the three official sources for the language (the Book, the Reference or the Nomicon).

The broader issues are extremely challenging to tackle, but that specific oversight is eminently fixable. If I understand the reborrowing semantics correctly, then while they're highly non-intuitive, they shouldn't be difficult to at least write down somewhere in the Reference. Even I'm wrong about that, a short section saying "this exists but the exact semantics are too complex to be rigorously described" would be a massive improvement over the status quo! What is needed for that to actually be done?

[bjorn3]

I can only dream of a tool which will let me inspect the lifetimes and other stuff like reborrowing or conversions etc.

That is rust-analyzer :) Try setting the rust-analyzer.inlayHints.expressionAdjustmentHints.enable option. There is also rust-analyzer.inlayHints.lifetimeElisionHints.enable for lifetime elisions.

[zjp-CN]

What is needed for that to actually be done?

AFAIK, reborrowing won't be officially documented until Chalk is adopted. rust-lang/rfcs#2364 (comment)

An old description I found lies in Niko's blog:

One of the less obvious but more important coercions is what I call reborrowing, though it's really a special case of autoborrow. The idea here is that when we see a parameter of type &'a T or &'a mut T we always "reborrow" it, effectively converting to &'b T or &'b mut T. While both are borrowed pointers, the reborrowed version has a different (generally shorter) lifetime.

Now UCG mentions the reborrowing with stack borrows.

[Kwarrtz]

Waiting to rigorously formalize reborrow semantics until Chalk is adopted is one thing, but I don't see the harm of at least mentioning them as they stand in the Reference now, with a caveat similar to the one in the section on coercions that the description is informal. Adopting Chalk may be the long-term fix, but it isn't going to happen overnight (the comment by Niko you linked is 5 years old!) and people continue to learn and use Rust in the meantime. It would be one thing if this were an esoteric feature only of interest to those curious about what's going on under the hood, but implicit reborrowing is ubiquitous, and it means that the description of Rust's move semantics given in every official source is insufficient to explain even incredibly common patterns. That seems like something that should be fixed as soon as possible, even if imperfectly.

[Kwarrtz]

Actually, isn't chalk basically dead anyway? Or at least the possibility of it being incorporated into rustc.

[bjorn3]

Indeed. Work is on the way for another trait solver to replace both the existing trait solver of rustc and chalk: rust-lang/rust#105661

[zjp-CN]

Actually, isn't chalk basically dead anyway?

The link is dead.

Just come here to yell out Chalk is not dead after I saw this recent quote somewhere

Chalk is ... being actively developed

[fmease]

Nope, Chalk is dead (rust-lang/rust#113303). Its replacement is the “next gen trait solver”.

The full quote is as follows (with context added):

Given that chalk is not going to be merged used as the next trait solver, maybe this RFC can be revived?

Chalk is really a shorthand here for "next gen trait solver", which is being actively developed

Niko's sentence means that you should replace the word “Chalk” with “next gen trait solver” in your mind when reading rust-lang/rfcs#1672 (comment). Lastly, “actively developed” refers to the next trait solver, not Chalk.

[btj]

Linking this issue to the trait solver work does not make sense to me. The trait solver work is relevant only to the issue of reborrowing for derived types? But the present issue is about plain old reborrowing of references, right?

Personally, I'd like to see some documentation on where reborrowing happens because it affects the aliasing restrictions, i.e. it affects whether code is UB or not!