Return position impl Trait in traits

[tmandry]

• Permit impl Trait in fn return position within traits and trait impls.
• Allow async fn in traits and trait impls to be used interchangeably with its equivalent impl Trait desugaring.
• Allow trait impls to #[refine] an impl Trait return type with added bounds or a concrete type.

Background

This RFC is a collaboration between myself and @compiler-errors, and is based on an earlier RFC by @nikomatsakis.

The primary changes from that RFC are:

• async fn is now allowed to be used interchangeably with its equivalent impl Trait desugaring.
• #[refine] is now included as a way to add more information about the returned type in an impl, either by adding bounds to the impl Trait type, or by using a concrete type. Refined trait implementations #3245 did not exist at the time of the previous RFC (return position impl trait in traits #3193), and was added partly in response to discussion there.
• You are now allowed to use dyn Trait on a trait with a method returning impl Trait, as long as that method has a where Self: Sized bound.

Rendered

[nikomatsakis]

I don't think this should have a T-types FCP for the feature (perhaps for stabilization).

@rustbot label -T-types

[nikomatsakis]

@rfcbot fcp merge

This feature has been a long time coming and feels like a no brainer to me. I'm going to start the merge proceedings.

[rfcbot]

Team member @nikomatsakis has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm
• @tmandry

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[spastorino]

There's this ...

XXX document:

    - you can capture type parameters
    - but not lifetimes, unless they appear in the bounds

in the explainer. It would be nice to explain a bit better that + what happens if all the lifetimes are captured as we do with types and consts too.

[lcnr]

Wrote this down after midnight so it's not as clear and concise as I would have liked. I only really thought about this in-depth after Niko proposed the FCP.

tl;dr: RPITIT is not a trivial extension of TAIT or RPIT and does not simply fall out of the desire for a consistent design. As I stated separately, I don't think we should add return type notation RTN. This causes RPITIT to be a major footgun for library authors without a significant benefit.

I very much agree that we should add support to async fn in traits but am not yet sold that RPITIT is the right approach for that.

RPITITs are not simple opaque types

moved to a hackmd as it is somewhat pedantic

I don't agree that RPITIT will result in a more consistent language given that the semantics of RPITIT are significantly different from RPIT. It is easy to provide context to the error message when trying to use it, recommending an associated type with TAIT.

RPITIT can result in worse library design

To keep this short, I prefer https://github.com/tmandry/rfcs/blob/rpitit/text/0000-return-position-impl-trait-in-traits.md#what-about-using-an-explicit-associated-type over RPITIT and don't think we should add RPITIT (at least for now).

The RFC uses IntoIterator as a good example for where RPITIT would be applicable. Searching for ::IntoIter: on github results in a non-negligible amount of crates which use bounds on IntoIter. Using RPITIT without a way to name the anonymous projection would have prevented this code. Given that I fairly strongly dislike RTN, this is a major issue with RPITIT imo.

Going through the list of cases where an explicit associated type should be used, we can reverse it to get requirements to correctly use RPITIT:

• it is irrelevant whether users provide concrete types as you can't expect them to always add #[refine]
• no user will ever need to constrain the impl Trait to a concrete value: also not the case for IntoIterator
• no user will ever need to add additional bounds for the impl Trait

We have proof that the second and third point do not hold for IntoIterator and I am certain that the first point also does not hold. I don't think RPITIT would be sensible for IntoIterator.

More generally, I cannot think of a trait where RPITIT would be appropriate given these requirements, unless the trait is local to your library or sealed. I assume that given the ability, library users will sometimes end up using RPITIT simply because it is possible, which will then require breaking changes to fix later on.

RPITIT is not necessary for async functions in traits

By going with the "explicit associated type" route, we can instead add some attribute to express the concept of "this named associated type may be implicitly defined by this function", something like

trait IntoIterator {
    type Item;
    type IntoIter: Iterator<Item = Self::Item>;
    #[may_implicitly_define(IntoIter)]
    fn into_iter(self) -> Self::IntoIter;
}

impl<T: Iterator> IntoIterator for T {
    type Item = T::Item;
    // does not explicitly define `IntoIter`,
    fn into_iter(self) -> Self::IntoIter {
        self
    }
    // or
    fn into_iter(self) -> impl Iterator<Item = T::Item> {
        self
    }
} 

A similar approach can then be used for async functions as well. I don't yet know how exactly these attributes should look but I do prefer such attributes over RPITIT.

RPITIT is at most a linear ergonomics improvement

As stated in the RFC, RPITIT should be equivalent to its desugaring using TAIT and associated types. This desugaring has some repetition but it should pretty much never significantly impact development speed or readability.

Given the fairly mechanical desugaring, we should be able to emulate RPITIT using TAIT and a set of macros for trait definitions and impls. If the manual desugaring poses a major issue, users should be drawn to such a set of macros, at which point we can reconsider adding RPITIT.

[comex]

I agree with @lcnr that this feature effectively requires return type notation. In some hypothetical future where Rust decided against adding return type notation, RPITIT would clearly not hold its weight. Therefore it does seem strange to be landing it without at least talking about return type notation.

Unlike @lcnr, I have no complaints about return type notation; I think it sounds great. But I am confused about the details. Is it going to be () or (..)? (The implementation PR suggests it might switch over to (..) in the future, but today's blog post uses ().) And since neither of those syntaxes actually lists the arguments, how does that interact with generic functions, where the return type might conditionally implement a trait depending on the argument types?

[dhardy]

We expect to introduce a mechanism for naming the result of -> impl Trait return types in a follow-up RFC.

Is deferring this acceptable? It seems like it should be a core part of the proposal. Then again, we still have no syntax for naming the return type of a closure or a regular impl Trait method, so perhaps not.

Should we at least have somewhere to discuss relevant issues:

• Syntax for naming the return type of RPITIT (or something more general like return_type_of<F>)
• Associated type aliases (which are not associated type defaults)

[dhardy]

An alternative approach:

trait T {
    auto type Fut = return_type_of<Self::fun>;

    async fn fun(&self) -> i32;
}

auto type defines an associated type, except it cannot be defined at the impl site. (Optionally, we could allow impl-site definitions that correlate exactly with the inferred type in order to allow migration from assoc. types to assoc. auto types, and just warn that the definition is redundant.)

async fn fun (and any other usage of RPIT in a trait method) could be legal only given a corresponding auto type. By making a corresponding auto type required, we (a) always have a named type for usage in bounds, field types of generic structs, etc., and (b) are more transparent about why dyn Trait is not valid (at least not without constraining the assoc. auto type, if that is even possible).

Putting both of these together would allow e.g. migrating ImplIterator::IntoIter into an auto type and into_iter into an RPITIT method.

---

Addendum: but is an auto type just a type alias? The key question is whether a single auto type can be used by multiple RPITIT methods, and with the above syntax the answer is clearly no (it's also likely not useful).

So the above is just:

• associated type aliases (a new feature, which could be more general than this)
• a suggestion that we require an assoc. type alias for RPITIT
• an optional compatibility rule allowing migration to RPITIT
• return_type_of syntax or something (specific to this use-case or more generally applicable)

[kpreid]

As a user of stable Rust, I'd like to see effort put first into stabilization of TAIT, which would enable many of the use cases that RPITIT would, and also provide more information about what benefits RPITIT would or would not provide and how it should be designed, since it results in a smaller delta (“write this trait using an associated type” vs. “can't practically write this trait at all”).

[eholk]

One consequence of this is that it basically stabilizes the impl Future desugaring. I don't imagine we'll want to change it, and enough things depend on the impl Future desugaring now that we probably can't change it practically already, but it still means we're committing to what in some ways feel like internal implementation details to me.

[eholk]

Presumably the reason for doing this is so that we can use #[refine] to make stricter promises on the impl? For example, adding a Send bound at the impl level.

[tmandry]

That's one (another example of an added promise would be that you don't capture self). A whole other reason would be to include code that runs at call time instead of during poll.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[aliemjay]

I'm concerned about an inconsistency with inherent impls regarding the capture rules for lifetimes. It can be shown in this example:

struct Foo<'a>(&'a str);

// ERROR: `impl Sized` doesn't capture 'a
impl<'a> Foo<'a> {
    fn into(self) -> impl Sized { self }
}

trait IntoImpl {
    fn into(self) -> impl Sized;
}

// OK! Allowed by the RFC.
impl<'a> IntoImpl for Foo<'a> {
    fn into(self) -> impl Sized { self }
}

We say that a generic parameter is captured if it may appear in the hidden type. These rules are the same as those for -> impl Trait in inherent impls.

Given the difference above, this statement is no longer true.

Additionally, in the capture rules, the text talks only about generic parameters of the trait implementation - it says nothing about the trait definition. It is not clear to me, for example, whether impl Sized captures the lifetime 'a here:

trait Trait<'a> {
    fn test() -> impl Sized;
}

[tmandry]

@aliemjay Oh, I think that text should have been removed. I've reorganized the sections related to scoping rules and removed the rules that were incorrect. RPITIT should now be consistent with RPIT in inherent methods.

I also noticed there's an inconsistency in how the implementation handles this; see rust-lang/rust#112194.

Additionally, in the capture rules, the text talks only about generic parameters of the trait implementation - it says nothing about the trait definition.

We are talking about the hidden type supplied by each implementation. The type supplied by the implementation can't name the parameters of the trait itself because those aren't in scope – instead, usually you would have a generic parameter of the impl that corresponds to each trait parameter. (I realize this is a bit subtle, but in the reference-level sections it helps to be pedantic.)

trait Trait<'a> {
    fn test() -> impl Sized;
}

In this example the return type is not allowed to capture 'a because it does not appear in impl Sized. On the other hand, if there were a type parameter in scope, the concrete type returned by our impl could capture that type parameter (and therefore would refer to any lifetimes the actual type for that parameter includes).

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

[aliemjay]

It is not clear to me, for example, whether impl Sized captures the lifetime 'a here:

trait Trait<'a> {
    fn test() -> impl Sized;
}

The decision here doesn't seem to be straightforward...

If it captures the lifetime 'a, we would have an inconsistency with inherent impls. If it doesn't, we should treat the types of <T as Trait<'a>>::test() and <T as Trait<'b>>::test() to be equal for two different lifetimes. This might seem reasonable but it contradicts with the desugaring specified in this RFC because projection types are generic over all trait inputs, i.e. <T as Trait<'a>>::Opaque#0 and <T as Trait<'b>>::Opaque#0 are not equal types unless 'a == 'b. For a concrete example, see how this example fails with the current implementation:

#![feature(return_position_impl_trait_in_trait)]

trait Trait<'a> {
    fn test() -> impl Sized;
}

fn test<'a, 'b, T: for<'x> Trait<'x>>() {
    let mut x;
    x = <T as Trait<'a>>::test();
    x = <T as Trait<'b>>::test();
    //^~ ERROR `'a` and `'b` must be the same: replace one with the other
}

Consequently we should change the desugaring to involve a supertrait that does not have the lifetime 'a in its parameters:

trait SuperTrait {
    type AnonOpaque;
}

trait Trait<'a>: SuperTrait {
    fn test() -> Self::AnonOpaque;
}

[aliemjay]

Default trait functions

impl Trait in trait functions with default body is underspecified. Is it allowed? If yes, we should specify the desugaring. It is not trivial as it depends on the feature of associated type defaults. We should also mention the drawback of not being able to apply #[refine] to it.

Legal positions for impl Trait in trait impls

The text specifies the legal positions in trait definitions but leaves trait implementations unspecified. Do we allow this?:

trait Trait {
    fn test() -> impl Sized;
}

impl Trait for () {
    fn test() -> Result<impl Sized, impl Sized> { Ok::<(), ()>(()) }
}

Forward compatiblity with named return types

Should we limit the legal positions for impl Trait to be maximally compatible with the naming schemes for return types?

fn f() -> impl Sized; // Ok.
fn f() -> Box<impl Sized>; // Not ok; cannot be named.
fn f() -> impl Iterator<Item = impl Sized>; // Ok.
fn f() -> impl Iterator<Item = Box<impl Sized>>; // Not ok; cannot name `impl Sized`.

[aliemjay]

This is an ambiguous method resolution with candidates from IntoIterator and NewIntoIterator.
There are multiple similar issues below.

[aliemjay]

Can you elaborate further? I'm not sure if I understand this correctly. Does that mean we allow impl Sized in the implementation to capture 'a because it captures the type parameter Self in the trait definition?

trait Trait {
    fn foo(self) -> impl Sized;
}
impl<'a> Trait for &'a u8 {
    fn foo(self) -> impl Sized { self } // Is this ok?
}

[tmandry]

The intent is to be consistent with inherent methods, but the implementation is not currently. I believe this is captured in the unresolved question around rust-lang/rust#112194.

[tmandry]

impl Trait in trait functions with default body is underspecified. Is it allowed? If yes, we should specify the desugaring. It is not trivial as it depends on the feature of associated type defaults. We should also mention the drawback of not being able to apply #[refine] to it.

Yes it is; I've updated the text to clarify that. I don't think there are any nonobvious interactions with associated type defaults, given that the types cannot be named in the first place.

Not being able to apply #[refine] is an existing drawback of that feature, not something specific to this RFC.

The text specifies the legal positions in trait definitions but leaves trait implementations unspecified. Do we allow this?

Elaborated what's allowed in the RFC. The short answer is no, but it might be an interesting future possibility.

Should we limit the legal positions for impl Trait to be maximally compatible with the naming schemes for return types?

My initial thought to this was no, but I think it's worth pondering. Added this as an unresolved question.

The decision here doesn't seem to be straightforward...

I think the decision is captured as an unresolved question around rust-lang/rust#112194. I updated the RFC text to point to your comment as well.

[nikomatsakis]

Huzzah! The @rust-lang/lang team has decided to accept this RFC.

Tracking issue: rust-lang/rust#91611

If you'd like to keep following the development of this feature, please subscribe to that issue, thanks! :)

[nikomatsakis]

(@aliemjay -- my sense was that the specific points you raised were (a) important details to be resolved but not reasons not to accept the RFC and (b) either captured in unresolved questions or clarified; if you feel otherwise, please follow-up either on tracking issue or zulip)

[tmandry]

Hm, it seems like the tracking issue is shared with async fn in traits. Should we make a separate tracking issue for RPITIT?