ranged-based for for user-defined types

[Pixep]

The goal of this proposal is to provide a way for user-defined types to support range-based iteration with for.
The current proposed solution exposes 3 interfaces that can be implemented by user types to enable support for
ranged-for loops.

Co-authored-by: josh11b josh11b@users.noreply.github.com

[zygoloid]

Thanks, this is looking promising!

I think we should have at least a sketch of a plan for mutability in this proposal. Given that our main way of providing mutable handles to objects is via pointers, I think we'd want some way to convert a container into a range of pointers to container elements. I think that can be done without any changes to this proposal, for example:

fn F() {
  var v: Array(i32, 5) = ...;
  for (p: i32* in v.MutableElements()) {
    // ...
  }
}

where MutableElements produces an Iterable where .ElementType == .Self.ElementType*. (Maybe we'd want a different interface for converting the container to a sequence of element pointers, but we don't need to decide on that here.)

[geoffromer]

It seems like it could also have an interoperability and migration cost, because Carbon ranged-for loops won't work with C++ containers, and vice-versa.

[jonmeow]

One option for addressing interoperability without binding ourselves to the C++ API would be to have the interop layer automatically inject adopters for language-conforming iterators. e.g., a Carbon Iterable could get adapted to look like a C++ iterator for C++ code, whereas a C++ iterator could have an external impl defined for Carbon code (and likely should need that regardless), letting each language keep its own idioms.

(it may make sense to mark interop as an open question since it's not designed out?)

[geoffromer]

I think it will be challenging to bridge the gap between these two APIs with adapters, because there's a substantial "impedance mismatch" between them that's hard to hide.

• With this design, a Carbon iterator basically plays the same role as a begin/end iterator pair in C++, so there isn't a one-to-one relationship between the objects on either side of the adaptation. That will be a particular problem for automatic adaptation, but may also cause problems for manual adaptation.
• A C++ forward iterator (or better) must support calling operator* multiple times, returning the same result so long as the iterator is not advanced. To satisfy that requirement, an adaptor would have to cache the value returned from Next.
• C++ iterators support mutation via operator*. That compounds the previous problem, because mutation has to apply to the original element, not a cached copy.
• We could avoid those last two problems by saying that Carbon iterators are exposed in C++ as input iterators, but that excludes mutating use cases, and could dramatically degrade performance even for read-only cases, because many algorithms can't be implemented efficiently (or at all) in terms of the input iterator API.

[Pixep]

The last points definitely highlight the need for a Get function, in place of a simpler Next-only approach. I propose to move in that direction.

Regarding the iterator pairs, it is an option to expose an way to get an end iterator for comparison purposes, and interoperability. From a user standpoint, I would probably prefer to not be forced to rely on this end operator, which can be akward at times (needing the container in scope and such), but as a additional feature I believe it makes sense, and likely to have other uses (reverse and such).

Edit: Proposal amended with Get for Iterator.

[jonmeow]

One option for addressing interoperability without binding ourselves to the C++ API would be to have the interop layer automatically inject adopters for language-conforming iterators. e.g., a Carbon Iterable could get adapted to look like a C++ iterator for C++ code, whereas a C++ iterator could have an external impl defined for Carbon code (and likely should need that regardless), letting each language keep its own idioms.

(it may make sense to mark interop as an open question since it's not designed out?)

[geoffromer]

I think it will be challenging to bridge the gap between these two APIs with adapters, because there's a substantial "impedance mismatch" between them that's hard to hide.

• With this design, a Carbon iterator basically plays the same role as a begin/end iterator pair in C++, so there isn't a one-to-one relationship between the objects on either side of the adaptation. That will be a particular problem for automatic adaptation, but may also cause problems for manual adaptation.
• A C++ forward iterator (or better) must support calling operator* multiple times, returning the same result so long as the iterator is not advanced. To satisfy that requirement, an adaptor would have to cache the value returned from Next.
• C++ iterators support mutation via operator*. That compounds the previous problem, because mutation has to apply to the original element, not a cached copy.
• We could avoid those last two problems by saying that Carbon iterators are exposed in C++ as input iterators, but that excludes mutating use cases, and could dramatically degrade performance even for read-only cases, because many algorithms can't be implemented efficiently (or at all) in terms of the input iterator API.

[geoffromer]

It's probably worth spelling out the difference between "cursor" and "iterator"; I'm not sure everyone is familiar with that terminology.

[geoffromer]

We should probably also mention language support for const, and language support for user-defined l-value expressions, because those will determine how iterators should support mutability.

[josh11b]

Two things:

• Something to connect these two requirements to make it clear that the type returned by begin() and end() is what has to support these operators.
• I believe the iterator type must also support inequality comparison (at least). According to https://en.cppreference.com/w/cpp/language/range-for a C++ range-based for loop gets rewritten to include for ( ; __begin != __end; ++__begin).

We may need to be careful about closely matching C++ so we can be as compatible as we can manage, as long as it doesn't impact the usability for ordinary users. In C++, my understanding is as long as the rewrite type checks, the ranged-based for loop is legal, like a template. This means we may need to support begin() and end() returning different types, as long as != is defined on those two types.

[Pixep]

Absolutely yes, clarified.

[josh11b]

What is Iterable<...> here? It looks like IterateIterator expects to be parameterized by a container type. Is Iterable some sort of adapter to define the CursorType and ElementType members? I don't see it defined anywhere else in this file.

[Pixep]

That's a typo, and should be Iterate<C, T>, which is the C++ equivalent to Carbon's type. Fixed.

[geoffromer]

I think this will still work if ElementType is move-only (but see the comment below), it just means that IterateIterator will be move-only in those cases. That means it won't be a valid iterator according to the standard library, but it should still work with range-based for loops.

[Pixep]

I just looked back at the code:

     if (const auto v = container_->Next(*cursor_); v) {
      value_ = *std::move(v);
    }

Won't Next() require ElementType to be copyable nonetheless, as we are effectively returning a copy of an element, and not moving anything from the container? Even if we avoid an optional-like wrapper, or just move from it, the ElementType still has to be copied at some point (likely withing Next()).

Maybe I forgot something between now and then.

[geoffromer]

In recent versions of C++, if Next returns an ElementType rather than a reference, the code you quote won't actually copy or move the return value; it will use v itself as the storage for the returned temporary. Now, you're right that in this context, the body of Next will very likely need to make a copy, but Next is written in pure Carbon, so that problem isn't caused by caching in the interoperability layer, and in fact it's not caused by the interop layer at all -- the same problem exists in pure Carbon.

[Pixep]

Agree that this is by design, and not caused by the interop. To expand, I think we could say "move-only" in the C++ domain, most that would be ignoring a very likely "copiable" requirement on Carbon side / Next. So all-in-all, "copyable" seem more correct, and the upper bound of reqs, vs the lower bound.

[geoffromer]

The problem is that this doc doesn't explicitly talk about copyability on the Carbon side, so by introducing it here as a new topic that's specifically tied to caching in the interop layer, it really makes it sound like native Carbon will not require copyability, but C++ interop will.

I'd suggest revising the "Large elements and Optional" section to explicitly talk about the fact that this design may force Next to make a copy, and hence require ElementType to be copyable. Then down here you can say that the caching requires ElementType to be movable, and note that this won't matter in practice if Carbon requires it to be copyable anyway.

[josh11b]

I think this is about ready for leads review

[josh11b]

@chandlerc I feel like we had a conversation about this and r-value representations. Could you recall where we ended up with this?

[josh11b]

I think this is about ready for leads review

[zygoloid]

I'm happy with this. It seems likely to me that we'll want to revisit some aspects of this eventually, such as ergonomic improvements for mutable iteration, but this seems like a good foundation to use as the basis for further exploration.

[zygoloid]

Also approving on behalf of leads. Thank you for working on this!