Discussion: ref on foreach variable in case of an array

[alrz]

Moved from dotnet/roslyn#16155

foreach (ref var item in array)
{
}

->

for (int i = 0; i < array.Length; ++i)
{
    readonly ref var item = ref array[i];
}

Note: The iterator variable is not and can't beref or ref readonly (a ref to something you can't write), i.e. we can't do item = ref something; but one should be able to explicitly make it readonly ref readonly (a ref that you can't reassign to something you can't write). By default, it's readonly ref (a ref that you can't reassign) so that all restrictions on ref and iterator variables apply, e.g. they can't be captured or reassigned.

Open question: can we allow this for other types via duck typing?

Rel: #38, #188

[gafter]

/cc @VSadov

[svick]

Also related: IRefEnumerable https://github.com/dotnet/roslyn/issues/14230.

In case extending this to non-array types is desired, what's the advantage of translating ref foreach into ref indexers instead of ref enumerables?

[VSadov]

Both ref and ref readonly iteration variables could make sense
The difference would be that ref readonly variable is not writeable.

NOTE: the ref variables themselves in either case should not be re-bindable. I.E. cant do item = ref something inside the block.
That is the same behavior as with ref parameters and ref locals which can be bound to a location only once. Even if that gets relaxed for parameters and locals, it might not necessarily be useful for iteration variables. In any case re-binding of ref variables is a separate thing.

the difference between ref and ref readonly foreach would be:

foreach(ref var item in array)
{
      // fetches something from item without copying out the whole deal
      var val = item.A.B.C;
 
      // updates a part of item in-place
      item.A.B.C = val + 1;
}

foreach(ref readonly var item in array)
{
      // fetches something from item without copying out the whole deal
      var val = item.A.B.C;
 
      // ERROR: item is a readonly variable
      // item.A.B.C = val + 1;
}

There could be some subtleties around reference/generic arrays because of array covariance.
If you get an object[] array variable that is assigned string[] value, then ref object item would need to fail, possibly at run time.
foreach , i think, is the only place where we implicitly have an explicit cast and run time failures of this sort are already possible. So perhaps it is ok.

In addition to arrays this can probably be generalized to some general ref indexing/iterating patterns.

Interesting idea.

[alrz]

@VSadov

In addition to arrays this can probably be generalized to some general ref indexing/iterating patterns.

If we choose to support duck typing on ref indexers, perhaps we could also support it in non-ref case?

NOTE: the ref variables themselves in either case should not be re-bindable. I.E. cant do item = ref something inside the block.

That was my intention by making it readonly ref, I was just unclear on details. Thanks for elaboration.

[alrz]

@svick

In case extending this to non-array types is desired, what's the advantage of translating ref foreach into ref indexers instead of ref enumerables?

IRefEnumerable is an interface, when it comes to perf tuning you'd probably prefer a struct enumerator, and a convention-based GetRefEnumerator to return it. I think providing a ref indexer and a length property is a much simpler convention to follow.

[svick]

@alrz It's also more general, e.g. it could make sense to iterate ImmutableList<LargeStruct> using a ref readonly foreach using a struct ref-enumerator, but a readonly ref indexer would be too slow (because it's implemented as a tree, so indexer is O(log n)).

And it's more consistent with the existing behavior of foreach.

[alrz]

NOTE: the ref variables themselves in either case should not be re-bindable. I.E. cant do item = ref something inside the block.

the difference between ref and ref readonly foreach would be:

Actually, I think if we are going to disallow item = ref something both those cases should be readonly ref and readonly ref readonly respectively, because item = ref something is exactly what a "readonly reference" disallows, though, we may use ref or ref readonly but the underlying variable is a readonly ref anyways.

[Thaina]

@svick I think it should not relate to indexer. foreach work on Current and MoveNext without the need to index. That could work on yield and other kind of iteration. The need for indexing made it need to remember previous being

[Thaina]

@alrz I propose IRefEnumerable as the same way we has IEnumerable which mean it really just being a contract. The important part is just that any class with ref T Current { get; } and it don't need to be cast to interface

foreach work on IEnumerable like that in general. What I said is it just need to accept ref too and that's all

[alrz]

IEnumerable<T> has its own issues, I don't think it's fair to apply the exact same design to ref iterators where the main motivation is performance.

[Thaina]

@ArlZ

IEnumerable has its own issues

As I said

I propose IRefEnumerable as the same way we has IEnumerable which mean it really just being a contract

You should first understand that foreach never directly use IEnumerable it use only Current and MoveNext explicitly. IEnumerable just make the same contract as foreach are using

The point here is. What we really need is just let foreach accept Current as ref. Any class implement ref return of Current then will be able to use ref foreach

Such as

public class RefList<T> : IRefEnumerable<T>
{
    private T[] array;
    private struct Iterator : IRefEnumerable<T>
    {
        int i;
        RefList<T> refList;
        public ref T Current { get => refList?.array?[i]; }
        public bool MoveNext() { i++; return !(i >= refList?.array?.Length); }
    }

    public IRefEnumerator<T> GetEnumerator() => new Iterator(this);
}

var refList = new RefList<int>();
foreach(ref var i in refList)
    i = 0;

Whenever it foreach it then became

// only one time interface casted. Could be solve with contravariant return type
var iter =  refList.GetIterator();

while(iter.MoveNext())
{
    ref var i = iter.Current;
    i = 0;
}

[lachbaer]

Other BCL functions should be updated as well, e.g. 'Array.ForEach()'

[jnm2]

@alrz The link is screwed fyi.

[alrz]

@jnm2 fixed. thanks.

[BreyerW]

This discussion: #535 triggered an idea: use temporary Span over whole array-like types (arrays, List maybe all types implementing IList?) in order to be able to ref foreach on already existing types but dont implementing ref indexers or RefEnumerable. Spans implement internal GetRawPointer that return ref and propsed API surface suggest there should be ref indexer too which makes them perfect as a trick working over all existing types

[svick]

@BreyerW What would modifying a List while iterating the Span do?

Given List<int> list:

foreach (int i in list)
{
    list.Add(i * 2);
}

This code currently throws an exception.

foreach (ref int i in list)
{
    list.Add(i * 2);
}

This code would presumably also throw an exception, at least when implemented using ref enumerables. I don't think it could throw the exception when implemented using ref indexers, which I think is another argument in favor of ref enumerables.

foreach (ref int i in list.Span)
{
    list.Add(i * 2);
}

This code could not throw an exception. And if you tried to modify i, it might not change the list, because the underlying array could have been exchanged to resize the list.

[BreyerW]

@svick Good points, though i think list could still be modified as long as GC can track address on ref field of span (not sure though if GC is capable of such adv tracking). Even then there might be a hole when list become larger or smaller and there is no way to automatically update length of span (althought in this case just use length of array instead of span?). Well that was just an idea to support ref on existing types in case there wont be any RefEnumerator/ref indexer.

[jcouv]

Note: foreach with ref variable is part of the C# 7.3 "ref local re-assignment" feature.

[VBAndCs]

I also suggest to shorten ref var to ref.