Support pattern-matching in let clauses in Linq

[gafter]

@gafter commented on Thu Nov 19 2015

Given the let-statement proposed in the draft pattern-matching specification for #206, it has been proposed by @alrz to extend the let clause in Linq to support pattern-matching as well:

because of the similarities with let statement I think that would be reasonable to support patterns in it.

from person in list
let Student { Grade is var grade, Name is var name } = person
where grade > 65
select new { Grade = grade, Name = name };
...
// equivalent to
...
from person in list
let $x = person switch(
  case Student { Grade is var grade, Name is var name }: new {grade, name},
  case *: null
)
where $x?.grade > 65        
select new { Grade = $x.grade, Name = $x.name }   
...
// complete patterns
...
from tuple in list
let (var x, var y) = tuple
where x > y
select x + y
...
// equivalent to
...
from tuple in list
let $x = tuple case (var x, var y) : new {x,y}
where $x.x > $x.y
select $x.x + $x.y

etc.

It isn't clear how this would be expected to work when the pattern is fallible.

---

@mattwar commented on Thu Nov 19 2015

You could make it also where as a where clause and omit anything that doesn't match.

---

@HaloFour commented on Thu Nov 19 2015

@gafter

I agree, letting let do double-duty here seems intuitive. But I also agree that failed patterns pose a potential problem. Having the failed patterns omitted automatically seems a little too implicit. Maybe support something like else continue?

from person in list
let Student { Grade is var grade, Name is var name } = person else continue
where grade > 65
select new { Grade = grade, Name = name };

---

@gafter commented on Thu Nov 19 2015

@mattwar I agree, but which Linq method would it translate into? .Where doesn't have a mechanism for adding new variables, and .Let doesn't have a mechanism for filtering.

---

@alrz commented on Thu Nov 19 2015

@HaloFour that's exactly the issue I was thinking about, it implicitly skips failed patterns. but else continue seems fair.

---

@HaloFour commented on Thu Nov 19 2015

With let already providing a bit of syntactic voodoo in LINQ I think that it's reasonable that it would be translated out into multiple extension method calls.

Going with implicit filtering:

from person in list
let Student { Grade is var grade, Name is var name } = person
where grade > 65
select new { Grade = grade, Name = name };

converted into:

list
    // beginning of let expansion
    .Select(person => person switch (
        case Student { Grade is var grade, Name is var Name } :
            (true, person, grade, name)
        case * :
            (false, person, default(int), default(string))
    ))
    .Where(tuple => tuple.Item1)
    .Select(tuple => new { person = tuple.Item2, grade = tuple.Item3, name = tuple.Item4 })
    // end of let expansion
    .Where(projected => projected.grade > 65)
    .Select(projected => new { Grade = projected.grade, Name = projected.name });

---

@alrz commented on Thu Nov 19 2015

I'd say it could be Let:

list
.Let(person => person switch( ... ))
.Where($x => $x.grade > 65)
.Select($x => new { Grade = $x.grade, Name = $x.name });

public static IEnumerable<TResult> Let<T, TResult>(
  this IEnumerable<T> source,
  Func<T, TResult> selector)
{
  foreach(var item in source) {
    var result = selector(item);
    if (result != null) yield return result;
  }
}

---

@gafter commented on Thu Nov 19 2015

@alrz What does result != null mean when you don't know if it is a reference type or not?

---

@alrz commented on Thu Nov 19 2015

@gafter It will be an anonymous type anyway, I forgot to put the class constraint.

---

@gafter commented on Thu Nov 19 2015

@alrz No, it is not required to be an anonymous type. We were hoping to start using tuples in many cases for translating let to reduce GC pressure.

---

@HaloFour commented on Thu Nov 19 2015

@gafter Great point. The compiler could spit out a private static function that accepts the original range variable (or a tuple of all current range variables) and returns an IEnumerable<tuple> of the original range variable(s) plus the variable patterns, then yield return a single result on a match.

Going with implicit filtering:

from person in list
let Student { Grade is var grade, Name is var name } = person
where grade > 65
select new { Grade = grade, Name = name };

converted (somewhat) into:

IEnumerable<(Person, int, string)> $match(Person person) {
    switch (person) {
        case Student { Grade is var grade, Name is var Name }:
            yield return (person, grade, name);
        default: yield break;
    };
};

list
    .SelectMany($match)
    .Where(projected => projected.Item2 > 65)
    .Select(projected => new { Grade = projected.Item2, Name = projected.Item3 });

---

@alrz commented on Thu Nov 19 2015

@gafter I was assuming the current implementation, I did mention this in the tuple issue, too. By the way, that's good to know that it _is_ going to use tuples. 👍

---

@gafter commented on Thu Nov 19 2015

@HaloFour The Linq pattern is not tied to IEnumerable... it would be a shame if this pattern-matching support worked for IEnumerable but no other types.

---

@HaloFour commented on Thu Nov 19 2015

@gafter Apologies for botching the example. I didn't imply that it would be tied to IEnumerable, aside the fact that let would use it to return either 0 or 1 results per attempted match, which I thought was what you were going for.

---

@HaloFour commented on Thu Nov 19 2015

@gafter Wait, my example was correct. I'm not sure where the confusion is?

---

@bbarry commented on Thu Nov 19 2015

from person in list
let Student { Grade is var grade, Name is var name } = person
where grade > 65
select new { Grade = grade, Name = name };

into

list
    .SelectMany(person => person switch (
        case Student { Grade is var grade, Name is var name } : 
            new [] { (person, grade, name) },
        case * : 
            Enumerable.Empty<...>()
    ))
    .Where(projected => projected.Item2 > 65)
    .Select(projected => new { Grade = projected.Item2, Name = projected.Item3 });

?

Is it possible to avoid the allocation for that array?

---

@HaloFour commented on Thu Nov 19 2015

@bbarry

The compiler wouldn't be forced to impose the restriction of closures not being iterators on itself. I assume that the compiler would emit a private static iterator function and then pass a delegate to that function to SelectMany.

Your example using arrays is much clearer than mine in illustrating the concept, I think.

---

@orthoxerox commented on Thu Nov 19 2015

What about this approach?

var students =
from person in people
where person is Student student
select student;
//is transformed into
var students = people
.Select(person => (person is Student student)?(true, student):(false, default(Student)))
.Where(tuple => tuple.Item1 = true)
.Select(tuple => tuple.Item2);

---

@HaloFour commented on Thu Nov 19 2015

@orthoxerox Looks pretty similar to my first suggestion for an expansion, although you're using is with a ternary rather than the switch expression.

---

@gafter commented on Thu Nov 19 2015

Why are all you folks tying this to Enumerable (e.g. using Enumerable.Empty or having the compiler generate a method that performs yield return)? The Linq feature can be used with any type that fits the API pattern (e.g. PLINQ, Linq to SQL, IObservable, etc). If we have this construct force the result to be Enumerable, then we've broken one of the important features of Linq.

---

@gafter commented on Thu Nov 19 2015

@HaloFour Your first proposed expansion (also @orthoxerox 's most recent proposed expansion) would work.

---

@bbarry commented on Thu Nov 19 2015

@gafter I thought you said SelectMany above; I was trying to figure out how that would have worked. I think @HaloFour's tuple expansion dotnet/roslyn#6877 (comment) is better. It seems odd to have let reduce the number of items in the statement...

---

@gafter commented on Thu Nov 19 2015

@bbarry I thought SelectMany could be used, but I tried and now I don't think so.

---

@HaloFour commented on Thu Nov 19 2015

@gafter Indeed, implementing this via SelectMany would lock it to IEnumerable<T> under the hood as the mechanism to return zero or one value. It was your idea, we were following it. If this tie to IEnumerable<T> is unacceptable then that's fine, but you forgot to mention that before chastising us for chasing your concept.

Anyway, that is all implementation details. I'd be more concerned with hammering down the semantics and syntax (and variations thereof) at this stage and worrying about how exactly that could be expanded later. Particularly of interest would be any capability of this feature functioning as a filter as well as a projection, either by default or through additional syntax.

Aside that, it is nice to hear that LINQ may take advantage of tuples for let projections.

---

@alrz commented on Thu Nov 19 2015

@HaloFour SelectMany is not tied to IEnumerable<T>, implement it for your desired type and it will work

static M<U> SelectMany<T, U>(this M<T> m, Func<T, M<U>> k)

No need to mention, it's the bind operator.

---

@gafter commented on Thu Nov 19 2015

@alrz How would you use SelectMany to implement a pattern-based let clause, i.e. so that it can return one result or none, without tying it to IEnumerable?

---

@alrz commented on Thu Nov 19 2015

@gafter you mean tying it to something else like maybe? because there should be something to represent one result or none.

---

@gafter commented on Thu Nov 19 2015

@alrz I don't have a particular solution in mind.

---

@alrz commented on Fri Nov 20 2015

@gafter Indeed, it can be implemented using SelectMany 😄 Here's a particular solution,

public static M<U> SelectMany<T, U>(this M<T> source, Func<T, U?> k)
    where U : struct
{
    return from item in source
           let result = k(item)
           where result.HasValue
           select result.Value;
}

public static M<V> SelectMany<T, U, V>(this M<T> m, Func<T, U?> k, Func<T, U, V> s)
    where U : struct
    where V : struct
{
    return m.SelectMany(x => k(x).SelectMany(y => new V?(s(x, y))));
}

public static U? SelectMany<T, U>(this T? m, Func<T, U?> k)
    where T : struct
    where U : struct
{
    return m.HasValue ? k(m.Value) : default(U?);
}

I simulated Maybe a with Nullable<T> for demonstration, so it can only be used with value types (see below), substitute it with Option<T> and it works for all types.

I'm using higher-kinded generic type M<T> you can substitute it with any type that has implemented SelectMany like IEnumerable<T> and it works for that particular type.

As an example,

// assuming
var list = new object[] { 1, 2.0, 3 };

// the following
var q = from item in list
        let double d = item
        select d;

// translates into
var q = from item in list
        from d in item is double num ? new double?(num) : default(double?)
        select d;

The return value of pattern matching can be an Option<T> where T is a tuple or anonymous type, to be used in the consequent LINQ operators.

The thing is, in async sequences, we have three container types to combine: Task<T>, IEnumerable<T> and Option<T> which complicates things exponentially! In that case, I think higher-kinded generics would be really helpful.

---

@gafter commented on Fri Nov 20 2015

@alrz None of your SelectMany methods are part of the existing Linq pattern described by the C# language specification. I meant using the same SelectMany methods that must already be implemented for the current Linq pattern - for example, the ones in Enumerable. No fair mixing monads.

---

@alrz commented on Fri Nov 20 2015

@gafter Well, it cannot be done I guess! They are not new implementations, I just implemented bind for Nullable<T> (third method) and a transformer for it (first method) and a helper function to be used in query expressions (second method). These are part of functional languages like Haskell and they are well integrated in the language, so you don't need to implement them yourself. The fact that Nullable<T> is not introduced as a monad in the language, restrict us in this kind of situations. Anywho, I'm really interested to see how this can be done using the existing SelectMany without tying it to a particular type.

---

@gafter commented on Fri Nov 20 2015

@alrz I'm guessing it cannot be done.

---

@HaloFour commented on Fri Nov 20 2015

@alrz All you're really doing is implementing the same Select/Where/Select implementation from above. The C# compiler can already emit tuples (or anonymous types) to project an analog to Option<T> so this would require no new BCL types or methods.

Also, to note, that second Select wouldn't be necessary as the compiler can simply choose to ignore the fact that the success flag is still projected. If the LINQ query performs another let match or select projection it could omit it at that point.

---

@alrz commented on Fri Nov 20 2015

@HaloFour This is just functional way of doing things (as if they were existed in the language for Option<T> etc) so they wouldn't be special just for implementing let in query expressions. If Select/Where is the prefered way, I do agree that it's more optimized and straightforward solution for this. In that case I think expanding where ... is ... would be more sensible than translating let to a where clause (in case of a fallible pattern). So let would be used for complete patterns, and where ... is ... otherwise.

---

@HaloFour commented on Fri Nov 20 2015

@alrz

My samples assume an implicit filtering of failed patterns, something that I don't know has been decided upon. If failed patterns wouldn't be filtered out then there would be no call to Where.

Well where ... is ... already means something today and doesn't convey the capture of those range variables. If you're arguing that declaration expressions or pattern variables should be lifted to range variables that might be a different proposal.

---

@alrz commented on Fri Nov 20 2015

Well where ... is ... already means something today and doesn't convey the capture of those range variables.

Yes it does, as for is itself. But there weren't any "variables" for is before. Don't forget we have two kind of is operator,

relational-expression:
 relational-expression is type
 relational-expression is complex-pattern

So, I'm saying that let can be used for complete patterns like

from item in list
let (var x, var y) = item
...

And the second form of is for fallible patterns.

from item in list
where item is Foo foo
...

And for the existing form of is it would behave as it always has.

---

@HaloFour commented on Fri Nov 20 2015

@alrz

Pattern variables don't leak out of scope with is, though. Nor would they with where. That where clause could similarly combine other Boolean operations, including with declaration expressions, where it was determined that the scope would not exceed the where clause.

---

@alrz commented on Fri Nov 20 2015

Now that you mentioned declaration expressions it's understandable for me that changing scoping just for pattern variables doesn't make sense. Since let clause variable scope is well defined I think that's the only option and for implicit skipping issue, as I said, else continue seems fair, as you suggested, but I'm not sure about using continue because it's already a statement and might be confusing.

---

@gafter commented on Fri Nov 20 2015

Remember that the semantics of query expressions are defined mostly in terms of translation to invocation of methods with expression lambdas. Pattern variables will not leak out of expression lambdas. If we want to change the scoping we need to define a different translation.

---

@alrz commented on Fri Nov 20 2015

@gafter

semantics of query expressions are defined mostly in terms of translation to invocation of methods with expression lambdas.

I presume that mostly means "except for let". I did take my suggestion back regarding expanding where. But still, I think it's not that illogical to have this behavior with let clause (without any change in scoping) — btw, I'm sure with introducing let statement for deconstruction, this one is expected as well. At least, it could just support complete patterns for deconstructing tuples (which I believe will be more commonly used than anonymous types in LINQ).

---

@gafter commented on Fri Nov 20 2015

It isn't let in particular that is the exception. There are a few things that make it not quite a syntactic transformation:

1. The query methods are "expected to have particular signatures and result types, as described in §7.16.3"
2. "Assignment to range variables is not allowed in query expressions. However a C# implementation is permitted to not always enforce this restriction, since this may sometimes not be possible with the syntactic translation scheme presented here." ... Roslyn always enforces this.
3. Some translation steps (like let and a non-initial select) involve the use of "transparent identifiers" which are described to be translated into the use of anonymous types. However, "An implementation of C# is permitted to use a different mechanism than anonymous types to group together multiple range variables." ... This gives us the flexibility to use tuples.

---

@paulomorgado commented on Mon Nov 23 2015

I got a bit confused around this.

Part of this discussion is about deconstruction in LINQ queries giving a new meaning to the let keyword and match it with the "new" let keyword outside of LINQ queries, right?

How would this behave outside of LINQ?

let Student { Grade is var grade, Name is var name } = person;

Should it behave differently in LINQ queries? I think not.

The other of this discussion is about a condition introducing new "variables", right?

In order for the compiler to handle this without the need for introducing new signatures in Where, how about this?

where person is Student { Grade is var grade, Name is var name }

would be logically translated to:

let isMatch = person is Student { Grade is var grade, Name is var name }
where isMatch

or concretely to:

.Select(person => person is Student { Grade is var grade, Name is var name }
    ? (true, person, grade, name)
    : (false, person, default(Grade), default(Name)))
.Where(tuple => tuple.Item1)
.Select(tuple => (tuple.Item2, tuple.Item3, tuple.Item3))

The last Select is just to drop the condition value, but it could be kept until the very end and be reused in other similar patterns.

---

@HaloFour commented on Mon Nov 23 2015

@paulomorgado The problem remains the scoping. If the pattern introduces new variables through variable patterns then in normal circumstances those variables would be out of scope as soon as the clause terminated. That would remain true using is within a let clause unless that behavior was defined. If that behavior would be reconsidered then this issue does go away:

from person in list
where person is Student { Name is var name, Grade is var grade } && grade > 65
select new { name, grade };

This is basically a continuation of the conversation at CodePlex except with patterns instead of variable declarations.

---

@gafter commented on Mon Nov 23 2015

@paulomorgado The expression person is Student { Grade is var grade, Name is var name } is already an ordinary boolean expression that could be used in a lambda (or will be with the addition of pattern-matching), so I think we would be reluctant to give it a special meaning in a query's where clause.

---

@paulomorgado commented on Mon Nov 23 2015

@HaloFour, doesn't the way I showed introduce those variables?

@gafter, the difference is the variables introduced. And I was not proposing to give a new meaning to the where clause. Was I?

---

@HaloFour commented on Mon Nov 23 2015

@paulomorgado No, not without other changes to the semantics of the let clause. The only additional range variable it would introduce is isMatch. The others would end up going out of scope.

---

@gafter commented on Mon Nov 23 2015

@paulomorgado Yes, you were proposing a change from the current specification for the translation of the where clause.

where person is Student { Grade is var grade, Name is var name }

would be logically translated to:

let isMatch = person is Student { Grade is var grade, Name is var name }
where isMatch

or concretely to:

.Select(person => person is Student { Grade is var grade, Name is var name }
    ? (true, person, grade, name)
    : (false, person, default(Grade), default(Name)))
.Where(tuple => tuple.Item1)
.Select(tuple => (tuple.Item2, tuple.Item3, tuple.Item3))

I say this is a change because the syntax after where is an expression, and the current spec already says what to do for where expression.

---

@paulomorgado commented on Tue Nov 24 2015

OK. It's a "change", not a "CHANGE".

@gafter, on hindsight, that last Select is kind of stupid. But, apart from that and the needed introduction of the need two variables (grade, name), what's the difference from translating this:

from i in Enumerable.Range(0, 10)
let c = i % 2 == 0
where c
select I

into this (now using the tuple proposed syntax)?

Enumerable.Range(0, 10)
    .Select(i => (i, i % 2 == 0))
    .Where(((i, c)) => c)
    .Select(((i, c)) => i)

---

@gafter commented on Tue Nov 24 2015

@paulomorgado the translation above is using precisely the set of methods specified in the language specification, and introduces precisely the set of range variables specified in the language specification, so it appears to comply with the spec (which allows an implementation to use other techniques for representing transparent identifiers).

Incidentally, the last two lambdas are syntax errors as written. I think it needs to be this:

Enumerable.Range(0, 10)
    .Select(i => (i, i % 2 == 0))
    .Where(t => t.Item2)
    .Select(t => t.Item1)

---

@paulomorgado commented on Tue Nov 24 2015

@gafter, I was trying to be "tuply".

---

@alrz commented on Tue Nov 24 2015

@paulomorgado It probably would be

Enumerable.Range(0, 10)
    .Select(i => (i, i % 2 == 0))
    .Where(t => t case (*, var c): c)
    .Select(t => t case (var i, *): i)

// or

Enumerable.Range(0, 10)
    .Select(i => (i, c: i % 2 == 0))
    .Where(t => t.c)
    .Select(t => t.i)

Since #6067 is not under consideration.

---

@gafter commented on Tue Nov 24 2015

@alrz Why would the compiler translate into an application of pattern-matching when it could just generate a member access?

---

@gafter commented on Tue Nov 24 2015

@paulomorgado You mean "Twoply"? ;)

---

@alrz commented on Tue Nov 24 2015

@gafter Compiler surely would do the second one.

---

@gafter commented on Tue Nov 24 2015

@alrz The compiler surely would not give itself busy-work by producing new forms of syntactic sugar that it would then have to undo.

---

@alrz commented on Tue Nov 24 2015

@gafter Isn't this the same behavior that currently has been done by anonymous types that is going to implemented using tuples?

---

@HaloFour commented on Tue Nov 24 2015

@gafter

You mean "Twoply"? ;)

There's probably a potty joke there somewhere.

---

@gafter commented on Tue Nov 24 2015

@alrz When we use anonymous types, we actually create them as specified. Of course the actual anonymous types have names from the CLR's point of view, but there isn't much lowering going on after that.

---

@alrz commented on Tue Nov 24 2015

@gafter In the code above I meant the t name is compiler generated

Enumerable.Range(0, 10)
    .Select(i => (i, c: i % 2 == 0))
    .Where($t => $t.c)
    .Select($t => $t.i)

Enumerable.Range(0, 10)
    .Select(i => new { i, c = i % 2 == 0})
    .Where($t => $t.c)
    .Select($t => $t.i)

I don't know what I'm missing here.

---

@gafter commented on Tue Nov 24 2015

I mean why would we generate names such as c and i that we would then have to bind and translate into Item2 and Item1? We're done binding; there is no reason to translate into names that do not exist after binding.

---

@alrz commented on Tue Nov 24 2015

@gafter I see, I know that these names will be ultimately erased, actually I meant "equivalent to" not "translated into" anywho, what if we are doing this over Expression<Func<,>> shouldn't the variable names be accessible for IQueryable<> translations? same issue with using tuples in select

from item in db.Products
select (item.Name, item.Model)

so you say libraries like EF won't get access to these names to do projection etc?

---

@gafter commented on Tue Nov 24 2015

An Expression<Func<...>> would see the expanded form, just like today. I expect we'd continue to generate anonymous types for transparent identifiers in expression trees so as not to break existing code. Lots of code is quite sensitive to the exact expression tree produced. But when you use a tuple, the expression tree would see the lowered form.

---

@alrz commented on Tue Nov 24 2015

@gafter Assume that user just wrote the above (to get a list of tuples), it will be "not supported" or compiler would generate an anonymous type (to be used for Expression<Func<...>>) and then yields a tuple?

---

@gafter commented on Tue Nov 24 2015

@alrz It will be exactly the same as if the user had written

from item in db.Products
select new ValueTuple<string, int>(item.Name, item.Model)

---

@alrz commented on Tue Nov 24 2015

So I presume that tuples can't be used in EF queries at all.

---

@bbarry commented on Tue Nov 24 2015

This tuple discussion risks getting lost by virtue of not being related to the issue. This probably isn't the correct place to have it. What @gafter was saying though was that the current anonymous types that get generated for range variables internally to the linq-expression (ones that never get exposed to the user) could be potentially replaced by tuples when that linq-expression is generating code that runs over the IEnumerable<> type. Perhaps some future types could hint to the compiler what kind of intermediate types should be generated (so the compiler doesn't expressly apply this change only for that specific instance).

the linq expression:

from i in Enumerable.Range(0, 10)
let c = i % 2 == 0
where c
select i

currently uses an anonymous tuple that gets created in the let-clause, but doesn't appear in the result.

If the query was instead:

from i in Enumerable.Range(0, 10)
let c = i % 2 == 0
select (c, i)

the type of the result item in that linq expression would be a tuple. There is still some anonymous background thing generated in the let-clause, but it has no affect on the result type of the expression.

---

@alrz commented on Tue Nov 24 2015

@bbarry as @gafter said

I expect we'd continue to generate anonymous types for transparent identifiers in expression trees so as not to break existing code.

So

from i in Enumerable.Range(0, 10)
let c = i % 2 == 0
select (c, i)

can be translated into

Enumerable.Range(0, 10)
    .Select(i => (i, i % 2 == 0))
    .Select(t => (t.Item2, i.Item1))

with no breaking change.

---

@HaloFour commented on Tue Nov 24 2015

@alrz @bbarry

Whereas:

from i in Enumerable.Range(0, 10).AsQueryable()
let c = i % 2 == 0
select (c, i);

would likely be translated as:

Enumerable.Range(0, 10)
    .AsQueryable()
    .Select(i => new { i = i, c = i % 2 == 0 })
    .Select(t => (t.c, t.i));

---

@bbarry commented on Tue Nov 24 2015

@alrz, I think so. That looks like a potential optimization the compiler could do for a linq expression that uses the extension methods on Enumerable. What heuristics would actually get used to determine if that optimization was good, I have no idea.

@HaloFour that is what I would expect.

---

@alrz commented on Tue Nov 24 2015

Nevertheless, I think for

from item in db.Products
select (item.Name, item.Model)

it would be reasonable that it be translated as

db.Products
.Select(item => new {item.Name,item.Model})
.Select(t=>(t.Name,t.Model))

Otherwise, if you accidentally use a tuple for projecting IQueryable<T> you are just accidentally screwed.

---

@HaloFour commented on Tue Nov 24 2015

@alrz

You'd be opting-into tuples explicitly in that case which is very different from whatever let automagickally does under the hood. Note that your translation would suffer the same amount of screwed-ness given both Selects would be in the expression tree and would have to be translated by Entity Framework or whatever queryable provider.

---

@alrz commented on Tue Nov 24 2015

@HaloFour I'm thinking that EF guys should be loud about this and write a big bold statement somewhere that states DON'T USE TUPLES, while the main motivation behind implementing the same behavior as anonymous types (inferring item/property names from the input expression) is using it with EF (at least this is what it would be expected).

---

@bbarry commented on Tue Nov 24 2015

There is nothing stating that linq expression tree visitor code couldn't be modified to support tuples, that is an enhancement on those things.

I think it would be useful for the expression tree representing that tuple to know the compile time names used, but I don't see why a person who "accidentally" projects an IQueryable<T> into a tuple shouldn't be permitted to get exactly the thing they expressed in code.

---

@HaloFour commented on Tue Nov 24 2015

@alrz I bet they add support for projecting to tuples long before any of this ships.

---

@alrz commented on Wed Nov 25 2015

@bbarry I would totally love that. Expression trees are generated in compile-time, so are tuple names. nothing should stop you from being able to get those sweet names from an expression tree (#2060).

---

@paulomorgado commented on Thu Nov 26 2015

@gafter,

Not "oneply", "twoply" or "threeply" in particular. Just "tuply" in general. 😄

/cc @HaloFour

---

@alrz commented on Tue Jan 26 2016

If we unify nullables, with help of higher kinded types I think we can implement generic LINQ operations for nullables and utilize it for incomplete let (because that's what it is! it changes the type of the collection to nullable).

Besides, although we have nullable reference types, T? meaning two things, depending on the type, is just awful.

---

@HaloFour commented on Wed Feb 03 2016

I'm going to posit that by default a failed pattern should result in a run-time exception. As with pattern matching in any other context the compiler should go to whatever lengths it can to prevent that. Where the pattern may be fallible I think that reusing the else keyword with an embedded statement is useful and should follow very similar rules to that of using let for destructuring outside of LINQ. The one difference is that the embedded-statement cannot contain a return statement since that wouldn't make a lot of sense in the LINQ query. But since the LINQ query can be considered a looping construct I think that continue and break should both be acceptable.

In the case of continue the LINQ query will omit the elements that do not match the pattern:

from person in list
let Student { Grade is var grade, Name is var name } = person else continue
where grade > 65
select new { Grade = grade, Name = name };

// translated into

list
    // beginning of let expansion
    .Select(person => person switch (
        case Student { Grade is var grade, Name is var Name } :
            (true, person, grade, name)
        case * :
            (false, person, default(int), default(string))
    ))
    .Where(tuple => tuple.Item1)
    .Select(tuple => new { person = tuple.Item2, grade = tuple.Item3, name = tuple.Item4 })
    // end of let expansion
    .Where(projected => projected.grade > 65)
    .Select(projected => new { Grade = projected.grade, Name = projected.name });

In the case of break the LINQ query will terminate the sequence:

from person in list
let Student { Grade is var grade, Name is var name } = person else break
where grade > 65
select new { Grade = grade, Name = name };

// translated into

list
    // beginning of let expansion
    .Select(person => person switch (
        case Student { Grade is var grade, Name is var Name } :
            (true, person, grade, name)
        case * :
            (false, person, default(int), default(string))
    ))
    .TakeWhile(tuple => tuple.Item1)
    .Select(tuple => new { person = tuple.Item2, grade = tuple.Item3, name = tuple.Item4 })
    // end of let expansion
    .Where(projected => projected.grade > 65)
    .Select(projected => new { Grade = projected.grade, Name = projected.name });

And, for completeness, support throw:

from person in list
let Student { Grade is var grade, Name is var name } = person else throw new InvalidOperationException("oops!")
where grade > 65
select new { Grade = grade, Name = name };

// translated into

list
    // beginning of let expansion
    .Select(person => person switch (
        case Student { Grade is var grade, Name is var Name } :
            (person, grade, name)
        case * :
            throw new InvalidOperationException("oops!")
    ))
    .Select(tuple => new { person = tuple.Item1, grade = tuple.Item2, name = tuple.Item3 })
    // end of let expansion
    .Where(projected => projected.grade > 65)
    .Select(projected => new { Grade = projected.grade, Name = projected.name });

And, of course, multiline embedded statements:

from person in list
let Student { Grade is var grade, Name is var name } = person else {
    Console.WriteLine("whoa, unexpected!");
    continue;
}
where grade > 65
select new { Grade = grade, Name = name };

// translated into

list
    // beginning of let expansion
    .Select(person => {
        switch (person) {
            case Student { Grade is var grade, Name is var Name } :
                return (true, person, grade, name);
            case * :
                Console.WriteLine("whoa, unexpected!");
                return (false, person, default(int), default(string));
        }
    })
    .Where(tuple => tuple.Item1)
    .Select(tuple => new { person = tuple.Item2, grade = tuple.Item3, name = tuple.Item4 })
    // end of let expansion
    .Where(projected => projected.grade > 65)
    .Select(projected => new { Grade = projected.grade, Name = projected.name });

---

@alrz commented on Wed Feb 03 2016

@HaloFour Still, you are translating let to more than one LINQ operator (and worse, depending on the else clause). Supporting break, continue or throw in query expressions just for let doesn't make much sense, and also, you assumed that continue and break are expressions, otherwise the else clause should have some weird production rules to support them besides of throw. However, I believe they should be expressions to be able to write something like #5143 comment.

My two cents,

extension<T> IEnumerable<T> {
  public IEnumerable<TResult> Select<TResult>(Func<T, Option<TResult>> selector) {
    foreach(var item in this) {
      let Some(var value) = selector(item) else continue;
      yield return value;
    }
  }
}

from obj in list
let Foo foo = obj
select foo;

list.Select(obj => obj is Foo foo ? Some(foo) : None());

from person in list
let Student { Grade is var grade, Name is var name } = person
where grade > 65
select new { Grade = grade, Name = name };

list.Select(person => person is Student { Grade is var grade, Name is var name } ?
  Some((grade,name)) : None())
.Where(t => t.grade > 65)
.Select(t => new { Grade = t.grade, Name = t.name});

Then you might think of else continue to make it explicit.

Using tuples for fallible patterns is not intuitive at all, because still you need to allocate default values which totally loses the point. I think an option type is the best solution to this kind of problems, regardless of what language you are coding in.

---

@HaloFour commented on Wed Feb 03 2016

@alrz

Still, you are translating let to more than one LINQ operator (and worse, depending on the else clause).

There is nothing that states that a LINQ clause must be directly translated to a single extension method. I am working with what I have. .NET 3.5 LINQ has no single function which performs both a map and a filter. If a future version of the BCL does include such a function then that would be a good target for optimization, but as it stands I see no reason to limit the feature to some not-yet-released CLR.

I am aware that I am using tuples in my example translation which would not exist until a new BCL is released. When targeting a currently released framework the translation would use anonymous types instead, just as it does today with let range variables.

Supporting break, continue or throw in query expressions just for let doesn't make much sense, and also, you assumed that continue and break are expressions, otherwise the else clause should have some weird production rules to support them besides of throw.

I think it makes a lot of sense and also provides good symmetry for the non-LINQ destructuring form of let. And none of those statements need to be treated as expressions as the else clause cannot produce a value. What follows the else clause is an embedded-statement which must not reach the end of the block. This would be the exact same syntax and behavior that would be expected of the non-LINQ destructuring statement when used within a loop.

Using tuples for fallible patterns is not intuitive at all, because still you need to allocate default values which totally loses the point.

The developer doesn't have to do anything. The use of anonymous types, tuples or even Option<T> would be strictly an implementation detail. And the compiler using default(T) to fill in those empty anonymous type or tuple properties is effectively free.

Also, I don't think that going for an "overload" of Select is a good idea for your proposed extension method. There are valid reasons to want to include the None results in the projected sequence.

---

@alrz commented on Wed Feb 03 2016

@HaloFour

There is nothing that states that a LINQ clause must be directly translated to a single extension method.

I think that was the first thing that came up and lead to SelectMany etc. Otherwise an additional Where would just work.

I see no reason to limit the feature to some not-yet-released CLR.

Same was true for await and it is true for tuple types now, but they probably can be supported through some libraries in earlier versions.

What follows the else clause is an embedded-statement

So, perhaps you need to end your embedded statement with a semicolon? Furthermore, embedding an embedded statement in a query expression doesn't seem like a good idea.

The developer doesn't have to do anything.

True, but those values will be allocated eventually right?

---

@HaloFour commented on Wed Feb 03 2016

@alrz

I think that was the first thing that came up and lead to SelectMany etc. Otherwise an additional Where would just work.

That was a conceptual implementation that would have avoided the multiple extension methods but would have created an unwanted dependency on IEnumerable<T>. If you read back @gafter was satisfied with my original transformation which I simply repeated here.

Same was true for await and it is true for tuple types now, but they probably can be supported through some libraries in earlier versions.

Actually await follows a convention and has no dependency on Task<T>. async does, however. It has been mentioned that the language team wanted to avoid this dependency but didn't have a good way to do so. There's no reason to create a dependency when one doesn't have to, and my transformations demonstrates that it's not necessary. That's not to mean that a newer better implementation can't be used when targeting a newer framework.

So, perhaps you need to end your embedded statement with a semicolon? Furthermore, embedding an embedded statement in a query expression doesn't seem like a good idea.

Why? It's not the end of the expression. I don't see any problem with embedding a statement or a block in a query expression. However if that would be considered confusing I personally would be fine with that statement being limited to break, continue or throw. In either case the clause does not need to be an expression since no value is being returned.

True, but those values will be allocated eventually right?

No, default(string) is literally the same as null, and default(int) is literally the same as 0. Zero allocations are required. I could have used null in their place in my example translations but then I'd probably have to include type-casts to keep the pseudo-compiler happy.

---

@alrz commented on Wed Feb 03 2016

@HaloFour "Why?" because the semicolon is a part of an embedded statement. "the clause does not need to be an expression since no value is being returned." That is also true for throw, then I don't see why continue or break can't be an expression to be used in switch in the example that I mentioned above. "No, default(string) is literally the same as null" I meant the additional wasted pointer, no allocations. PS: At first your comment looked like the unfinished portrait of Franklin Roosvelt 😄

---

@HaloFour commented on Wed Feb 03 2016

@alrz

because the semicolon is a part of an embedded statement.

embedded-statement is a list of other potential statements. Some of those statements do have semicolon requirements, although block doesn't. I'd be fine if the spec for let in this context differed syntactically as obviously a semicolon at that part of the query makes no sense.

That is also true for throw, then I don't see why continue or break can't be an expression to be used in switch in the example that I mentioned above.

Probably because you're using the expression form of switch and it was mentioned that it was a bad idea to let the middle of an expression use control statements. That's a different proposal altogether, though. The non-LINQ let would permit continue or break in it's else clause so this would provide a nice symmetry, and these control statements can only affect the query, not the declaring method.

I meant the additional wasted pointer, no allocation.

64-bits of empty space, no big deal. I'm all for going with a more optimal route given a newer BCL but I think it's ridiculous to not permit a language feature to work on the older frameworks when there is clearly an easy way to implement it, even if it might be slightly less efficient.

PS: At first your comment looked like the unfinished portrait of Franklin Roosevelt

I managed to accidentally find some kind of keyboard combination which submits the comment immediately. I don't even know what I hit 😦

---

@alrz commented on Wed Feb 03 2016

@HaloFour As the last inquire, would it make sense to turn let clause to let statement in case of presence of do as the last part of the query (#1938) so it doesn't take the unpleasant translation of an incomplete let to tuples with an additional bool?

from obj in list
let Foo foo = obj else continue
do WriteLine(foo.Bar);

foreach(var obj in list) {
  let Foo foo = obj else continue;
  WriteLine(foo.Bar);
}

---

@HaloFour commented on Wed Feb 03 2016

@alrz Works for me. I'd want the compiler to use the most efficient implementation that it could.

---

@orthoxerox commented on Wed Feb 03 2016

@alrz LINQ from-syntax is always rewritten into LINQ functional syntax. You would need to add a lookahead translation rule like the one for from-from-select, for example.

from x1 in e1
let P1 x2 = e2 else continue
do s3;

would be rewritten into:

e1.Select(x1 => {
    let P1 x2 = e2 else return;
    { s3; }
})

---

@alrz commented on Wed Feb 03 2016

@orthoxerox

Per @gafter's comment, it will be a new statement form, not on top of the existing query-expression, but I'm not sure about let translation. And your example will be translated to foreach. See #1938.

[nth-commit]

I find it quite jarring that destructuring tuples is not supported in LINQ expressions. Even if it's not widely requested, I think language consistency should come into play in terms of prioritisation of features.