[Little Feature Request] LINQ query materialization

[gordanr]

I am pretty sure that C# community would appreciate improvements in LINQ technology in the next C#7 release. Maybe it is time to add some new language elements in query expression syntax. There are a lot of standard query operators which are not included in query expression syntax. Some of them are really hard to integrate in syntax without a loss of readability.

The best candidates for syntax improvements could be element operations (Single, FirstOrDefault, ...), aggregation operations (Max, Count, ...) or any other method which materialize LINQ query (ToList, ToArray, ...)

Here is the proposal for the ToList and ToArray methods.

var filesQ = from file in ctx.Files
             where file.Name.StartsWith("Foo")
             orderby file.Name
             select new { Name = file.Name, Size = file.Size }
var files = filesQ.ToList();

Very common and annoying ToList could be integrated in query expression with 'as List'.

var files = from file in ctx.Files
            where file.Name.StartsWith("Foo")
            orderby file.Name
            select new { Name = file.Name, Size = file.Size }
            as List;

[svick]

This is very similar to #100.

[ghost]

Going by SQL semantics, it should be CAST operation and AS should be used for aliasing.

[gordanr]

@jasonwilliams200OK
My first association with 'as' in LINQ context is not 'SQL aliasing', but C# 'as operator'. SQL aliasing is already well implemented in LINQ with anonymous types.

[gordanr]

@svick
I admit that this proposal is very similar to #100.

New issue is opened deliberately because I restrict proposal only to 'as List' and 'as Array'. There are not any new reserved or contextual keywords. List and Array in proposed syntax are type names. Small feature in form 'select expression [as Type]'

I would like to have element operations and aggregation operations in syntax but I really have any good idea.

I would say that this proposal uses too much new contextual keywords. It is readable but it's not aways 'writeable' because of complex syntax. Probably is hard to implement. There is a lack of generality.

select sum of item.foo;

select single item.foo;

select first item.foo or default;

This is in form 'select [Method] expression'
General? Readable? Any other problem in implementation?

select Sum item.foo;

select Single item.foo;

select FirstOrDetault item.foo;

This is in form 'select expression [as Method]'.
General? Readable? Any other problem in implementation?

select item.foo as Sum;

select item.foo as Single;

select item.foo as FirstOrDefault;

For now, I stay only with 'as List' ans 'as Array'.

[miloush]

Do you expect the 'as Array' to return Array or strongly typed array like the ToArray() method returns? Because that contradicts a bit how the 'as' operator works and it definitely feels wrong to just have 'List' without any type arguments.

[gordanr]

@miloush
Simply, this piece of code

var q1s = (from l in list select new StrongType() { Id = l.Id, Name = l.Name}).ToList();
var q1a = (from l in list select new { Id = l.Id, Name = l.Name }).ToList();

var q2s = (from l in list select new StrongType() { Id = l.Id, Name = l.Name }).ToArray();
var q2a = (from l in list select new { Id = l.Id, Name = l.Name }).ToArray();

is equivalent with

var q1s = from l in list select new StrongType() { Id = l.Id, Name = l.Name } as List;
var q1a = from l in list select new { Id = l.Id, Name = l.Name } as List;

var q2s = from l in list select new StrongType() { Id = l.Id, Name = l.Name } as Array;
var q2a = from l in list select new { Id = l.Id, Name = l.Name } as Array;

Any deeper comparison between 'linq as operator' and 'as operator' from the paragraph 7.10.11 of the C# Language Specification is bad analogy. Contextual keyword 'where' also has different meanings in different contexts.

[miloush]

Okay so you would only allow as List and as Array constructs? How about as IEnumerable which now works? Actually as Array is currently legal code so it might be a breaking change. How would you distinguish whether as is the regular operator or your syntax sugar in select new string[0] as Array?

Finally do I understand correctly that in the case of

var q2a = from l in list select new StrongType { Id = l.Id, Name = l.Name } as Array;

the q2a woud be of type StrongType[] while in the case of

var q2a = from l in list select new StrongType { Id = l.Id, Name = l.Name } as Array as Array;

the q2a would be of type Array?

[GeirGrusom]

My opinion is that .ToList() and .ToArray() does not belong in a query language.

edit: it's fine as a C# extension function, but it has no place in the LINQ syntax.

[AdamSpeight2008]

I agree @GeirGrusom that query language (LINQ) should be an concept abstraction independent of the underlying data. '.ToList' to '.ToArray' are only applicable to LINQ-To-Objects.

[gordanr]

Thank you all for your comments. I work predominantly with databases using Entity Framework. There are thousands (without exaggeration) of these ugly sentences in my codebase which decrease readability.

int c = (from ... where ... select ...).Count();
var o = (from ... where ... select ...).SingleOrDefault();

I am specially annoyed with .ToList() construct.

var l = (from ... where ... select ...).ToList();

I really can't give you any stronger argument, except my intuition that ToList (ToArray) method is the key method regarding this problem. Maybe the mentioned problem with breaking change could be solved anyhow if the idea with 'as List' is worth enough.

Any idea that removes braces around the expression is welcomed.

[Joe4evr]

var l = (from ... where ... select ...).ToList();

This is basically why I started preferring just chaining LINQ methods instead of using the query syntax.

var l = db.Foos.Where(f => ...)
               .Select(f => new {
                   //...
               })
               .ToList();

[GeirGrusom]

This is basically why I started preferring just chaining LINQ methods instead of using the query syntax.

I find this a little bit silly. You want to replace parenthesis with white space more or less. The difference is so small and there are expressions that are cumbersome to express using the extension methods directly:

var query = from customer in Customers
    let deliveryDate = DateTime.Today + customer.Patience
    where deliveryDate < acceptedDeliveryDate
    select new { Customer, DeliveryDate = deliveryDate };

What is more of an issue is Skip and Take as they will impact the expression but is not representable in LINQ. ToList and ToArray are postfix expressions, so they do no impact usage much, and they are basically no-op in a query expression.

First, Single, Last are not no-op in an query, so I would think they had a place in LINQ.

Count, Sum and Average probably also has a place, but I don't currently take much offense to the fact that they are separate aggregate methods.

[ghost]

@gordanr, I agree and they can probably co-exist.

CAST {EXPRESSION} AS <TYPE>

For example:

    cast from file in ctx.Files
    where file.Name.StartsWith("Foo")
    as IList<File>

CAST keyword might come handy (perhaps act like a separator) for scenarios where we have chained and/or nested expressions.

[GeirGrusom]

They are collectors, not cast operations.

[chrisaut]

How about into? It is already used with group by

var myList = from file in ctx.Files
    where file.Name.StartsWith("Foo")
    select file.Name into List;
// myList.GetType() == typeof(List<string>)

and for sum/count/average etc, we replace select

var avg = from file in ctx.Files
    average file.Name.Length;

maybe we want to keep select

var sum = from file in ctx.Files
    select sum file.Name.Length;

[gordanr]

@chrisaut

var myList = from file in ctx.Files
    where file.Name.StartsWith("Foo")
    select file.Name into List;

It solves 'extra braces' problem, but maybe makes other.
On the first look, 'into List' resembles me to put result into variable with name List.
That's way I proposed 'as List' (List=type).

What would be sum/count/average, new reserved or contextual keyword?
Maybe it's better Sum/Count/Average as 'Method' name.

[HaloFour]

@chrisaut The select into syntax is already used to project into a new range variable.

[weitzhandler]

#100 (comment)

I strongly vote for this one.

I never use LINQ language queries and instead my coding guidelines is avoid using them and always use the extension methods directly, and that's because the ugly parentheses each query has to be wrapped in order to materialize it (ToArray, ToList, Sum, SingleOrDefault etc.).

Until this issue is addressed, language-build-in LINQ is merely useless.
I really hope to see this implemented soon. Maybe to a more limited extent (avoiding the introduction of a gazillion new language keywords.

I'd say the syntax should provide an operator that expects a shortcutted extension method available for IEnumerable<TElement>, for instance:

//parents is Parent[]
var parents = from student in students
                     where student.Age < 18
                     select student.Parent
                     call ToArray()

//student is Student
var student = from st in students
                      call SingleOrDefault(st => st.Id == id);

Asynchronous methods should also be supported:

   var student = from st in students
                         call await SingleOrDefaultAsync(st => st.Id == id);

Maybe there should be a verbose LINQ fashioned way to pass the arguments and completely avoid the use of parenthesis, but I personally don't see it as necessary.

Anyway this feature is crucial for the completeness of the LINQ syntax.

Some suggestions above proposed the ToList at the beginning of the query, but that's a no go, since we want to be able to process it after the selection, and we don't want to be limited to parameterless ex. methods only. What if we wanna call ToLookup with a key selector, bottom line we can't tie it up to the language, we just need to find a way to call any Enumerable ex. methods on the current query state and treat its result as the final type of the query.

[miloush]

@weitzhandler So you would allow for

var young = from st in students
              select st
              call Where(st => st.Age < 18);

even when select st where st.Age < 18 is not allowed, but not this?

var young = from st in students
              call Where(st => st.Age < 18)
              select st;

What would be the reason not to allow the keyword everywhere?

var s = 14 call ToString()?

[weitzhandler]

@miloush
And what's the problem with calling of ToString?

Same about the duplicate Where.
It's like the user decides to write:

var query = from item in items
                   where item.Id == id
                   where item.Id == id
                   select item;
query = query.Where(item => item.Id == id);

[gafter]

We are now taking language feature discussion in other repositories:

• https://github.com/dotnet/csharplang for C# specific issues
• https://github.com/dotnet/vblang for VB-specific features
• https://github.com/dotnet/csharplang for features that affect both languages

Features that are under active design or development, or which are "championed" by someone on the language design team, have already been moved either as issues or as checked-in design documents. For example, the proposal in this repo "Proposal: Partial interface implementation a.k.a. Traits" (issue 16139 and a few other issues that request the same thing) are now tracked by the language team at issue 52 in https://github.com/dotnet/csharplang/issues, and there is a draft spec at https://github.com/dotnet/csharplang/blob/master/proposals/default-interface-methods.md and further discussion at issue 288 in https://github.com/dotnet/csharplang/issues. Prototyping of the compiler portion of language features is still tracked here; see, for example, https://github.com/dotnet/roslyn/tree/features/DefaultInterfaceImplementation and issue 17952.

In order to facilitate that transition, we have started closing language design discussions from the roslyn repo with a note briefly explaining why. When we are aware of an existing discussion for the feature already in the new repo, we are adding a link to that. But we're not adding new issues to the new repos for existing discussions in this repo that the language design team does not currently envision taking on. Our intent is to eventually close the language design issues in the Roslyn repo and encourage discussion in one of the new repos instead.

Our intent is not to shut down discussion on language design - you can still continue discussion on the closed issues if you want - but rather we would like to encourage people to move discussion to where we are more likely to be paying attention (the new repo), or to abandon discussions that are no longer of interest to you.

If you happen to notice that one of the closed issues has a relevant issue in the new repo, and we have not added a link to the new issue, we would appreciate you providing a link from the old to the new discussion. That way people who are still interested in the discussion can start paying attention to the new issue.

Also, we'd welcome any ideas you might have on how we could better manage the transition. Comments and discussion about closing and/or moving issues should be directed to #18002. Comments and discussion about this issue can take place here or on an issue in the relevant repo.

There are already a number of issues in the csharplang repo suggesting extensions to query expressions. See, for example, dotnet/csharplang#333.