Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query: Translate IQueryable.Concat/Union/Intersect/Except/etc. to server #6812

Closed
simpleway2016 opened this issue Oct 19, 2016 · 49 comments
Closed
Assignees
Labels
closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. punted-for-2.1 type-enhancement
Milestone

Comments

@simpleway2016
Copy link

EF version:1.0.1.0

var q1 = from m in db.Table1
                     select new
                     {
                         name = m.Name,
                     };
            var q2 = from m in db.Table2
                     select new
                     {
                         name = m.Name,
                     };
            q1 = q1.Concat(q2);
            var result = q1.ToArray();

I watch the sql string by ILogger, the sql string is not union two table, the dbconnection will excute two dbcommand for each table like this:
command 1:

SELECT "m"."Name"
FROM "Table1" AS "m"

command 2:

SELECT "m"."Name"
FROM "Table2" AS "m"

at EF 6.1 version, it's make a sql string like this:

(SELECT "m"."Name"
FROM "Table1" AS "m") union
(SELECT "m"."Name"
FROM "Table2" AS "m")
@divega
Copy link
Contributor

divega commented Oct 19, 2016

@simpleway2016 In general the LINQ implementation of EF Core only attempts to translate specific patterns to equivalent SQL expressions for relational databases.

In the particular case of Queryable.Concat() an appropriate translation could be two SQL queries with UNION ALL in between but instead we concatenate the results in memory.

Changing this to producing a single SQL query on the server isn't currently a top priority because we believe that for many common scenarios (e.g. for the sample queries you posted) there isn't a significant difference.

Could you elaborate on what scenario you think performing the union on the server would be more useful for you?

@divega divega removed this from the 1.2.0 milestone Oct 19, 2016
@divega divega removed their assignment Oct 19, 2016
@simpleway2016
Copy link
Author

simpleway2016 commented Oct 20, 2016

var q1 = from m in db.Table1
                     select new
                     {
                         name = m.Name,
                     };
            var q2 = from m in db.Table2
                     select new
                     {
                         name = m.Name,
                     };
            q1 = q1.Concat(q2);
var result = q1.OrderBy(m=>m.Name).Skip(1000).Take(10).ToArray();

If i use the ef core like that,it also make two sql string , excute two dbcommand,and concatenate the results in memory, and then excute "OrderBy(m=>m.Name).Skip(1000).Take(10)",
I think it's bad performance

@rowanmiller rowanmiller changed the title the IQueryable.Concat<> method is incorrect Query: Translate IQueryable.Concat<> Oct 24, 2016
@rowanmiller rowanmiller added this to the Backlog milestone Oct 24, 2016
@rowanmiller rowanmiller changed the title Query: Translate IQueryable.Concat<> Query: Translate IQueryable.Concat/Union/Intersect/Except/etc. Oct 24, 2016
@negatedx
Copy link

I would really like to see this feature too. As @simpleway2016 said above, paging results of a union in the database is much more efficient.

@smitpatel smitpatel changed the title Query: Translate IQueryable.Concat/Union/Intersect/Except/etc. Query: Translate IQueryable.Concat/Union/Intersect/Except/etc. to server May 30, 2017
@kingmotley
Copy link

It would have a significant performance difference if there is a large amount of latency between the client and the server (such as being in a different timezone).

@nicholasyin
Copy link

Consider a banking application where it needs to display a number of types of transactions, each of which has its own specific properties but also shares some common data with other types, in one page, with paging and sorting. This is a perfect case where translating the concat query into one union all statement would be needed.

@John0King
Copy link

Please put this feature on a higher priority ,We really need this feature, it should have much higher priority than Lazy loading, Imagine there are 100,000 rows of data and we only need top 10

@John0King
Copy link

@ajcvickers what's the state of this ?

@ajcvickers
Copy link
Contributor

@John0King It's still scheduled for 2.1, but as far as I am aware @smitpatel has not implemented it yet. (It's always possible it won't make 2.1 given we don't know the full schedule yet.)

@djuuuuus
Copy link

Prompt to me, where I can read all supported things for EF Core? (e.t. Concat....)

@ajcvickers ajcvickers modified the milestones: 2.1.0-preview1, 2.1.0 Jan 17, 2018
roji added a commit that referenced this issue Jun 25, 2019
Navigation/include support not included.

Fixes #6812
Fixes #12549
@roji roji closed this as completed in 16c9a09 Jun 25, 2019
@kingmotley
Copy link

kingmotley commented Jun 25, 2019

@roji please reverify that SQL Server does not support take/order by within subset operations. I believe it should, but not if you only specify order by without a top/take.

I believe:
(SELECT TOP 1 1 as x) UNION (SELECT TOP 1 2)
and
(SELECT TOP 1 1 as x ORDER BY 1) UNION (SELECT TOP 1 2)
should both work, while
(SELECT 1 as x ORDER BY 1) UNION (SELECT TOP 1 2)
makes very little sense because the union doesn’t guarantee order, so having the order by in the first subset operation is useless.

@roji
Copy link
Member

roji commented Jun 25, 2019

@kingmotley trying to run your 2nd query ((SELECT TOP 1 1 as x ORDER BY 1) UNION (SELECT TOP 1 2)) errors with "Incorrect syntax near the keyword 'ORDER'". (MSSQL 2017). TOP is indeed allowed in set operands (as I wrote above).

Are you seeing a different behavior?

roji added a commit to roji/efcore that referenced this issue Jun 25, 2019
* Include/nav rewriting is now supported.
* We now push down to subquery when OrderBy, Take or Skip are applied to set operations, to avoid using the hack where ColumnExpression with no table alias was used. dotnet#16244 was opened to track for post-3.0.
* Added missing support for union over subselect projection mappings.
* Added Other type to SetOperationType so that providers can define extra set operations (e.g. PostgreSQL `INTERSECT ALL`, `EXCEPT ALL`).

Completes dotnet#6812
Fixes dotnet#13196
Fixes dotnet#16065
Fixes dotnet#16165
roji added a commit that referenced this issue Jun 25, 2019
* Include/nav rewriting is now supported.
* We now push down to subquery when OrderBy, Take or Skip are applied to set operations, to avoid using the hack where ColumnExpression with no table alias was used. #16244 was opened to track for post-3.0.
* Added missing support for union over subselect projection mappings.
* Added Other type to SetOperationType so that providers can define extra set operations (e.g. PostgreSQL `INTERSECT ALL`, `EXCEPT ALL`).

Completes #6812
Fixes #13196
Fixes #16065
Fixes #16165
@kingmotley
Copy link

Sorry, I didn't have access to an instance of Sql Server when I originally posted, but was pretty certain that you could do two unions of a limited set. I assumed (wrongly) that since views have the requirement that they can only have a TOP clause when there is also an ORDER BY, then the 2nd query would work. But in practice, using a view like that in a query (and then inlining it) would look more like the following, which does work:

SELECT * FROM (SELECT TOP 1 1 as x ORDER BY 1) t1 UNION SELECT * FROM (SELECT TOP 1 2 as x ORDER BY 1) t2

roji added a commit to roji/efcore that referenced this issue Jun 26, 2019
* Include/nav rewriting is now supported.
* We now push down to subquery when OrderBy, Take or Skip are applied to set operations, to avoid using the hack where ColumnExpression with no table alias was used. dotnet#16244 was opened to track for post-3.0.
* Added missing support for union over subselect projection mappings.
* Added Other type to SetOperationType so that providers can define extra set operations (e.g. PostgreSQL `INTERSECT ALL`, `EXCEPT ALL`).

Completes dotnet#6812
Fixes dotnet#13196
Fixes dotnet#16065
Fixes dotnet#16165
roji added a commit to roji/efcore that referenced this issue Jun 27, 2019
* Include/nav rewriting is now supported.
* We now push down to subquery when OrderBy, Take or Skip are applied to set operations, to avoid using the hack where ColumnExpression with no table alias was used. dotnet#16244 was opened to track for post-3.0.
* Added missing support for union over subselect projection mappings.
* Added Other type to SetOperationType so that providers can define extra set operations (e.g. PostgreSQL `INTERSECT ALL`, `EXCEPT ALL`).

Completes dotnet#6812
Fixes dotnet#13196
Fixes dotnet#16065
Fixes dotnet#16165
roji added a commit that referenced this issue Jun 27, 2019
* Include/nav rewriting is now supported.
* We now push down to subquery when OrderBy, Take or Skip are applied to set operations, to avoid using the hack where ColumnExpression with no table alias was used. #16244 was opened to track for post-3.0.
* Added missing support for union over subselect projection mappings.
* Added Other type to SetOperationType so that providers can define extra set operations (e.g. PostgreSQL `INTERSECT ALL`, `EXCEPT ALL`).

Completes #6812
Fixes #13196
Fixes #16065
Fixes #16165
roji added a commit to roji/efcore that referenced this issue Jun 27, 2019
* Include/nav rewriting is now supported.
* We now push down to subquery when OrderBy, Take or Skip are applied to set operations, to avoid using the hack where ColumnExpression with no table alias was used. dotnet#16244 was opened to track for post-3.0.
* Added missing support for union over subselect projection mappings.
* Added Other type to SetOperationType so that providers can define extra set operations (e.g. PostgreSQL `INTERSECT ALL`, `EXCEPT ALL`).

Completes dotnet#6812
Fixes dotnet#13196
Fixes dotnet#16065
Fixes dotnet#16165
roji added a commit that referenced this issue Jun 27, 2019
* Include/nav rewriting is now supported.
* We now push down to subquery when OrderBy, Take or Skip are applied to set operations, to avoid using the hack where ColumnExpression with no table alias was used. #16244 was opened to track for post-3.0.
* Added missing support for union over subselect projection mappings.
* Added Other type to SetOperationType so that providers can define extra set operations (e.g. PostgreSQL `INTERSECT ALL`, `EXCEPT ALL`).

Completes #6812
Fixes #13196
Fixes #16065
Fixes #16165
roji added a commit that referenced this issue Jun 28, 2019
* Include/nav rewriting is now supported.
* We now push down to subquery when OrderBy, Take or Skip are applied to set operations, to avoid using the hack where ColumnExpression with no table alias was used. #16244 was opened to track for post-3.0.
* Added missing support for union over subselect projection mappings.
* Added Other type to SetOperationType so that providers can define extra set operations (e.g. PostgreSQL `INTERSECT ALL`, `EXCEPT ALL`).

Completes #6812
Fixes #13196
Fixes #16065
Fixes #16165
roji added a commit that referenced this issue Jun 28, 2019
* We no longer allow ColumnExpression without table alias
* Set operations over operands with different types have been disabled
  for now.
* We now push down set operations into a subquery on Orderby, Skip or
  Take, which shouldn't be necessary (#16244).
* Some test consolidation and cleanup.

Continues #6812
roji added a commit that referenced this issue Jun 28, 2019
* We no longer allow ColumnExpression without table alias
* Set operations over operands with different types have been disabled
  for now.
* We now push down set operations into a subquery on Orderby, Skip or
  Take, which shouldn't be necessary (#16244).
* Some test consolidation and cleanup.

Continues #6812
roji added a commit that referenced this issue Jun 28, 2019
* We no longer allow ColumnExpression without table alias
* Set operations over operands with different types have been disabled
  for now.
* We now push down set operations into a subquery on Orderby, Skip or
  Take, which shouldn't be necessary (#16244).
* Some test consolidation and cleanup.

Continues #6812
roji added a commit that referenced this issue Jun 28, 2019
roji added a commit that referenced this issue Jun 28, 2019
roji added a commit that referenced this issue Jul 1, 2019
roji added a commit that referenced this issue Jul 1, 2019
@ajcvickers ajcvickers added the closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. label Jul 2, 2019
@ajcvickers ajcvickers modified the milestones: 3.0.0, 3.0.0-preview7 Jul 2, 2019
@ajcvickers ajcvickers modified the milestones: 3.0.0-preview7, 3.0.0 Nov 11, 2019
@chester89
Copy link

chester89 commented Feb 17, 2020

Guys, you should try https://github.com/tuespetre/Impatient

That's good suggestion for Sql Server, but we use MySQL

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
closed-fixed The issue has been fixed and is/will be included in the release indicated by the issue milestone. punted-for-2.1 type-enhancement
Projects
None yet
Development

No branches or pull requests