ArgumentOutOfRangeException upon certain queries with includes

[joshcomley]

Certain queries with includes are causing an ArgumentOutOfRangeException.

When using OData, almost (or possibly) all queries with more than one expand of a collection and translating to SQL fails with the following error:

Steps to reproduce

A very quick repro, download this zip, extract it in the src folder, open All.sln, add the zipped project as an existing project, make sure the project references are set correctly and run. It will create a local SQL database on your machine. And it will fail to run the following query:

http://localhost:15580/odata/Movies?$expand=Stars,Writers

If you remove either one of the expands, it works. I wasn't sure whether to post to OData or here, but I think the issue lies here.

Further technical details

EF Core version: 3.1.1
Database provider: Microsoft.EntityFrameworkCore.SqlServer
Target framework: .NET Core 3.0
Operating system: Windows 10
IDE: Visual Studio 2019 16.5 Preview 1

[joshcomley]

I've managed to significantly simplify the reproduction of this error:

var movies = _context.Movies.Select(movie => new 
    {
        Value1 = movie.Stars.Select(star => new
        {
            Anything = true
        }),
        Value2 = movie.Stars.Select(writer => new
        {
            AnythingElse = true
        })
    })
    .ToList();

This code will throw it, you just need to substitute Movies and Movies.Stars for whatever example data structure you'd like, where Stars is a many side of a one-to-many relationship.

If you remove either one of Anything or AnythingElse, the query works.

Edit: removed unnecessary extra entity type

[joshcomley]

Useful additional info (the ones that fail have the same compile-time value for the projected property):

Fails

var movies = _context.Movies.Select(movie => new 
    {
        Value1 = movie.Stars.Select(star => new
        {
            Anything = 1
        }),
        Value2 = movie.Stars.Select(writer => new
        {
            AnythingElse = 1
        })
    })
    .ToList();


var movies = _context.Movies.Select(movie => new 
    {
        Value1 = movie.Stars.Select(star => new
        {
            Anything = false
        }),
        Value2 = movie.Stars.Select(writer => new
        {
            AnythingElse = false
        })
    })
    .ToList();

Succeeds

var movies = _context.Movies.Select(movie => new 
    {
        Value1 = movie.Stars.Select(star => new
        {
            Anything = 1
        }),
        Value2 = movie.Stars.Select(writer => new
        {
            AnythingElse = 2
        })
    })
    .ToList();


var movies = _context.Movies.Select(movie => new 
    {
        Value1 = movie.Stars.Select(star => new
        {
            Anything = true
        }),
        Value2 = movie.Stars.Select(writer => new
        {
            AnythingElse = false
        })
    })
    .ToList();

[joshcomley]

I believe I have fixed this bug, I am just organising a pull-request

[joshcomley]

Could someone please help guide me where would be best to write a test for this? I am looking in EFCore.Relational.Tests but am unable to find exactly where or how I should put this in. I know I'm supposed to submit a test with every PR, but it's going to take me half a day just to work out the test structure!

[ajcvickers]

@maumar Can you point @joshcomley to a good place for a test?

[maumar]

ComplexNavigationsQueryTestBase is a good place, it has the model structure needed.

[joshcomley]

Thanks @maumar - is there any way to actually debug the test? I want to see what the output SQL for the SQL test is, but it's all very abstracted. I managed to run tests via command line, but I get a lot of Cannot open database "Northwind" requested by the login errors.

[maumar]

The process I follow is run build.cmd on the efcore repo, then open Runtime.sln using startvs.cmd, then open Test Explorer, find the test you want to run, right click -> debug.

To see the generated sql, you can override the test method in ComplexNavigationsQuerySqlServerTest, call base and use AssertSql("") method afterwards. The generated sql is compared against the baseline and if they don't match you can see the actually generated sql in QueryBaseline.txt file in the root directory of efcore repo.

Do you have localdb installed in your environment? Tests run against it by default and should automatically create all the necessary databases.

[cornem]

Could this please be fixed and released ASAP? Right now this is blocking anyone that relies on OData from upgrading to .NET Core 3.1. We cannot upgrade our APIs, because client apps are doing multiple expands against our endpoints, which apparently no longer work because of this bug.

So we're essentially stuck on the - now deprecated - .NET Core 2.2 until this is fixed.

[ajcvickers]

@maumar @smitpatel Are there any workarounds here?

@cornem There is a community contribution to fix this from @joshcomley. Now that @smitpatel is back we are working through that PR and hope to get it merged soon. We don't want to jump in and take over because that can be quite demoralizing to the contributor who has been working hard on a fix.

Once we have the fix we will consider if for a patch release as described in the release planning process.

[smitpatel]

No work-around since query is generated by ODATA dynamically.

[jasonsheridan79]

I'm with @cornem , I'm completely blocked by this. OData calls that were fine in 2.2 are now seeing "No coercion operator is defined between types 'System.Int32' and 'System.Boolean'" and " Index was out of range. Must be non-negative and less than the size of the collection." errors.

[DFSko]

my solution has 10 projects, 9 of them upgraded to 3.1 and 1 on 2.2 version (because Odata blocking update). Checking this issue everyday

[Landsec-HadrianP]

This is also a blocker for our application upgrade to 3.1, desperately waiting for a fix.

[nikolai-mb]

I also have several projects blocked by this. @ajcvickers it seems like @joshcomley might not be available as he was last active on github january 8. Is it possible for @smitpatel to take over this issue in order to speed up the process of fixing this?

[ajcvickers]

@nikolai-mb Yes, @smitpatel will take over here.

[smitpatel]

Re-opening to make sure that we merge this in 3.1 too.

[jannikbuschke]

This fix seems to not been released, despite being a blocker for many to upgrade to dotnet 3.1. The issue was reported on January. Now is mai. The fix seems to be merged already, but 3.1.4 is not released yet. Can this be prioritized please?

[ajcvickers]

@jannikbuschke Patch releases scheduled are unified for all of .NET Core. 3.1.4 is being validated as part of this process and will be shipped with the other 3.1.x patches.

[jannikbuschke]

Sorry for ranting a bit here, but I hope in the future odata, aspnetcore and sql server ef core will be better streamlined and the basic usages such as described in this issue will not break.

And I hope for 3.1.4 to happen soonish, there is no planned date yet or?

[ajcvickers]

@jannikbuschke I'm not at liberty to give an exact date, but it is planned for this month.

[gunter79]

With the 3.1.4 release available, has anybody got it working ? I still get the same error when querying two lists using expands in my query.

[ajcvickers]

@gunter79 We may have missed something, or you may be hitting a different bug. Please file a new issue and attach a small, runnable project or post a small, runnable code listing that reproduces what you are seeing so that we can investigate.

[gunter79]

👍
I must apologize, after upgrading all my nugget packages (Microsoft.EntityFrameworkCore and
Microsoft.EntityFrameworkCore.SqlServer), the 3.1.4 release solves it.
I'm an happy camper now. Thank you very much, you did an awesome job guys !