GroupJoin to parent with no child throws invalid operation exception

[knightpfhor]

I have a query that selects parent records sets all child records to be a property. This query work in EF Core 1.1 even if it was rather inefficient and performed the matching in memory. With EF Core 2.0 it is now efficiently running just one query and attempting to build the objects from the one result set. This works fine so long as all parent records have a child record. If one of the parents does not have a child record it throws the following error.

System.InvalidOperationException occurred
  HResult=0x80131509
  Message=An exception occurred while reading a database value for property 'Child.ParentId'. The expected type was 'System.Int32' but the actual value was null.
  Source=Microsoft.EntityFrameworkCore
  StackTrace:
   at Microsoft.EntityFrameworkCore.Metadata.Internal.EntityMaterializerSource.ThrowReadValueException[TValue](Exception exception, Object value, IPropertyBase property)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.EntityMaterializerSource.TryReadValue[TValue](ValueBuffer valueBuffer, Int32 index, IPropertyBase property)
   at Microsoft.EntityFrameworkCore.Query.ExpressionVisitors.Internal.ProjectionShaper.TypedProjectionShaper`3.Shape(QueryContext queryContext, ValueBuffer valueBuffer)
   at Microsoft.EntityFrameworkCore.Query.QueryMethodProvider.<_GroupJoin>d__23`4.MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at EFCoreGroupJoin.Program.Main(String[] args) in C:\Git\GitHub\EFCore2GroupJoinIssue\EFCoreGroupJoin\Program.cs:line 42

Inner Exception 1:
NullReferenceException: Object reference not set to an instance of an object.

Steps to reproduce

If have created a sample application (with EF migrations) that exhibits this issue, the repo is available here

The query that is causing the problem is:

var results = context.Parents.GroupJoin(
    context.Children.Select(x => new
    {
        ParentId = x.ParentId,
        OtherParent = x.OtherParent.Name
    }),
    p => p.Id,
    c => c.ParentId,
    (parent, child) => new
    {
        ParentId = parent.Id,
        ParentName = parent.Name,
        Children = child.Select(x => x.OtherParent)
    });

if I materialise the child query and then pass it into the group join, then it works as expected i.e:

var preCalcChildren = context.Children.Select(x => new
{
    ParentId = x.ParentId,
    OtherParent = x.OtherParent.Name
}).ToList();

var results = context.Parents.GroupJoin(
    preCalcChildren,
    p => p.Id,
    c => c.ParentId,
    (parent, child) => new
    {
        ParentId = parent.Id,
        ParentName = parent.Name,
        Children = child.Select(x => x.OtherParent)
    });

I have other very similar queries which work as expected, however they return an EF typed object from the first parameter to the GroupJoin method. If it is an anonymous type (like here) or another POCO, I get this error.

If have tried the following query

var results = context.Parents.GroupJoin(
    context.Children,
    p => p.Id,
    c => c.ParentId,
    (parent, child) => new
    {
        ParentId = parent.Id,
        ParentName = parent.Name,
        Children = child.Select(x => x.OtherParent.Name)
    });

But this seems to query OtherParent for each Parent that has at least one Child.

Further technical details

EF Core version: 2.0.0
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: Win10
IDE: Visual Studio 2017 (15.3.5)

[ajcvickers]

Notes for triage: I was able to reproduce this and the data in the database looks valid. This is reported as a regression, so assigning to @smitpatel to do some initial investigation and putting in 2.0.1 until we know more.

[smitpatel]

I downloaded zip file from repro. Updated connection strings. Ran 'dotnet ef database update' and 'dotnet run' it passes successfully. What am i missing.

[ajcvickers]

@smitpatel Did you set ensureOneParentHasNoChildren to true in the repro?

[smitpatel]

I got repro.

[ajcvickers]

@smitpatel Can you write the patch Risk/Justification for this?

[smitpatel]

Risk: Low
Impact: When EF Core is evaluating a GroupJoin query without DefaultIfEmpty where Inner source is non-entity type, we combine both the queries to generate Left Join but the inner shaper is null safe which would be only generated due to left join now causing exception like above.
Justification: Fix is to identify only those cases when the shaper is not on entitytype and don't combine queries.

[Eilon]

This patch bug is approved for the 2.0.x patch. Please send a PR to the rel/2.0.1 branch and get it reviewed and merged.

[Eilon]

Hi, we have a public test feed that you can use to try out the ASP.NET/EF Core 2.0.3 patch!

To try out the pre-release patch, please refer to the following guide:

• ASP.NET/EF Core 2.0.3 patch pre-release test feed aspnet/Announcements#274 (info)
• ASP.NET/EF Core 2.0.3 patch pre-release test feed - discussion aspnetcore#2254 (discussions and questions)

We are looking for feedback on this patch. We'd like to know if you have any issues with this patch by updating your apps and libraries to the latest packages and seeing if it fixes the issues you've had, or if it introduces any new issues. If you have any issues or questions, please reply on this issue to let us know as soon as possible.

Thanks,
Eilon

[knightpfhor]

I can confirm that I no longer get this error. A little annoyingly it still runs two queries to get the results, but I'll take that over an error.

[mchenx]

Still having this issue with 2.0.1 packages (including Microsoft.EntityFrameworkCore.SqlServer 2.0.1)... here is my package reference list

  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore" Version="2.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="2.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.Identity" Version="2.0.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="2.0.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="2.0.1">
      <PrivateAssets>All</PrivateAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.AspNetCore.Mvc" Version="2.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.Server.Kestrel" Version="2.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.Server.IISIntegration" Version="2.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.StaticFiles" Version="2.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.Identity.EntityFrameworkCore" Version="2.0.1" />
    <PackageReference Include="Microsoft.AspNetCore.Cors" Version="2.0.1" />
    <PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="2.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.FileExtensions" Version="2.0.0" />
    <PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="2.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging" Version="2.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="2.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Debug" Version="2.0.0" />
    <PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="2.0.0" />
    <PackageReference Include="System.IdentityModel.Tokens.Jwt" Version="5.1.5" />
    <PackageReference Include="System.Security.Cryptography.Csp" Version="4.3.0" />
  </ItemGroup>

dotnet 2.0.3
Entity Framework Core .NET Command Line Tools 2.0.1-rtm-125

[ajcvickers]

@mchenx Can you post a new issue with a runnable project/solution or code listing that reproduces what you are seeing?