3.0 regression: Translate Expression.Invoke

[joakimriedel]

Pre-3.0 I've been using Expression.Lambda to be able to re-use frequent expressions. When debugging another issue (#17617), I had to upgrade to 3.0 and noted that I cannot use the same kind of construct anymore.

The below code works in 2.2.6 without client evaluation, but not in 3.0 using daily builds. Am I using Expression.Lambda wrong, or is this not allowed in 3.0?

As a workaround, I have manually inlined all my queries dependent on re-used expressions. However, due to the resulting code duplication that will increase the risk of introducing bugs unintentionally in the code base.

Steps to reproduce

(the locator and predicate below are normally arguments and might be any kind of different expression in the actual code base)

Expression<Func<UserConnection, ApplicationUser>> locator = uc => uc.User;
Expression<Func<ApplicationUser, bool>> predicate = u => u.FirstName == "Somename";
var exp = Expression.Lambda<Func<UserConnection, bool>>(
    Expression.Invoke(predicate, locator.Body),
    locator.Parameters
);

var users = await context.UserConnections.Where(exp).ToListAsync();

System.InvalidOperationException
  HResult=0x80131509
  Message=The LINQ expression 'Where<TransparentIdentifier<UserConnection, ApplicationUser>>(
    source: Join<UserConnection, ApplicationUser, string, TransparentIdentifier<UserConnection, ApplicationUser>>(
        outer: DbSet<UserConnection>, 
        inner: DbSet<ApplicationUser>, 
        outerKeySelector: (u) => Property<string>(u, "UserId"), 
        innerKeySelector: (a) => Property<string>(a, "Id"), 
        resultSelector: (o, i) => new TransparentIdentifier<UserConnection, ApplicationUser>(
            Outer = o, 
            Inner = i
        )), 
    predicate: (u) => Invoke(u => (u.FirstName == "Somename"), u.Inner[ApplicationUser])
)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync(). See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
  Source=Microsoft.EntityFrameworkCore
  StackTrace:
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.<VisitMethodCall>g__CheckTranslated|8_0(ShapedQueryExpression translated, <>c__DisplayClass8_0& )
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.VisitMethodCall(MethodCallExpression methodCallExpression)
   at System.Linq.Expressions.MethodCallExpression.Accept(ExpressionVisitor visitor)
   at System.Linq.Expressions.ExpressionVisitor.Visit(Expression node)
   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass12_0`1.<ExecuteAsync>b__0()
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQueryCore[TFunc](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult](Expression query, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetAsyncEnumerator(CancellationToken cancellationToken)
   at System.Runtime.CompilerServices.ConfiguredCancelableAsyncEnumerable`1.GetAsyncEnumerator()
   at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.<ToListAsync>d__64`1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at EfCoreBugExpression2.Program.<Main>d__0.MoveNext() in C:\Users\Joakim\Source\Repos\EfCoreBug\EFCoreBugExpression2\Program.cs:line 28

Further technical details

EF Core version: 3.0.0-rc2
Database provider: SqlServer & InMemory
Target framework: .NET Core 3.0
Operating system: Win10
IDE: Visual Studio 2019 preview

[smitpatel]

Issue is

Expression.Invoke(predicate, locator.Body),

If you unwind the expression yourself it would work.

[Shane32]

This is quite simple to work around (and probably equally simple to fix in EF). You need to use the ExpressionVisitor class to visit the child lambda expression body and replace all references to the parameter with the parent lambda expression's body. Then create a new lambda expression with the new body and the parent expression's parameters. Here's the class wrote for my project:

using System;
using System.Linq.Expressions;

namespace MyExpressionHelpers
{
    public static class ParameterReplacer
    {
        // Produces an expression identical to 'expression'
        // except with 'source' parameter replaced with 'target' expression.     
        public static Expression Replace(Expression expression, ParameterExpression oldParameter, Expression newBody)
        {
            if (expression == null) throw new ArgumentNullException(nameof(expression));
            if (oldParameter == null) throw new ArgumentNullException(nameof(oldParameter));
            if (newBody == null) throw new ArgumentNullException(nameof(newBody));
            if (expression is LambdaExpression) throw new InvalidOperationException("The search & replace operation must be performed on the body of the lambda.");
            return (new ParameterReplacerVisitor(oldParameter, newBody)).Visit(expression);
        }

        //Chains two lambda expressions together as in the following example:
        //given these inputs:
        //  parentExpression = customer => customer.PrimaryAddress;
        //  childExpression = address => address.Street;
        //produces:
        //  customer => customer.PrimaryAddress.Street;
        //this function only supports parents with a single input parameter, and children with a single output parameter
        //many more overloads could be added for other common delegate types
        public static Expression<Func<A, C>> ChainWith<A, B, C>(this Expression<Func<A, B>> parentExpression, Expression<Func<B, C>> childExpression)
        {
            //could call Chain, but some of the checks are unnecessary since the inputs are strongly typed
            if (parentExpression == null) throw new ArgumentNullException(nameof(parentExpression));
            if (childExpression == null) throw new ArgumentNullException(nameof(childExpression));
            //since the lambda is strongly defined, we can be sure that there exists one and only one parameter on the parent and child expressions
            return Expression.Lambda<Func<A, C>>(
                Replace(childExpression.Body, childExpression.Parameters[0], parentExpression.Body), 
                parentExpression.Parameters);
        }

        //Chains two lambda expressions together as in the following example:
        //given these inputs:
        //  parentExpression = (customers, index) => customers[index].PrimaryAddress;
        //  childExpression = address => Console.WriteLine(address.Street);
        //produces:
        //  (customers, index) => Console.WriteLine(customers[index].PrimaryAddress.Street);
        //this function supports parent expressions with any number of input parameters (including 0), and child expressions with no output value (Action<>s)
        //however, it is not strongly typed, and validity cannot be verified at compile time
        public static LambdaExpression Chain(LambdaExpression parentExpression, LambdaExpression childExpression)
        {
            if (parentExpression == null) throw new ArgumentNullException(nameof(parentExpression));
            if (childExpression == null) throw new ArgumentNullException(nameof(childExpression));
            if (parentExpression.ReturnType.Equals(typeof(void))) throw new ArgumentException("The parent expression must return a value.", nameof(parentExpression));
            if (childExpression.Parameters.Count != 1 || !childExpression.Parameters[0].Type.Equals(parentExpression.ReturnType)) throw new ArgumentException("The child expression must have a single parameter of the same type as the parent expression's return type.", nameof(childExpression));
            //this code could provide add a conversion between compatible types, but for now just throws an error; the types must be identical
            return Expression.Lambda(Replace(childExpression.Body, childExpression.Parameters[0], parentExpression.Body), parentExpression.Parameters);
        }

        private class ParameterReplacerVisitor : ExpressionVisitor
        {
            private ParameterExpression _source;
            private Expression _target;

            public ParameterReplacerVisitor(ParameterExpression source, Expression target)
            {
                _source = source;
                _target = target;
            }

            protected override Expression VisitParameter(ParameterExpression node)
            {
                // Replace the source with the target, visit other params as usual.
                return node.Equals(_source) ? _target : base.VisitParameter(node);
            }
        }
    }
}

To use the class, just "chain" the parent and child expressions together before calling the Queryable methods, like this:

Expression<Func<UserConnection, ApplicationUser>> locator = uc => uc.User;
Expression<Func<ApplicationUser, bool>> predicate = u => u.FirstName == "Somename";
var exp = locator.ChainWith(predicate);

var users = await context.UserConnections.Where(exp).ToListAsync();

@smitpatel It would be great to see support for Expression.Invoke added back into EF Core 3.

[joakimriedel]

@Shane32 that's very useful, and seems to work like a charm. Thanks!

[fsmirne]

I'm still getting this error in version 3.1.0. I got here from #18333 and I'm also using PredicateBuilder.
In the simplest test case, the condition is the following (DebugView):

.Lambda #Lambda1<System.Func2[A,System.Boolean]>(A $f) { True && .Invoke (.Lambda #Lambda2<System.Func2[A,System.Boolean]>)($f)
}

.Lambda #Lambda2<System.Func2[A,System.Boolean]>(A $f) { False || .Invoke (.Lambda #Lambda3<System.Func2[A,System.Boolean]>)($f)
}

.Lambda #Lambda3<System.Func2[A,System.Boolean]>(A $d) { (System.Nullable1[System.Int32])$d.Name == (.Constant<TestController+<>c__DisplayClass21_0>(TestController+<>c__DisplayClass21_0).model).Name
}

When I call a .Count() on this predicate, I get "The Linq expression ... could not be translated..." message.