[Regression] Unexpected unbound variable error with closure

[BladeWise]

Using EFCore 9 the following code (which uses a closure in an order-by condition)

static async Task OrderByClosureInt(SampleDbContext dbContext)
{
    int v = 1;
    Expression<Func<Dummy, object>> f = x => v;
    var q = dbContext.Set<Dummy>()
                     .OrderBy(f);

    await q.FirstOrDefaultAsync();
}

throws at runtime with the following error:

System.InvalidOperationException: An exception was thrown while attempting to evaluate the LINQ query parameter expression 'x => Convert(__v_0, Object)'. See the inner exception for more information.
       ---> System.InvalidOperationException: unbound variable: __v_0

Stacktrace

      System.InvalidOperationException: An exception was thrown while attempting to evaluate the LINQ query parameter expression 'x => Convert(__v_0, Object)'. See the inner exception for more information.
       ---> System.InvalidOperationException: unbound variable: __v_0
         at System.Linq.Expressions.Interpreter.LightCompiler.EnsureAvailableForClosure(ParameterExpression expr)
         at System.Linq.Expressions.Interpreter.LightCompiler.CompileQuoteUnaryExpression(Expression expr)
         at System.Linq.Expressions.Interpreter.LightCompiler.Compile(Expression expr)
         at System.Linq.Expressions.Interpreter.LightCompiler.CompileConvertUnaryExpression(Expression expr)
         at System.Linq.Expressions.Interpreter.LightCompiler.Compile(Expression expr)
         at System.Linq.Expressions.Interpreter.LightCompiler.CompileTop(LambdaExpression node)
         at System.Linq.Expressions.Expression`1.Compile(Boolean preferInterpretation)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.<Evaluate>g__EvaluateCore|70_0(Expression expression, String& parameterName, Boolean& isContextAccessor)
         --- End of inner exception stack trace ---
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.<Evaluate>g__EvaluateCore|70_0(Expression expression, String& parameterName, Boolean& isContextAccessor)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.Evaluate(Expression expression, String& parameterName, Boolean& isContextAccessor)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.ProcessEvaluatableRoot(Expression evaluatableRoot, State& state, Boolean forceEvaluation)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.EvaluateList(IReadOnlyList`1 expressions, State[] expressionStates, List`1& children, Func`2 pathFromParentGenerator)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.VisitMethodCall(MethodCallExpression methodCall)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.Visit(Expression expression)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.Visit[T](ReadOnlyCollection`1 expressions, Func`2 elementVisitor, StateType& aggregateStateType, State[]& expressionStates, Boolean poolExpressionStates)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.Visit(ReadOnlyCollection`1 expressions, StateType& aggregateStateType, State[]& expressionStates, Boolean poolExpressionStates)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.VisitMethodCall(MethodCallExpression methodCall)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.Visit(Expression expression, State& state)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.ExtractParameters(Expression expression, IParameterValues parameterValues, Boolean parameterize, Boolean clearParameterizedValues, Boolean precompiledQuery, IReadOnlySet`1& nonNullableReferenceTypeParameters)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.ExtractParameters(Expression expression, IParameterValues parameterValues, Boolean parameterize, Boolean clearParameterizedValues)
         at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExtractParameters(Expression query, IParameterValues parameterValues, IDiagnosticsLogger`1 logger, Boolean compiledQuery, Boolean generateContextAccessors)
         at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteCore[TResult](Expression query, Boolean async, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult](Expression query, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, Expression expression, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.FirstOrDefaultAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
         at Program.<<Main>$>g__OrderByClosureInt|0_2(SampleDbContext dbContext) in C:\repos\EFIssueRepo\EFIssueRepo\Program.cs:line 67
         at Program.<Main>$(String[] args) in C:\repos\EFIssueRepo\EFIssueRepo\Program.cs:line 52

The same code completes successfully using EFCore 8.
Moreover, if the boxing is moved out of the expression like below

static async Task OrderByClosureInt(SampleDbContext dbContext)
{
    object v = 1;
    Expression<Func<Dummy, object>> f = x => v;
    var q = dbContext.Set<Dummy>()
                     .OrderBy(f);

    await q.FirstOrDefaultAsync();
}

the queryable is interpreted successfully.

The database provider is not relevant, because the exception is thrown at interpretation time (nevertheless, I have tested with both InMemory and PostgreSQL).

I have created a gist to reproduce the issue

Gist output

dbug: Microsoft.EntityFrameworkCore.Infrastructure[10401]
      An 'IServiceProvider' was created for internal use by Entity Framework.
warn: Microsoft.EntityFrameworkCore.Model.Validation[10400]
      Sensitive data logging is enabled. Log entries and exception messages may include sensitive application data; this mode should only be enabled during development.
dbug: Microsoft.EntityFrameworkCore.Infrastructure[10403]
      Entity Framework Core 9.0.0 initialized 'SampleDbContext' using provider 'Microsoft.EntityFrameworkCore.InMemory:9.0.0' with options: SensitiveDataLoggingEnabled StoreName=test
dbug: Microsoft.EntityFrameworkCore.ChangeTracking[10806]
      Context 'SampleDbContext' started tracking 'Dummy' entity with key '{Id: 1}'.
dbug: Microsoft.EntityFrameworkCore.Update[10004]
      SaveChanges starting for 'SampleDbContext'.
dbug: Microsoft.EntityFrameworkCore.ChangeTracking[10800]
      DetectChanges starting for 'SampleDbContext'.
dbug: Microsoft.EntityFrameworkCore.ChangeTracking[10801]
      DetectChanges completed for 'SampleDbContext'.
info: Microsoft.EntityFrameworkCore.Update[30100]
      Saved 1 entities to in-memory store.
dbug: Microsoft.EntityFrameworkCore.ChangeTracking[10807]
      The 'Dummy' entity with key '{Id: 1}' tracked by 'SampleDbContext' changed state from 'Added' to 'Unchanged'.
dbug: Microsoft.EntityFrameworkCore.Update[10005]
      SaveChanges completed for 'SampleDbContext' with 1 entities written to the database.
info: Program[0]
      Executing test OrderByClosureInt...
fail: Program[0]
      Test OrderByClosureInt failed
      System.InvalidOperationException: An exception was thrown while attempting to evaluate the LINQ query parameter expression 'x => Convert(__v_0, Object)'. See the inner exception for more information.
       ---> System.InvalidOperationException: unbound variable: __v_0
         at System.Linq.Expressions.Interpreter.LightCompiler.EnsureAvailableForClosure(ParameterExpression expr)
         at System.Linq.Expressions.Interpreter.LightCompiler.CompileQuoteUnaryExpression(Expression expr)
         at System.Linq.Expressions.Interpreter.LightCompiler.Compile(Expression expr)
         at System.Linq.Expressions.Interpreter.LightCompiler.CompileConvertUnaryExpression(Expression expr)
         at System.Linq.Expressions.Interpreter.LightCompiler.Compile(Expression expr)
         at System.Linq.Expressions.Interpreter.LightCompiler.CompileTop(LambdaExpression node)
         at System.Linq.Expressions.Expression`1.Compile(Boolean preferInterpretation)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.<Evaluate>g__EvaluateCore|70_0(Expression expression, String& parameterName, Boolean& isContextAccessor)
         --- End of inner exception stack trace ---
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.<Evaluate>g__EvaluateCore|70_0(Expression expression, String& parameterName, Boolean& isContextAccessor)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.Evaluate(Expression expression, String& parameterName, Boolean& isContextAccessor)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.ProcessEvaluatableRoot(Expression evaluatableRoot, State& state, Boolean forceEvaluation)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.EvaluateList(IReadOnlyList`1 expressions, State[] expressionStates, List`1& children, Func`2 pathFromParentGenerator)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.VisitMethodCall(MethodCallExpression methodCall)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.Visit(Expression expression)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.Visit[T](ReadOnlyCollection`1 expressions, Func`2 elementVisitor, StateType& aggregateStateType, State[]& expressionStates, Boolean poolExpressionStates)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.Visit(ReadOnlyCollection`1 expressions, StateType& aggregateStateType, State[]& expressionStates, Boolean poolExpressionStates)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.VisitMethodCall(MethodCallExpression methodCall)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.Visit(Expression expression, State& state)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.ExtractParameters(Expression expression, IParameterValues parameterValues, Boolean parameterize, Boolean clearParameterizedValues, Boolean precompiledQuery, IReadOnlySet`1& nonNullableReferenceTypeParameters)
         at Microsoft.EntityFrameworkCore.Query.Internal.ExpressionTreeFuncletizer.ExtractParameters(Expression expression, IParameterValues parameterValues, Boolean parameterize, Boolean clearParameterizedValues)
         at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExtractParameters(Expression query, IParameterValues parameterValues, IDiagnosticsLogger`1 logger, Boolean compiledQuery, Boolean generateContextAccessors)
         at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteCore[TResult](Expression query, Boolean async, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.ExecuteAsync[TResult](Expression query, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryProvider.ExecuteAsync[TResult](Expression expression, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, Expression expression, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.ExecuteAsync[TSource,TResult](MethodInfo operatorMethodInfo, IQueryable`1 source, CancellationToken cancellationToken)
         at Microsoft.EntityFrameworkCore.EntityFrameworkQueryableExtensions.FirstOrDefaultAsync[TSource](IQueryable`1 source, CancellationToken cancellationToken)
         at Program.<<Main>$>g__OrderByClosureInt|0_2(SampleDbContext dbContext) in C:\repos\EFIssueRepo\EFIssueRepo\Program.cs:line 67
         at Program.<Main>$(String[] args) in C:\repos\EFIssueRepo\EFIssueRepo\Program.cs:line 52
info: Program[0]
      Executing test OrderByClosureObject...
dbug: Microsoft.EntityFrameworkCore.Query[10111]
      Compiling query expression:
      'DbSet<Dummy>()
          .OrderBy(x => __v_0)
          .FirstOrDefault()'
dbug: Microsoft.EntityFrameworkCore.Query[10107]
      Generated query execution expression:
      'queryContext => ShapedQueryCompilingExpressionVisitor.SingleOrDefaultAsync<Dummy>(
          asyncEnumerable: new QueryingEnumerable<Dummy>(
              queryContext,
              new ResultEnumerable(() => InMemoryShapedQueryCompilingExpressionVisitor.Table(
                  queryContext: queryContext,
                  entityType: EntityType: Dummy)
                  .OrderBy(valueBuffer => InMemoryExpressionTranslatingExpressionVisitor.GetParameterValue<object>(
                      queryContext: queryContext,
                      parameterName: "__v_0"))
                  .Select(valueBuffer => new ValueBuffer(new object[]{ (object)ExpressionExtensions.ValueBufferTryReadValue<int>(
                      valueBuffer: valueBuffer,
                      index: 0,
                      property: Property: Dummy.Id (int) Required PK AfterSave:Throw ValueGenerated.OnAdd) }))
                  .FirstOrDefault()),
              Func<QueryContext, ValueBuffer, Dummy>,
              SampleDbContext,
              False,
              True
          ),
          cancellationToken: queryContext.CancellationToken)'
info: Program[0]
      Test OrderByClosureObject succeeded
dbug: Microsoft.EntityFrameworkCore.Infrastructure[10407]
      'SampleDbContext' disposed.

[roji]

Confirmed regression from 8.0 to 9.0 - another thing to come out of the funcletizer rewrite, most likely.

Repro

await using var context = new BlogContext();
await context.Database.EnsureDeletedAsync();
await context.Database.EnsureCreatedAsync();

int v = 1;
Expression<Func<Blog, object>> f = x => v;
var q = await context.Set<Blog>().OrderBy(f).ToListAsync();

public class BlogContext : DbContext
{
    public DbSet<Blog> Blogs { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        => optionsBuilder
            .UseSqlServer("Server=localhost;Database=test;User=SA;Password=Abcd5678;Connect Timeout=60;ConnectRetryCount=0;Encrypt=false")
            .LogTo(Console.WriteLine, LogLevel.Information)
            .EnableSensitiveDataLogging();
}

public class Blog
{
    public int Id { get; set; }
    public string Name { get; set; }
}

[roji]

The issue isn't the Expression variable, it's the convert-to-Object node that's added inside; the same error happens with:

var v = 1;
var q = await context.Set<Blog>().OrderBy(x => (object)v).ToListAsync();

This means that it's possible to work around this simply by having the lambda return the exact type, so no up-cast is needed:

Expression<Func<Dummy, int>> f = x => v;

The problem is that when we process the body of a lambda in the funcletizer, we incorrectly leave the state as EvaluatableWithCapturedVariables, after we've already evaluated the body inside (EDIT: the actual bug was in VisitUnary, not VisitLambda - see below); this causes the whole lambda to be evaluated again (technically the wrapping Quote node), and the attempt to run the LINQ interpreter over the ParameterExpression inside causes the exception. The correct state after evaluating something is NotEvaluatable, so that we don't attempt to re-evaluate something that already has been evaluated.

Note that the issue no longer reproes after recent switch to QueryParameterExpression in 10.0 (#35101), but for the reasons aren't material to this issue: the evaluated node is now a QueryParameterExpression rather than a ParameterExpression, and the LINQ interpreter surprisingly doesn't error on that unknown node type. But the bug is still there and could probably manifest in other ways.

[BladeWise]

@roji I noticed that the issue was with conversion and applied the exact workaround you proposed.
Thanks for sharing the actual root cause!

[roji]

Upon a deeper look, VisitLambda turned out to be fine - the problem was in VisitUnary's EvaluateOperand, which left the unary's state as evaluatable instead of setting it to NoEvaluatable (after having evaluated the operand).