Add support for dynamically extended queries as per LinqKit

[fkorak]

The package LINQKit, a solution for making queries dynamically extendable at runtime, currently sits at half a million downloads for EF Core alone, with much more for traditional EF.

The old uservoice forums had an issue with 114 upvotes, which puts it on the frontpage of that now shuttered forum.

Dynamic queries like PredicateBuilder currently only exist as an external solution (as part of LINQKit). It would be preferable to have this functionality be part of EF Core.
There are a dozen advantages for including this upstream:

• No breakage of LINQKit with new EF Core Versions
• Functionality should be part of the Core anyway, since what it offers fits the use cases attempted by EF Core pretty well
• The maintainers themselves link to the uservoice forum, asking people to upvote inclusion in upstream
• some parts of LINQKit could be optimized since then not everything would need to be extension methods, there are a lot of possibilities with API and Code Generation

[ajcvickers]

Note from triage: we should investigate what makes sense in this space. For example, that might be making sure EF Core works well with LINQKit and making it easier for the two to keep working across releases.

[KLuuKer]

We mostly use LINQKit for the reuse of .Where(....) for special business logic that is reused in allot of places that might need to be changed in the future (or just have a simple\easier name for it),
and .Select(.....) for selecting some common models that are reused in a bunch of places

if only that feature would make it into EF Core itself i would be very happy

also i have noticed that EF Core is not able to optimize the sql generated when a select\where (expression) has been invoked using LINQKit (we have a bunch of heavy queries so i open sql profiler allot and look at sql being generated)

[KLuuKer]

And now i'm blocked from upgrading to EF Core 3.0 #scottksmith95/LINQKit#95

Official support for the feature i meantioned above would remove my dependency on LINQKit (I really like the library but built in support would be better of course 😄 )

[schmitch]

well I think PredicateBuilder is something that people want. But most functionality of that is already in EFCore. you can AND where parts together when writing code like that:

var query = ...
foreach (...)
{
query = query.Where(...)
}

However OR'ing is completly missing. Probably because it would need a good api design to not confuse users.

WhereOr Expression can be simulated like that:

    internal class RangeMemberExpressionModifier : ExpressionVisitor
    {
        private readonly ParameterExpression _parameterExpression;

        public RangeMemberExpressionModifier(ParameterExpression parameterExpression)
        {
            _parameterExpression = parameterExpression;
        }

        public Expression Modify(Expression expression)
        {
            return Visit(expression);
        }

        protected override Expression VisitParameter(ParameterExpression node)
        {
            return _parameterExpression;
        }
    }
    public class QueryPartOrBuilder<TSource>
    {
        internal readonly List<Expression<Func<TSource, bool>>> PredicateLists = new List<Expression<Func<TSource, bool>>>();

        public void Add(Expression<Func<TSource, bool>> predicate)
        {
            PredicateLists.Add(predicate);
        }
    }

        public static IQueryable<TSource> WhereOr<TSource>(this IQueryable<TSource> query, QueryPartOrBuilder<TSource> orBuilder)
        {
            var predicates = orBuilder.PredicateLists;

            if (predicates.Count == 0)
            {
                return query;
            }

            if (predicates.Count == 1)
            {
                return query.Where(predicates[0]);
            }

            var parameter = predicates.FirstOrDefault()?.Parameters.FirstOrDefault();
            if (parameter == null)
            {
                throw new ArgumentException("invalid parameter name");
            }

            var modifier = new RangeMemberExpressionModifier(parameter);
            var aggregatedExpressions = predicates.Aggregate<Expression>((first, second) =>
            {
                Expression me1;
                switch (first)
                {
                    case LambdaExpression le:
                        me1 = modifier.Modify(le.Body as BinaryExpression);
                        break;
                    case BinaryExpression be:
                        me1 = be;
                        break;
                    default:
                        throw new ArgumentException("arguments should not be null");
                }

                var body2 = (second as LambdaExpression)?.Body;
                if (body2 == null)
                {
                    throw new ArgumentException("arguments should not be null");
                }

                var me2 = modifier.Modify(body2 as BinaryExpression);
                return Expression.OrElse(me1, me2);
            });

            return query.Where(Expression.Lambda<Func<TSource, bool>>(aggregatedExpressions, parameter));
        }

[ghost]

Schmitch's code works perfectly! It would it be nice to have a similar WhereOr function in EF Core.

[KifoPL]

Any updates on this idea? I'd love to see LinqKit as part of ef core library. Any chance for this to be added in .NET 7?

[taylorchasewhite]

Any updates on this idea? I'd love to see LinqKit as part of ef core library. Any chance for this to be added in .NET 7?

Seconded

[roji]

It would be good to make clearer exactly what people are looking for here, i.e. is it making it easier/more reliable to use LinqKit with EF, or to bring certain LinqKit functionality into EF (so that LinqKit is no longer required at all).

For the 2nd part, it seems like some people are looking for an easier way to construct dynamic queries, e.g. dynamically adding multiple ORs with PredicateBuilder. It's important to note that such queries produce different SQLs based on which predicates are added (they're dynamic); this means that such queries generally aren't cached well by the database (query plan cache) or by EF. As a general rule, we try to avoid generating dynamic SQLs based on the same "simple" LINQ query tree, to make it very clear where dynamic querying is being used, and prevent users from unintentionally falling into this perf trap.

But in any case, if LinqKit provides a mechanism that helps with dynamic query construction (e.g. PredicateBuilder), then I'm not sure there's a good reason to do something with EF - any reason to not just using LinqKit for these queries?

[fkorak]

It would be good to make clearer exactly what people are looking for here, i.e. is it making it easier/more reliable to use LinqKit with EF, or to bring certain LinqKit functionality into EF (so that LinqKit is no longer required at all).

I'm not really sure how you would do the former without doing the latter. LinqKit is downstream from you, meaning every alpha, beta, release etc. they play catch-up. They currently have a separate package for every release of EF Core. Making their implementation more stable would either mean upstreaming LinqKit as whole (or in parts, i.e. PredicateBuilder) or creating some form of very stable API/Access, which given how development of EF Core generally has been going could severely restrict upstream, and sounds difficult.

For the 2nd part, it seems like some people are looking for an easier way to construct dynamic queries, e.g. dynamically adding multiple ORs with PredicateBuilder. It's important to note that such queries produce different SQLs based on which predicates are added (they're dynamic); this means that such queries generally aren't cached well by the database (query plan cache) or by EF. As a general rule, we try to avoid generating dynamic SQLs based on the same "simple" LINQ query tree, to make it very clear where dynamic querying is being used, and prevent users from unintentionally falling into this perf trap.

Absolutely agree, but some usecases are dynamic purely by the requirements given to me by the product owner. A dynamic search page or a report generator with freely choose-able parameters for example. (and yes, there are naturally more performant solutions, they just tend to also be more cumbersome. I do not want to run a report server or Lucene every time I let users access a database)

But in any case, if LinqKit provides a mechanism that helps with dynamic query construction (e.g. PredicateBuilder), then I'm not sure there's a good reason to do something with EF - any reason to not just using LinqKit for these queries?

I think I listed quite a few reasons for why just using LinqKit can be inconvenient in my opening post.

[roji]

I'm not really sure how you would do the former without doing the latter. LinqKit is downstream from you, meaning every alpha, beta, release etc. they play catch-up.

I don't quite understand this argument... As long as LinqKit generates valid expression trees which conform to what the compiler would generate, these should work just fine in EF Core itself. The impression given above is that EF somehow breaks many queries with every release - specifically those dynamically generated with LinqKit - and that doesn't correspond to how we evolve EF. I'm not saying there haven't been breaks ever - there have been - but these should have been quite rare, if LinqKit generates standard query forms.

So I'm looking for some very concrete examples of where LinqKit was broken when a new EF version came out, and to understand why - since at this point this issue is somewhat general and vague.

The other argument here - regardless of any breaking - is that this functionality should be in in EF itself, rather than in a 3rd-party extension. We very frequently get this argument on almost all kinds of EF extensions, and this usually simply amounts to "I don't want to use a 3rd-party tool, Microsoft should build everything for me". We very explicitly believe in an open .NET ecosystem, where EF provides good extensibility hooks and unlocks useful extensions rather than building everything into the box - that last approach simply isn't scalable.

So here, once again, I'm looking to understand exactly why something like PredicateBuilder belongs in EF; if it can exist outside as a clean layer on top of EF, that sounds like a great overall story to me and a good sample of community augmentation of an extensible product.

I hope the above is clear - I'm not necessarily pushing back, but I do believe this issue in its current form raises some very general arguments and lacks very concrete examples/problems to back the request.

[circleupx]

I think it might be best for those that work in Linqkit to chime in on this feature, perhaps they can provide better insights into the issue they are facing in terms of compatibility/breaking changes.

My experience, I used LinqKit with EF and I have been using LinqKit with EF Core since 2.1, I don't see why this needs to live within EF Core, I mean the main use case appears to be a predicate builder, you can easily handle that yourself, see the above example or continue to use LinqKit.

Additionally, please remember that it is very easy to say "Put in EF Core" and move on, but remember adding this feature to EF would mean that the EF team would need to maintain it, they would now need to put time and resources into a feature that I would argue provides very little benefits to EF itself.

Thanks,