[5.8][WIP] Support eager loading with limit

[staudenmeir]

There is currently no way to limit eager loading results per parent:

class User extends Model {
    public function posts() {
        return $this->hasMany(Post::class)->latest()->limit(5);
    }
}

User::with('posts')->get(); 

# Loads 5 posts for all users combined.
select * from "posts" where "posts"."user_id" in (1, 2, 3) order by "created_at" desc limit 5

There are multiple issues about this (#16217, #4835, #18014) and it also comes up regularly on StackOverflow etc.

We can support this feature with a window function (we are already using this technique to implement OFFSET queries on SQL Server):

select * from (
  select *, row_number() over (partition by "user_id" order by "created_at" desc) as row_num
  from "posts"
  where "posts"."user_id" in (1, 2, 3)
) as temp_table
where row_num <= 5
order by row_num

The query is supported by all four databases, but requires very recent versions of MySQL and SQLite:

• MySQL 8.0 (2018-07-27): We could support older versions with this workaround.
• PostgreSQL 9.3 (2013)
• SQLite 3.25 (2018-09-15)
• SQL Server 2008

I wrote a sample implementation for PostgreSQL because it's the only supported database in Homestead at the moment.

The relationships override the query builder's limit() and call the new partitionLimit() method. By checking the parent model's exists property, we can detect eager loading. This is necessary to support lazy loading ($user->posts). The window function would also work for lazy loading but the database might not support it.

Before I implement the other databases: Would you consider this for the core?

[deleugpn]

Promising. Great job.

[X-Coder264]

This looks very nice. It'd be great if this would be in the core.

[Lelectrolux]

Currently using my own version of the mysql workaround, happy to see that mysql 8.0 helps to clean the madness even if I'm on my own, but I would love that in laravel's core.

For an example of use in real life, it's a needs that comes often when doing travelling best offers (like find the 3 best price for this plane/train in the next week). My current work is full of that.

[staudenmeir]

@Lelectrolux Are you using the MySQL workaround with BelongsToMany or HasManyThrough relationships?

[Lelectrolux]

@staudenmeir Currently, I'm using it on a hasMany relationship, but I'm hotswapping the table with a SQL view to have some computed prices on a join.

[staudenmeir]

I added the remaining implementations. We can use the same SQL in all four databases. The only exception is that SQL Server always requires an ORDER BY clause (we use a dummy clause if necessary).

• MySQL:
  I implemented the mentioned workaround for MySQL < 8.0. It works for HasMany relationships but not for BelongsToMany or HasManyThrough. For some reason, MySQL ignores the ORDER BY clause and always uses the row insertion order (fiddle). Does anyone have an idea on how to fix this? Unfortunately, this is quite hacky for BelongsToMany relationships.

• SQLite:
  SQLite 3.25.0 has only been released a month ago, so this feature doesn't work with the current PHP distributions. You have to compile the latest pdo_sqlite extension yourself (I tested it on Windows). So, no integration tests.
  Since SQLite is mainly used for testing, I suggest that we completely ignore the partition limit on unsupported versions. This PR is "only" a performance improvement and I don't think it's worth breaking the users' tests.

• SQL Server:
  I moved compileOver() and compileTableExpression() up to Grammar and refactored them to work with the new features.

[miki131]

@staudenmeir what about some LimitTrait and move there take and limit functions?

[staudenmeir]

@miki131 That's not possible for limit(): All three methods are different from each other.

[miki131]

@staudenmeir take a look on getExistenceCompareKey() method. It can be useful for BelongsToMany.php and HasOneOrMany.php. Later i'll try find out about third relation.

[staudenmeir]

Yes, but we still need a separate implementation for HasManyThrough. I don't think it makes sense to create a trait for two relationships and override it in the third.

[miki131]

@staudenmeir You can create getExistenceCompareKey method for HasManyThrough. Also override getParent method. I think it can be useful in future and obviously help you separate code into trait.

[staudenmeir]

@miki131 The HasManyThrough relationship does have a $parent property, so getParent() can't just return the $farParent.

[ludo237]

What about MariaDB?

Promising feature tho

[staudenmeir]

@ludo237 MariaDB has been supporting window functions since 10.2, but they don't work in strict mode (bug report).

[taylorotwell]

Do any other frameworks support this?

[staudenmeir]

Rails doesn't support it:

If you eager load an association with a specified :limit option, it will be ignored, returning all the associated objects.

I don't know about Symfony, I couldn't find anything. So probably not.

[taylorotwell]

Honestly I don't see a big incentive to take on the maintenance burden of maintaining this functionality. If it's not even supported by Rails that indicates to me it's not that widely useful.

[staudenmeir]

I released the code as a package: https://github.com/staudenmeir/eloquent-eager-limit

[yaquawa]

Rejected as usual.

Tips: To get this into the core, the best way is send a PR to Rails.

[atrauzzi]

Just hit this today, a bit confused as to why it was rejected. This would be a great - if not essential - addition to Eloquent. If people have the syntax available, it stands to reason they'll assume this is possible.

[driesvints]

Hey @atrauzzi, there's a package for this which you may use: https://github.com/staudenmeir/eloquent-eager-limit

[tillkruss]

When using PostgreSQL use DISTINCT ON: #4835 (comment)

[staudenmeir]

Eager loading with limit will be supported natively in Laravel 11 (#49695).