Infinite Scroll query update

[sadakchap]

issue - #1551

• updated initial query to sort records according to 'objectId'
• updated fetch next page query to get new records according to 'objectId'

[davimacedo]

Thanks for the PR. LGTM!

[Klaitos]

Hello guys, @sadakchap since this commit we have big performance issues because of the initial sort done on the objectId + createdAt.
We have collection with millions of document (GB of data) where sorting by objectId+createdAt is not possible even with database indexes. Do you have any idea of a workaround ?

[davimacedo]

Would you be able to identify the query that is being sent to MongoDB? Could you also share the indexes that you have in place for this collection?

[Klaitos]

By workaround I meant why do we have to order by createdAt AND objectId at first load ?
Moreover objectId is alphanumeric characters string. Before this PR, the first load was only done by createdAt.

[mtrezza]

It's unclear to me how this PR intends to solve #1551. @sadakchap can you give more details?

We are lacking the high-level discussion here in the issue that should describe the approach of this PR and how it intends to solve the issue.

[sadakchap]

Earlier we were using createdAt as default sort on first load & infinite scroll was working fine with this, as subsequent request will be fetching next records based on last record.
But, if we were to sort records other than createdAt, the infinite scroll was getting stuck.. not at 30 records but maybe somewhere around 100 or 200.

Reason -

For infinite scroll we are not using skip, we are using sorting attribute ( can be createdAt, name or other fields in class ) & the last record. If we give wrong last record, we will get wrong next set of records.
E.g. let's say we are sorting with name, the first request will get records sorted according to name, and next subsequent request will get records sorted according to name, objectId & last record from the first request. And these two queries are different & gives wrong last record for next query, resulting in infinite scroll getting stuck.

Approach -

Making initial query also sorted according to objectId (why only objectId? as this will work as a tiebreaker, when some records have same value in sorting attribute since its always unique & its natural feature which makes objectId incremental). Please refer this.

[davimacedo]

By workaround I meant why do we have to order by createdAt AND objectId at first load ?
Moreover objectId is alphanumeric characters string. Before this PR, the first load was only done by createdAt.

There are some cases in which there are multiple objects with exact same createdAt, making the infinite scroll to stop working for not having a tiebraker. objectId was introduced to be this tiebraker. An index with both createdAt and objectId should fix any performance issue.

[mtrezza]

@Klaitos Did the suggested composed index of createdAt and objectId solve the perf issues you experienced? I would like to verify this, because both fields on their own would create highly selective indices, so I wonder whether that would create a performant index at all.

[Klaitos]

@Klaitos Did the suggested composed index of createdAt and objectId solve the perf issues you experienced? I would like to verify this, because both fields on their would create highly selective indices, so I wonder whether that would create a performant index at all.

@mtrezza Unfortunately it depends on the collection data volume, we have a very very huge collection where we can't explore anymore via the parse-dashboard because of that new default sort. Even with the compound index mongo can't make it. On other collections it works, we had to implement a solution to create the new index on every collection (100+), not a big deal but i think this should be noticed somewhere for future users.

However we have a custom build of the parse dashboard in which we override some default behavior (default count on every collection is also an expensive operation for us), removing the default compound sort is on the list.

[mtrezza]

Even with the compound index mongo can't make it.

It would be interesting to get more details about your case if you'd like to share. Specifically the query planner explain log, to understand what is going on. Whether there is a general issue here or there is something specific to your use case. In any case, ideally we can make a change to the dashboard so that you don't have to maintain your custom workaround. If you'd like to explore that, you could open a new issue to deep-dive into that.

[Klaitos]

I sincerely think the issue is on our side because of the data architecture on one of our collection. We do not use the parse dashboard as data explorer but mostly for useful quick crud operation with precise find criteria. So reverting this commit was effective for us. If we are the only one affected by this commit perf issue let it go as it is no worries.

[mtrezza]

Sounds good; if you change your mind at any point in the future, please feel free to open a new issue.

[mtrezza]

@Klaitos I came across the same issue you described. This is actually a breaking change, that can be solved with compound indices. I think the challenge with this approach is that it is a breaking change that requires special indices just for dashboard browsing.

For example, if an app is using a complex or resource intensive index, dashboard requires the same index plus the additional objectId in the index compound. I don't think this is a feasible solution, given the high cost of indices on a DB. Maybe we need to look for an alternative solution on the dashboard side, instead of on the server side.

[mtrezza]

I have opened #1799 for this. This PR should probably be reverted or modified to not force-add the objectId to the data query.