Efficient Pagination of experiments list straight from mongodb

[J-avery32]

The current method of pagination using cursor.skip() is not scalable however it is the only way I see currently to allow the client to choose what page they would like to start from. You may find this method here https://www.codementor.io/arpitbhayani/fast-and-efficient-pagination-in-mongodb-9095flbqr This page also lays out another more efficient method using the ObjectId however this is not desirable as we can't allow the client to specify what page they would like to start from. They must start at the beginning of the results and continue paging through until finding the correct page.

One method to consider Is giving each document an id that increases by 1.
For example,

{page_id: 1, data: "foo"},

{page_id: 2, data: "bar"},

{page_id: 3, data: "baz"},

...

Then we combine this with the second method described in that webpage.
This will allow the client to specify a page to start at because we can easily utilize the db.find() method to start at the correct document.

However, arbitrarily numbering the documents like this may not be necessary if we would like to page according to date and time of the inserted document. This is probably much easier to implement through the $currentDate operator. It will also be more useful as the user will probably wish to see experiments by dates anyway. Again the client can easily specify from which date they would like to start at as flask can use the db.find(desiredDate) on the date field.

[dylanmcreynolds]

Thanks for doing all of this research.

Before talking about the solution, it's useful to talk about what we want to achieve. We want to build a framework for paginating a mongo query which can include filter and ordering. The ordering part makes the proposed solutions untenable...we won't always be using order of creation or time of creation.

It is possible that supporting these requirements with a single framework is untenable. However, at this stage in the project, let us just get something going and use skip and limit.