Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Efficient Pagination of experiments list straight from mongodb #10

Open
J-avery32 opened this issue Jul 23, 2019 · 1 comment
Open

Efficient Pagination of experiments list straight from mongodb #10

J-avery32 opened this issue Jul 23, 2019 · 1 comment
Assignees
Labels
enhancement New feature or request

Comments

@J-avery32
Copy link
Collaborator

J-avery32 commented Jul 23, 2019

The current method of pagination using cursor.skip() is not scalable however it is the only way I see currently to allow the client to choose what page they would like to start from. You may find this method here https://www.codementor.io/arpitbhayani/fast-and-efficient-pagination-in-mongodb-9095flbqr This page also lays out another more efficient method using the ObjectId however this is not desirable as we can't allow the client to specify what page they would like to start from. They must start at the beginning of the results and continue paging through until finding the correct page.

One method to consider Is giving each document an id that increases by 1.
For example,

{page_id: 1, data: "foo"},

{page_id: 2, data: "bar"},

{page_id: 3, data: "baz"},

...

Then we combine this with the second method described in that webpage.
This will allow the client to specify a page to start at because we can easily utilize the db.find() method to start at the correct document.

However, arbitrarily numbering the documents like this may not be necessary if we would like to page according to date and time of the inserted document. This is probably much easier to implement through the $currentDate operator. It will also be more useful as the user will probably wish to see experiments by dates anyway. Again the client can easily specify from which date they would like to start at as flask can use the db.find(desiredDate) on the date field.

@J-avery32 J-avery32 added the enhancement New feature or request label Jul 23, 2019
@dylanmcreynolds
Copy link
Collaborator

Thanks for doing all of this research.

Before talking about the solution, it's useful to talk about what we want to achieve. We want to build a framework for paginating a mongo query which can include filter and ordering. The ordering part makes the proposed solutions untenable...we won't always be using order of creation or time of creation.

It is possible that supporting these requirements with a single framework is untenable. However, at this stage in the project, let us just get something going and use skip and limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants