API: Pagination with offset instead of page

[ErisDS]

This is needed in order to fully support channel customisations, see #5091

---

As part of the move towards refactoring the Ghost frontend into a fully customisable and extensible set of channels, one of the features or customisations we want to make available is the ability to set a different number of posts-per-page for the first page vs subsequent pages in a channel.

E.g. make it so that the homepage shows 1 post, and every subsequent page in the index channel shows 10.

At the moment, pagination in the API is managed by passing in a limit (how many posts per page) and a page number. This enforces that every page in the set must be the same size. This limitation can be overcome by telling the API a limit (how many posts you want) and an offset (where in the list to start from). Calculations around the required offset for a given page can then be done based on the channel settings.

Therefore, we should replace the page concept in the API with an offset concept instead, for all of our findPage methods.

See also #2896

[RaoHai]

The question is, if we replace the page concept with offset, how can we calculate the next and prev value in pagination ?

[ErisDS]

@RaoHai it's relatively easy math to get from one to the other :)

We only need to replace the page concept in the Posts API, not on the frontend of the blog, currently the maths for getting from page to offset is hidden away in the depths of the model. Removing this abstraction out from the model layer makes sense to me. Further, when we add an API for Channels that will probably have a page option, rather than offset and will abstract the maths behind a reusable interface for figuring out all manner of details to do with a specific channel including next & previous pages.

[edsadr]

Question on this one, the findPage method right now is like:

/**
     * #### findPage
     * Find results by page - returns an object containing the
     * information about the request (page, limit), along with the
     * info needed for pagination (pages, total).
     *
     * **response:**
     *
     *     {
     *         posts: [
     *         {...}, {...}, {...}
     *     ],
     *     page: __,
     *     limit: __,
     *     pages: __,
     *     total: __
     *     }
     *
     * @param {Object} options
     */

After solving this should look like:

/**
     * #### findPage
     * Find results by page - returns an object containing the
     * information about the request (offset, limit), along with the
     * info needed for pagination (total).
     *
     * **response:**
     *
     *     {
     *         posts: [
     *         {...}, {...}, {...}
     *     ],
     *     offset: __,
     *     limit: __,
     *     total: __
     *     }
     *
     * @param {Object} options
     */

is that right?

[ErisDS]

Sorry I was convinced I already replied to this - the answer is yes, the input and output should be offset rather than page :)

[ErisDS]

I've cleaned up the API layer such that pagination only exists in one location now: /server/models/base/pagination.js - meaning this change should be a bit easier to do.

[ErisDS]

I've been mentally flip-flopping on this particular change for a while.

Without it, there are some limitations to how the API can be paginated. The case of wanting a different number of items in the first set to the second set, E.g. homepage shows 1 post, subsequent pages show 20 posts, doesn't work because if you change limit between page 1 and page 2 you'll either miss out or duplicate some items because the offset cannot be correctly calculated.

However, wholesale switching from using page to using offset is relatively tricky, not just in terms of making the switch and moving to using an offset calculation between the frontend controller and the API, but also in terms of knowing what to return from the API in terms of pagination meta data.

Additionally, we still have the idea that our API probably needs to move back towards conforming to JSON API (although this is still being considered). JSON API's position on pagination is very specific:

The page query parameter is reserved for pagination. Servers and clients SHOULD use this key for pagination operations.

It further specifies that any pagination strategy can be used, by using keyed parameter names like page[offset] or page[number].

Given that our API is going to move all filter keys down under a filter parameter - e.g. if you want to filter based on the page property of a post, you'd do ?filter=page:true rather than page:true, we don't need to free up the page property as we originally thought.

Therefore I propose that we keep page as is for now and add the possibility to use offset directly later as an additional way to specific pagination results in the API. The main difficulty I see is in resolving what the meta data returned for paginated requests should look like.

At the moment we return the following, all as integers:

page: - current page number E.g. 2
limit: - current no. items in a page E.g. 15
pages: - total number of pages in the result set E.g. 3
total: - total number of items in the result set E.g. 38
next: - the next page in the set E.g 3
prev: - the previous page in the set E.g 1

JSON API specifies that a response should returnfirst, last, prev & next and that they should be links - our API doesn't return links at present but this is something to keep in mind.

Long term, I think we could add a page[offset] style parameter, and then if we changed the meta data to return URIs rather than integers, the URI would change to match the type of pagination request made.

Main question is, anyone have any reason why we can't punt this to deal with more fully later?

[kevinansfield]

All seems like sound reasoning 👍 My main concern is about the public API - if we're definitely intending to add page[offset] and page[limit] at some point would it make sense to switch from page=1 to page[number]=1 now?

[ErisDS]

I had the same thought, and figured that making this change now 'just in case' we make another change later doesn't really make sense. If we were 100% that this is what we were going to do then I'd be in favour, but as it doesn't match what any other params look like it seems weird.

Also, in future if we do go down this road, we can output deprecation warnings for page whilst still supporting both it and page[number] for a short time before switching over - i.e. as they are not mutually exclusive the upgrade path is smoother than a wholesale switch.

[kevinansfield]

👍

[TrevorHinesley]

I posted about this on the json-api Github (json-api/json-api#1163) but since Ghost uses offset pagination, I'm curious:

1. How are first/last links generated? Let's say the offset is 2 and limit is 6. Is the first link going to have offset 0 and limit 2?

2. How would consecutive requests know the previous request's context? For instance, assume the example above. Once I go to the first page, how would I know to make the next link have offset 2 and limit 6 rather than offset 2 and limit 2?