Add explicit link to next page of results in search API

[robertknight]

The search API supports cursor-based pagination using the search_after parameter. The responses do not include an explicit link to the next page of results, so API clients have to figure out how to construct that themselves, and also how to tell when they have reached the end of the results. This has caused some problems:

• The lack of explicit "last page" indicator and an h issue with deleted annotations caused a client bug where in some rare cases it only fetches a subset of the annotations it should
• I have fielded several queries from users who had problems using search_after because they did not construct the next-page link correctly, or got confused by the fact that "after" refers to the order of entries in the results, and not necessarily "after" in a chronological sense. Providing an explicit link would avoid the need for manual construction for the most common use case, and also provide a live example of how the parameter is used.

What I would suggest we do is add an explicit "next page" link to the results, which is omitted for the last page. Some examples in other APIs:

• https://developer.atlassian.com/server/confluence/pagination-in-the-rest-api/ (uses _links object in JSON response)
• https://docs.github.com/en/rest/guides/using-pagination-in-the-rest-api?apiVersion=2022-11-28 (uses Link headers)

Of the approaches above, I think I would favor a field in the JSON response, as being easier to discover and to parse.

[acelaya]

+1 to the _links approach. I agree it's easier to discover. Anyone debugging the API will quickly find about it, as the response body is what one usually inspects first when messing around with an API.

I've also seen that more often.

[robertknight]

The Confluence API is similar to https://stateless.co/hal_specification.html (see also https://datatracker.ietf.org/doc/html/draft-kelly-json-hal-08), but slightly different, as link values are just strings, as opposed to objects in the HAL spec.

[robertknight]

Some more resources:

Blog post about different ways of embedding links in APIs: https://evertpot.com/json-links/
JSON API profile for cursor pagination: https://jsonapi.org/profiles/ethanresnick/cursor-pagination/#profile-intro.

[kael]

Some examples in other APIs:

• https://developer.atlassian.com/server/confluence/pagination-in-the-rest-api/ (uses _links object in JSON response)
• https://docs.github.com/en/rest/guides/using-pagination-in-the-rest-api?apiVersion=2022-11-28 (uses Link headers)

There's also a possible use of the HTTP Range header field - although not sure it completely fits the requirements.

From Range header, I choose you (for pagination)! :

• Querying the root of the collection :

GET /users

200 OK
Accept-Ranges: users
Content-Range: users 0-9/200

[ 0, …, 9 ]

• Requesting past the end of the collection :

GET /users
Range: users=1000-

416 Requested Range Not Satisfiable

• Last-n requests :

GET /users
Range: users=-5

206 Partial Content
Accept-Ranges: users
Content-Range: users 195-199/200

[ 195, …, 199 ]

Pity there's no standard JS API for parsing <Link/> headers.

[kael]

There's also a possible use of the HTTP Range header field - although not sure it completely fits the requirements.

It does not fit the requirements, indeed. I was a bit too happy to get all informations using a HEAD request, but it's not easily parsable out-of-the box

(I'm querying the API to get the total count of annotations per URI using limit=0, I was thinking that a HEAD could replace it).

[kael]

What I would suggest we do is add an explicit "next page" link to the results, which is omitted for the last page.

It'd be interesting to include the first and last annotations of the whole result.

With pagination, to be able to display next and previous buttons, for a query I'm also fetching the first (order=asc&limit=1) and last (order=desc&limit=1) annotations so that if the first annotation of the whole result is equal to the first annotation of the annotations slice the next <button/> is disabled.