Pagination and/or filtering for GET /_snapshot/<repo>/_all endpoint

[bgagnon]

Some repository plugins can have high latency (in my case, Wikimedia's Swift Repository Plugin), or simply very large listings of snapshots. Pagination is essential to prevent timeouts, reduce payload size and in general reduce the workload for the plugin and upstream services.

In most cases a date filter would suffice, but it's not clear the plugin would be able to apply such filtering efficiently.

[bgagnon]

See the other issue I opened at wikimedia/search-repository-swift#24

[s1monw]

I am not sure if pagination is the right term here, are you interested in parital results based on a timeout? ie. if a repo won't respond in time you get the listing that have already arrived? Something like this?

[bgagnon]

Hmm, no, in this case I would not be interested in partial results due to timing out. I am interested in limiting the amount of listing I am requesting from the repo to give it a chance to respond within reasonable time. A timeout would still be considered a failure in this case.

I'm not sure if all or any of the repo implementations use a fast-access index for listing purposes. It might just be that the Swift one is doing an exhaustive listing of objects in the container where it could have done something less expensive.

Maybe our usage of snapshots is a little bizarre, but we can have several hundreds snapshot points over a long period of time. A way to filter by date range would be quite handy to maintain responsiveness in a list/select/restore UI.

In any case, having a way to specify parameters on GET /_snapshot/<repo>/... seems like a useful addition to a repo plugin's API.

[tlrx]

Thanks for the feedback. I think that you can achieve what you want by creating multiple repositories like repository-2016, repository-2017, repository-2018 and list all snapshots created in 2017 using
GET /_snapshot/repository-2017/_all?pretty.

You can apply the same naming convention in snapshot names, like snapshot-2018-01-01, snapshot-2018-02-13 and list all snapshots created in January 2017 using something like
GET /_snapshot/repository-2017/snapshot-2017-01-*?pretty or GET /_snapshot/repository/snapshot-2017-01-*?pretty.