Implementing pagination for _cat/indices API

[gargharsh3134]

Description

Changes include:

1. Introducing new _list/indices API which will display paginated results.
2. Since, for paginated responses, next_token and what element is getting paginated will also have to be displayed, RestTable and Table classes have been extended to accommodate pagination related metadata
3. New RestListAction (similar to RestCatAction), to output all _list APIs.
4. Support of paginationStrategies and PageTokens. Different Pagination Strategies can implement PaginationStragety interface. The interface has been kept very open, considering the strategies can greatly vary in terms of what all parameters they might require to parse a token and generate a list of page from a data store (for e.g. ClusterState is a data store for IndexBasedPaginationStrategy, which has been added as part of this PR).

Functional Testing on a local cluster:

_list API:

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list"       
:‑|
/_list/indices
/_list/indices/{index}

All the indices in the cluster

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_cat/indices"                                                                                               
green open test-index  FclPNwqmQie3RnpoOYpLTg 2 3 0 0 1.6kb 416b
green open test-index2 k5FNOQcvSV679Hcs1g-OwQ 2 3 0 0 1.6kb 416b
green open index4-test s2AJ5CoYRru-8lbB871pWA 2 3 0 0 1.6kb 416b

Retrieving indices in pages of size=1, using _list API, in plain text format:

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/indices?size=1"         
green open index4-test s2AJ5CoYRru-8lbB871pWA 2 3 0 0 1.6kb 416b
next_token MSQxNzE5NDI3NDk3NDQ0JDE3MTk0Mjc1NTYzODEkaW5kZXg0LXRlc3Q=

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/indices?size=1&next_token=MSQxNzE5NDI3NDk3NDQ0JDE3MTk0Mjc1NTYzODEkaW5kZXg0LXRlc3Q="
green open test-index2 k5FNOQcvSV679Hcs1g-OwQ 2 3 0 0 1.6kb 416b
next_token MiQxNzE5NDI3NTM3MzcwJDE3MTk0Mjc1NTYzODEkdGVzdC1pbmRleDI=

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/indices?size=1&next_token=MiQxNzE5NDI3NTM3MzcwJDE3MTk0Mjc1NTYzODEkdGVzdC1pbmRleDI="
green open test-index FclPNwqmQie3RnpoOYpLTg 2 3 0 0 1.6kb 416b
next_token null

Retrieving indices in pages of size=1, using _list API, in JSON format:

gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/indices?size=1&format=json&pretty"                                                 
{
  "next_token" : "MSQxNzE5NDI3NDk3NDQ0JDE3MTk0Mjc1NTYzODEkaW5kZXg0LXRlc3Q=",
  "indices" : [
    {
      "health" : "green",
      "status" : "open",
      "index" : "index4-test",
      "uuid" : "s2AJ5CoYRru-8lbB871pWA",
      "pri" : "2",
      "rep" : "3",
      "docs.count" : "0",
      "docs.deleted" : "0",
      "store.size" : "1.6kb",
      "pri.store.size" : "416b"
    }
  ]
}
gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/indices?size=1&format=json&pretty&next_token=MSQxNzE5NDI3NDk3NDQ0JDE3MTk0Mjc1NTYzODEkaW5kZXg0LXRlc3Q="
{
  "next_token" : "MiQxNzE5NDI3NTM3MzcwJDE3MTk0Mjc1NTYzODEkdGVzdC1pbmRleDI=",
  "indices" : [
    {
      "health" : "green",
      "status" : "open",
      "index" : "test-index2",
      "uuid" : "k5FNOQcvSV679Hcs1g-OwQ",
      "pri" : "2",
      "rep" : "3",
      "docs.count" : "0",
      "docs.deleted" : "0",
      "store.size" : "1.6kb",
      "pri.store.size" : "416b"
    }
  ]
}
gkharsh@bcd07443f159 OpenSearch % curl "localhost:9200/_list/indices?size=1&format=json&pretty&next_token=MiQxNzE5NDI3NTM3MzcwJDE3MTk0Mjc1NTYzODEkdGVzdC1pbmRleDI="
{
  "next_token" : null,
  "indices" : [
    {
      "health" : "green",
      "status" : "open",
      "index" : "test-index",
      "uuid" : "FclPNwqmQie3RnpoOYpLTg",
      "pri" : "2",
      "rep" : "3",
      "docs.count" : "0",
      "docs.deleted" : "0",
      "store.size" : "1.6kb",
      "pri.store.size" : "416b"
    }
  ]
}

Related Issues

Resolves #14258

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 4924a71: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 0cfb0e9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for d665722: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[dblock]

This looks like a great start.

I would consider (don't feel strongly about it though) wrapping additional metadata into metadata to support other fields like total count.

{
  "metadata": {
      "next_token" : "MSQxNzE5NDI3NDk3NDQ0JDE3MTk0Mjc1NTYzODEkaW5kZXg0LXRlc3Q=",
      "total_count": 200
   },
  "indices" : [
    {
      "health" : "green",
      "status" : "open",
      "index" : "index4-test",
      "uuid" : "s2AJ5CoYRru-8lbB871pWA",
      "pri" : "2",
      "rep" : "3",
      "docs.count" : "0",
      "docs.deleted" : "0",
      "store.size" : "1.6kb",
      "pri.store.size" : "416b"
    }
  ]
}

From the implementation POV, can we build RestIndicesListAction to be a lot thinner where all pagination parts are generic except data fetching? I think a goal should be that to opt into _list/* one adds a ListAction interface to some existing classes and fills out the parts that actually fetch a page of data given a token and that's it. Otherwise it's too easy to accidentally start returning a different object from a non-paginated API vs. a paginated API, which I think is a red flag.

[github-actions]

❌ Gradle check result for fc4b6d9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for c8261d4: SUCCESS

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-14718-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 0dba562f896a7294fc16eddf64081fe45b3aa3bb
# Push it to GitHub
git push --set-upstream origin backport/backport-14718-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-14718-to-2.x.

[dblock]

@gargharsh3134 @shwetathareja please don't forget https://github.com/opensearch-project/opensearch-api-specification!

[gargharsh3134]

@gargharsh3134 @shwetathareja please don't forget https://github.com/opensearch-project/opensearch-api-specification!

Sure @dblock. I'm already tracking this. Once _list/shards changes are merged, will immediately follow those up with a combined PR for both _list/indices and _list/shards specs

[ohltyler]

Is there any tracking of documentation updates? I don't see anything on https://opensearch.org/docs/latest/api-reference, and don't see any open PRs on the website repo.

[dbwiddis]

Is there any tracking of documentation updates? I don't see anything on https://opensearch.org/docs/latest/api-reference, and don't see any open PRs on the website repo.

Looks like the checklist on the PR template wasn't completed.

[gargharsh3134]

@ohltyler @dbwiddis Sorry for the inconvenience, I'm working on adding API specs for now. Let me open the respective Github issues for public documentation in parallel.

Is there any tracking of documentation updates? I don't see anything on https://opensearch.org/docs/latest/api-reference, and don't see any open PRs on the website repo.

As for this, it would be made available from OS version 2.18 in opensource/public documentation.