Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate Index Management away from using the cat APIs #57286

Closed
cjcenizal opened this issue Feb 11, 2020 · 5 comments · Fixed by #122867
Closed

Migrate Index Management away from using the cat APIs #57286

cjcenizal opened this issue Feb 11, 2020 · 5 comments · Fixed by #122867
Labels
enhancement New value added to drive a business result Feature:Index Management Index and index templates UI Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more

Comments

@cjcenizal
Copy link
Contributor

cjcenizal commented Feb 11, 2020

Per elastic/elasticsearch#51772, cat APIs are not intended for consumption by applications. Index Management currently uses _cat/aliases and _cat/indices to list index aliases and indices.

As a side effect of using the cat APIs, Elasticsearch skips shard refreshes by default as of 7.0. This means that if you index a document and go to Index Management, the doc count for that index will be unchanged until that index is searched.

We can use the index stats API in lieu of cat indices:

GET _all/_stats/docs

However, these results are still affected by the skipped shard refreshes. Per discussion with Jason Tedor, we'd have to make an explicit refresh call to keep indices up-to-date, but refreshing every index can be very expensive and often unnecesssary:

for lots of use cases today (time series), users will have a mix of indices that are actively being indexed into, and those that are no longer being indexed into
in that case, of active indexing, a refresh doesn’t help because while the page is loading, more docs are being indexed so the count isn’t going to be the latest anyway, because it will have already been rendered wrong
in the case of indices that are no longer being indexed into, the counts will be accurate after a little bit of time has elapsed
for these use cases, a refresh/toggle to refresh doesn’t do anything except an unnecessary refresh call on the actively indexing indices (hurting performance)

Based on the above input, we should not bake a refresh into the request for fetching indices. If users need to see the most up-to-date doc count of an index, they should refresh it manually:

image

@cjcenizal cjcenizal added enhancement New value added to drive a business result Feature:Index Management Index and index templates UI Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more labels Feb 11, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/es-ui (Team:Elasticsearch UI)

@cjcenizal
Copy link
Contributor Author

Related to #46126

@cjcenizal
Copy link
Contributor Author

Tangentially related to #106041

@cjcenizal
Copy link
Contributor Author

Related to #107621

@cjcenizal
Copy link
Contributor Author

Also tangentially related to elastic/elasticsearch#64099

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New value added to drive a business result Feature:Index Management Index and index templates UI Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants