[Index Management] Large number of indices take a long time to load

[yuliacech]

Problem

Index Management gets noticeably slow when loading a large number of indices. Currently, the app is loading all existing indices in the browser and the pagination for the indices table is calculated on the client side. This approach works but the page gets slower the more indices there are in the deployment. This issue is to investigate the risk and limitations of the current implementation and work on performance improvements.

• Test when loading time in Index Management gets noticeably slow
• Test how many indices the current implementation can handle until the loading time becomes cumbersome for users
• Investigate where the loading time is spent when indices are being retrieved

Loading time when fetching indices

Cloud deployment with many indices

In #126169, I added a logger to "fetch indices" route to track where the time is spent when indices are loaded. Following "checkpoints" were monitored:

• Before and after Get all Indices request to ES is completed
• Before and after Get Indices Stats request to ES is completed
• Before and after each index data enricher:
• ILM Explain lifecycle request
• Rollup job capabilities request
• CCR follower indices request

I added sequentially 1000, 2000, 3000, 4000, 5000 and 8000 indices to the deployment via this script. I stopped the testing at 8000 indices because the deployment kept failing. That will be investigated more in other tasks of this issue.

The logs were shipped to the deployment and I could filter them in discover and export to a csv file. The raw data is collected in this doc.

Average durations in ms

Label	1000 indices	2000 indices	3000 indices	4000 indices	5000 indices	8000 indices
Get indices request	42.14	69.41	496.85	684.18	1,438.09	1,151.35
Get indices stats request	59.86	105.95	1117.7	1,257.64	3,290.73	2,944.82
ILM index enricher	32.27	57.45	249.6	378.50	806.09	712.18
Rollup index enricher	135.86	319.50	1256.2	1,625.68	3,792.09	5,013.53
CCR index enricher	153.73	326.18	731.35	1,014.09	761.55	1,855.88

Performance improvements based on loading times findings

• We could add an option to the indices table that turns off index enrichers, either one by one or all at once
• We could decrease the number of columns displayed in the indices table to only request either Get Indices and Get Indices Stats endpoints. The more detailed information would be available in the flyout and would be loaded only for 1 index at a time.
• We could split Get Indices and Get Indices Stats requests and index enrichers and call them directly from the UI. That would allow the UI to display partial information while the remaining requests are loading. The merging of data would then happen in the browser.
• We could add a pagination to ES endpoints to load only limited number of indices Support pagination in REST APIs elasticsearch#64099

[elasticmachine]

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

[alisonelizabeth]

This is still an open issue and we are still investigating the best solution.

[elasticmachine]

Pinging @elastic/kibana-management (Team:Kibana Management)