[FEATURE] Better configuration endpoint to control index lifecycle

[ansjcy]

Is your feature request related to a problem?

Query Insights supports rollover historical indices but not deletion. We should enhance Query Insights plugin to better manage query insights index lifecycle, including deletion (and further, adaptive deletion).

What solution would you like?

• Configuration Endpoint for QI Index Management: In the first phase, we can add a configuration endpoint (and a job) within Query Insights that allows users to define retention policies for their query insights data. By default, the system can be configured to delete query insights data older than 7 days.
• Support for Adaptive Rollover: In the next phase, the configuration endpoint can also enable adaptive rollover based on data volume or configuration, giving us more control over retention without relying on external plugins like ISM.

What alternatives have you considered?

Relying on ISM for index management, but that means if the ISM plugin faces performance issues or changes, it could impact Query Insights functionality. Customers who disable the ISM plugin might not be able to use this feature neither.

Do you have any additional context?

Add any other context or screenshots about the feature request here.

[ansjcy]

We will need this feature in 2.19