Return existing system indices from GET features API

[williamrandolph]

The GET /_features API currently returns minimal information about which features are present in the cluster: just a name and a text description. For example:

{
    "features": [
        {
            "name": "tasks",
            "description": "Manages task results"
        },
        {
            "name": "kibana",
            "description": "Manages Kibana configuration and reports"
        }
    ]
}

A user may want to know which existing indices in particular will be included in a snapshot for a feature. Additionally, having an endpoint that associates system indices, associated indices, and system data streams with their respective features may be diagnostically useful.

It would be simple to return a structure like this:

{
    "features": [
        {
            "name": "tasks",
            "description": "Manages task results",
            "system_indices": [
                {
                    "pattern": ".tasks*"
                    "indices": [
                        ".tasks"
                    ]
                }
            ]
        },
        {
            "name": "kibana",
            "description": "Manages Kibana configuration and reports"
            "system_indices": [
                {
                    "pattern": ".kibana-event-log-*"
                    "indices": [
                        ".kibana-event-log-8",
                        ".kibana-event-log-7-archive"
                    ]
                },
                {
                    "pattern": ".kibana_task_manager*",
                    "indices": [
                        ".kibana_task_manager_8"
                    ]
                }
            ]
        }
    ]
}

Since this could potentially be a lot of data for a human to read, we could put it behind a verbose request flag, or we could add a {feature} element to the path to return results for a particular feature.

We would also make sure that this is not a problem from a security perspective.

Relates #50251 (comment)

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[williamrandolph]

I raised this issue in a team meeting. One point from the discussion is that a potential downside of exposing the index names through an API like this is that the information becomes part of our backwards-compatible API contract, and users might reasonably assume that they can use this information in applications, even though we intend for it to be an "implementation detail."

We had a suggestion that a _cat API might be a more appropriate place to expose information that we intend to be purely diagnostic. The output would probably be an "identifier" per line, with "index/alias/datastream" and associated feature as fields in the row, sorted by feature.

[SSSSam141]

Hello! We are currently using elasticsearch as our school projects. But we are kind of confused about where we should modify the codes. Could you help us with the file path of where we could write this feature?

[rdnm]

Based on William's comment closing this issue.

[renshuki]

@williamrandolph @rdnm is there any plan to do something about this with the _cat API? It's still a pain to identify which index is part of a feature state during restore. These system indices / data streams need to be restored via the feature_states instead of the indices param in the Restore API, however, there is currently no other way than diving in ES source code to determine which index is part of a feature.