Convert Kibana system indices to be hidden

[cjcenizal]

EDIT: The below issue was written under the mistaken belief that the system indices work had already been implemented on the ES side, and that any indices now showing up in Index Management needed to be updated to be hidden. It's starting to sound like this was incorrect, and that this work is still pending. Once we confirm this to be the case I'll close this issue.

---

In 8.0, Index Management is no longer treating dot-prefixed indices as hidden (#112990). Instead, these indices will need to be defined as index.hidden.

When Kibana boots up, I see the following indices using dot-prefixes. They need to be migrated to use the index.hidden setting too. See #81536 for some background on these indices.

• .security-7
• .kibana_8.0.0_001
• .apm-custom-link
• .apm-agent-configuration
• .kibana_task_manager_8.0.0_001

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[pgayvallet]

When Kibana boots up, I see the following indices still using dot-prefixes. They need to be migrated to use the index.hidden setting instead

.kibana_8.0.0_001
.kibana_task_manager_8.0.0_001

Shouldn't Index Management automatically consider system indices as hidden?

[cjcenizal]

@pgayvallet We could, which is the way we used to treat dot-prefixed indices. But the point of this issue is that ES now supports the hidden index setting, which is intended to replace dot-prefixes, and we should migrate to using it.

[cjcenizal]

My suggestion would be to continue using dot-prefixes and also to use the hidden index setting. This way we conform to the way ES expects this setting to be used and we can use the dot-prefix to identify those indices that manage Kibana internals.

[pgayvallet]

This way we conform to the way ES expects this setting to be used and we can use the dot-prefix to identify those indices that manage Kibana internals.

Just to be sure, are ES internal system indices using this index.hidden setting?

[sebelga]

As per my comment in #112990

Why we can't have a simple naming convention and stick to it everywhere in the stack. system indices are system, and hidden indices are simply hidden. Why the UI has to diverge from that and mention to the user that one of his hidden index is suddenly a system index?

[rudolf]

Performing the following API requests on the latest Elasticsearch snapshot confirms that .kibana* is currently treated as a system index.

1. Creating a system index is allowed

rudolf@rudolf-elastic-mbp kibana % curl -X PUT "elastic:changeme@localhost:9200/.kibana_8.0.0_001"
{"acknowledged":true,"shards_acknowledged":true,"index":".kibana_8.0.0_001"}

2. Deleting it issues a deprecation warning

rudolf@rudolf-elastic-mbp kibana % curl -I -X DELETE "elastic:changeme@localhost:9200/.kibana_8.0.0_001"
HTTP/1.1 200 OK
X-elastic-product: Elasticsearch
Warning: 299 Elasticsearch-8.0.0-SNAPSHOT-a8c25fd596fa8ff23c9bcb84e778f64e2b2f9db5 "this request accesses system indices: [.kibana_8.0.0_001], but in a future major version, direct access to system indices will be prevented by default"
content-type: application/json
content-length: 21

Given that system and hidden indices are distinct concepts it wouldn't make sense for us to mark the .kibana* indices as hidden too.

At some point before 8.0 the Elasticsearch system index behaviour will change and system indices shouldn't be returned from API requests such as _cat/indices. This will effectively hide the .kibana* and e.g. the .security* indices from the UI.

[sebelga]

Thanks for looking into this @rudolf. So you are saying that all the indices starting with a period below will not be returned by ES at some point before 8.0?

[rudolf]

Thanks for looking into this @rudolf. So you are saying that all the indices starting with a period below will not be returned by ES at some point before 8.0?

Yes, that's been my understanding. Unfortunately I don't have an issue to link to.

[cjcenizal]

@elastic/es-core-infra Could you please confirm Rudolf's understanding and link to an issue we can follow?

[rdnm]

That understanding is correct. We expect to flip the switch to block access to system indices under elastic/elasticsearch#67384.

[rdnm]

@rudolf sorry I misread this - for the GET API it will not be returned. For the CAT API I'm assuming it's also the same answer, but I want to double check with @gwbrown @williamrandolph first.

[williamrandolph]

We've had some late-breaking changes to expected behavior for system indices.

• In 8.0, system indices will all be restricted indices. An index permission will have to have allow_restricted_indices set to true in order to apply to system indices, so most users will not be able to view system indices.
• Currently, the superuser will be able to view system indices on the CAT API. We will look at this behavior and see if there is anything we can do to improve it before the 8.0 release. I'm looking at some options and trying to figure out what implementation will work best, but I think we have consensus that we want system indices to behave more or less like hidden indices.

[cjcenizal]

Thanks everyone for the discussion and updates. Closing this issue because it's obsolete.