Task manager to become a system index in Elasticsearch?

[mikecote]

No description provided.

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[pmuellr]

So many opportunities here!

The biggest is that since ES has a notion of a cluster of nodes (unlike Kibana which doesn't know if there are other Kibana's running as peers), we can likely control the writes and perhaps reads to route through a single node, and avoid some of the concurrency issues we've dealt with, in the existing TM implementation.

I was also wondering if we could arrange some alternate coupling on the transport, between ES and Kibana, so we don't have to actually "poll" in Kibana. Either use websockets, or some long-poll story, or make-it-slow async returns, or even have ES send requests to Kibana that a task needs to be run (this last one seems fragile, but kinda fun).

[pmuellr]

@mikecote Should we re-prioritize this? Do we even have a choice given saved objects are going to eventually become system indices?

[mikecote]

@pmuellr good question! @elastic/kibana-core regarding saved object indices, is the core team planning to handle the conversion for .kibana_task_manager, or is there some work necessary from the @elastic/kibana-alerting-services team to complete?

[mikecote]

I added this issue to the alerting project board.

[pgayvallet]

regarding saved object indices, is the core team planning to handle the conversion for .kibana_task_manager

Sorry, but conversion of what exactly? Isn't .kibana_task_manager de facto already a system index by following the .kibana* pattern?

[mikecote]

regarding saved object indices, is the core team planning to handle the conversion for .kibana_task_manager

Sorry, but conversion of what exactly? Isn't .kibana_task_manager de facto already a system index by following the .kibana* pattern?

With the system indices project, from what I understand, we need to convert Kibana owned indices to either system or hidden by 8.0? I'm wondering if it's the core team or alerting team who will convert the task manager index. It's a saved object index. See #81536 for more details.

[pgayvallet]

we need to convert Kibana owned indices to either system or hidden by 8.0?

Nothing to be done on our side, it was done in ES in https://github.com/elastic/elasticsearch/pull/68919/files#diff-ae6fc8fbb1c04f7efb1c5672ec0bb072f796b2d5968a2a34783d471a466aa099L57

.kibana_task_manager matches the .kibana_* pattern, so it's already an external system index. The associated restrictions will only apply on 8.0+ though

[mikecote]

@pgayvallet awesome, thank you! I will close this issue 🙏 8.0+ aligns with my thinking as well