Add data streams to Index Management app

[matt-davis-elastic]

Why should we add this?

Elasticsearch is adding two new primitives. Index template v2 and data streams for time series data to ease the requirements to bootstrap an index and set up an alias before setting up ILM.

Everywhere in the UI where a user would need to interact with an Index, they will need to be able to interact with a data stream.

What are data streams?

elastic/elasticsearch#53100

What needs to be supported

Data streams have a dependency on version 2 of index templates. Creating a data streams can be done through the following API PUT /_data_stream/[name] or by adding the field to your index template v2 data_stream which is a JSON object that has a required timestamp field.

The index manager page should be able to list data streams. Data streams can be listed by calling the GET /_data_stream/ API. Data streams create hidden indices. To list which indices are part of the data stream use the GET /_data_stream/[name] API call.

When a data stream is deleted by issuing a DELETE /_data_stream/[name] API call the hidden indices are also removed.

[elasticmachine]

Pinging @elastic/es-ui (Team:Elasticsearch UI)

[matt-davis-elastic]

@jethr0null @cjcenizal Let me know if you need any more details. Thanks.

[cjcenizal]

Proposal

Major changes

• We'll add a new "Data Streams" tab to Index Management for managing data streams.
• We'll support cross-linking between the "Data Streams" tab and the "Indices" tab.
• We also propose renaming the "Index Management" app to "Data Management" to reinforce the notion that indices are no longer the sole primary unit of data in Elasticsearch, and that there are other first-class citizens like data streams.

Data stream consumption

CRUD basics. Users will be able to search and filter a paginated table of their data streams. Clicking the name of a data stream will open a detail panel that surfaces the data stream's underlying details.

Viewing backing indices. Clicking the "Indices" count in a data stream row will open up the "Indices" tab, with a filter pre-applied to show you the data stream's backing indices. Note that this requires three major technical changes:

• We need to implement support for hidden indices per Update Index Management to support hidden indices #56532.
• We need to add support to the "Indices" tab for a query parameter that turns on the "Show hidden indices" toggle, since backing indices are hidden.
• We need to add support for filtering by data stream name in the indices table.

Data stream actions

Delete action. Users will be able to select and delete data streams. Should we direct users to Ingest Manager for deleting managed data streams?

Cross-linking to Ingest Manager. Users will be able turn on a "View managed data streams" toggle to view data streams that were installed as part of a package by Ingest Manager. Will this be useful? Do we want to discourage users from deleting these data streams? Is "managed" the right term to use?

When the managed data streams are visible, they'll be marked with a "Managed" badge and the table will gain a "Package" column. This column names the package which installed this data stream, with a link back to the package in Ingest Manager.

Update: Based on conversation with @mostlyjason, we'll leave out the "Package" column because a single data stream could be associated with multiple packages. Instead of linking back to a package, we'll link back to the Data Sources page and filter the table down using the name of the data stream.

✂ This feature is at risk due to time constraints and will be the last thing we cut.

Cross-linking from Ingest Manager. Ingest Manager will be able to allow users to click a link to view a package's data stream(s) within the "Data Streams" tab. This tab will allow a query parameter to be set in the URL which will translate to a table filter so only the specified data streams are visible.

ILM integration. We'll build in the same ILM integration that currently exists in the "Indices" tab:

• "Add policy", "Remove policy", and "Retry lifecycle step" actions.
• A "phase" table column and the ability to filter the table by phase.
• Surfaced lifecycle errors and lifecycle information in detail panel.

Note that much of this functionality can be implemented by reusing the existing index management extension in ILM.

Should we direct users to Ingest Manager for performing ILM actions with managed data streams?

✂ This feature is at risk due to time constraints and will be the first thing we cut.

Integration with Indices

Viewing backed data streams. There will be a "Data stream" column in the table that shows you the data stream that an index backs. Clicking this data stream will open the "Data streams" tab, apply a filter so that only that data stream is visible, and open the data stream's detail panel.

[cjcenizal]

@ruflin @mostlyjason Could you please review the above proposal? I'm particularly interested to know if you think the section "Cross-linking from Ingest Manager" sounds like it will help users dive deeper into the assets installed by a package. If so, would you mind sharing a screenshot of the UI that gives the user this link, so we can get an idea of the cross-app UX?

[ruflin]

Here is a screenshot of our data streams page. As you see, we have different columns and different details as this is built for our use case. Your page is the generic view. I think we should allow to link out to your page but not sure if the other way around make sense. Also I don't think there is a difference between managed and unmanaged data streams. Ours could come from agents in standalone or managed by fleet.

[jethr0null]

Met with @ruflin @mostlyjason and @matt-davis-elastic and we discussed making the following changes to the current Ingest Manager data streams UI to help clarify the separation of concerns and improve the flow between Ingest Manager and the data streams management UI:

• Change the name of the Ingest Manager data streams UI (possibly something like Managed Data Streams to make it clear that these are a subset of the possible data streams in the system and reduce confusion/ambiguity)

• Create a top-level data stream view in the table view which encompasses the packages associated with that data stream as nested items (e.g. expand the data stream to see all of the data sources which direct data there as nested items)

• We will enable linking back to Ingest Manager from the ES UIs. This will be done by clicking on the data stream name (need to figure out exactly where in Ingest Manager we want to link back to (I'm thinking linking generically to the "Managed data streams" table view in Ingest Manager would be fine.

• The ES UI team will apply a badge to all items which are "managed" in the relevant UIs (e.g. created by the Ingest Manager) by a data stream (similarly to the "Managed" label shown above (@ruflin @mostlyjason @mukeshelastic we could use some input on what label to use here). This includes pipelines, index templates, and indices (@cjcenizal please chime in if I'm forgetting anything here). We will also create warning dialogs in the event that a user attempts to manually delete or edit one of these items which encourage users to manage at the data streams level via Ingest Manager. Blocker: We need consistent metadata to do this (@ruflin is there an issue to track this currently or would you like me to open one?)

[mostlyjason]

Thanks @jethr0null I tagged these in our issue list. I like that "managed" as a label could apply to Agent and Beats, if we choose to send metadata from both.

[cjcenizal]

In terms of labeling, WDYT of "Packaged" as a label, instead of "Managed"? I feel like this draws a clear connection back to the packages installed by Ingest Manager.

[ruflin]

@jethr0null No issue on our end yet. You mentioned it as a blocker: I think it is a blocker to link back but should not be a blocker on for the Data Streams page itself. If and how we can group data streams, I would like to get @hbharding involved. Important to not here: Not all data streams are related to a package.

[jethr0null]

@mostlyjason would you mind linking the issues you mentioned so we can track progress? I assume that you mentioned tagging in your issues list in the context of providing the "managed" metadata when Ingest Manager creates ingest node pipelines, index templates, indices, etc.

[jethr0null]

Important to not here: Not all data streams are related to a package.

@ruflin can you help me to understand this? Are all data streams which are created by Ingest Manager related to a package?

@jethr0null No issue on our end yet. You mentioned it as a blocker: I think it is a blocker to link back but should not be a blocker on for the Data Streams page itself. If and how we can group data streams, I would like to get @hbharding involved.

Also, I'd like to make sure we're on the same page here. The comment about this metadata being a blocker relates to our ability to identify and warn when a user tries to delete something that was created by Ingest Manager (ingest node pipelines, index templates, indices, etc.).

[mostlyjason]

@jethr0null This link is to a private repo. GH automatically adds a link in the activity thread, but it doesn't deliver a notification.

Beats used have a concept called modules instead. The "package" is a new concept introduced in Ingest Manager. "Managed" could potentially apply to assets from both modules and packages.

+1 on warning users when they try to change/delete managed assets. A link would be a nice to have but not a requirement IMHO.

[ruflin]

• All data streams are related to packages: This is not the case. All of them are tied to a package if Fleet is used to configure the Agent. But if the Elastic Agent is running in standalone, the user can configure whatever data stream he wants but it will follow the indexing strategy. For example the user can send logs to logs-foo-bar and will discover this data stream. Also there can be a hybrid where logs-nginx.access-default data stream receives data from an agent in standlone and one managed by Fleet.
• Meta Data: Yes, it is a blocker to make it possible to show it managed by it should not be a blocker to ship the data streams UI in general.

[mostlyjason]

There is already metadata that we already ship with Beats. This can be used to identify the "managed" index templates. I'm not sure if ingest pipelines have similar metadata or not. Interestingly, this index metadata may also be used for telemetry. Making sure you're aware @matt-davis-elastic

[cjcenizal]

Addressed by #67806