Skip to content

Commit 60045e6

Browse files
yetanothertwrhr323
authored andcommitted
Adding Streams information into the Manage data section (elastic#3399)
Refers to [elastic#274](elastic/docs-content-internal#274) * On the [Data streams](https://www.elastic.co/docs/manage-data/data-store/data-streams) page added an admonition to mention Streams and link to it. * [Tutorial: Update the lifecycle of a data stream](https://www.elastic.co/docs/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream) now has a new section Manage data retention in Streams. * [Tutorial: Configure data stream retention](https://www.elastic.co/docs/manage-data/lifecycle/data-stream/tutorial-data-stream-retention#retention-configuration) has been updated as well. This page may need to be reworked separately in the future as most of its content fits under conceptual type of information rather than tutorial. * [Failure store](https://www.elastic.co/docs/manage-data/data-store/data-streams/failure-store#set-up-failure-store-existing) -- updated the [Set up for existing data streams](https://www.elastic.co/docs/manage-data/data-store/data-streams/failure-store#set-up-failure-store-existing) section to add a tip about performing that in Kibana. * [Modify a data stream](https://www.elastic.co/docs/manage-data/data-store/data-streams/modify-data-stream) -- updated but need to update links to Streams content after Mike's PR is merged. * [View the lifecycle status of an index or data stream](https://www.elastic.co/docs/manage-data/lifecycle/index-lifecycle-management/policy-view-status) -- updated the Kibana tab to add steps for viewing the lifecycle of a data stream in Streams. * [Manage a data stream](https://www.elastic.co/docs/manage-data/data-store/data-streams/manage-data-stream) -- updated to add a new section about managing data streams in the Streams UI. * [Tutorial: Migrate ILM managed data stream to data stream lifecycle](https://www.elastic.co/docs/manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle) -- updated to add a note about the ability to switch between ILM and data stream lifecycle in the Streams area in Kibana. --------- Not sure about the impact on these topics: * [Manually apply a lifecycle policy to an index](https://www.elastic.co/docs/manage-data/lifecycle/index-lifecycle-management/policy-apply) -- I was considering updating this procedure, but I'm not sure anymore because it seems to mostly refer to applying an ILM policy to standalone indices, and not data streams (so the streams feature wouldn't really apply). * [Manage the lifecycle policy for integrations data](https://www.elastic.co/docs/manage-data/lifecycle/index-lifecycle-management/manage-lifecycle-integrations-data) -- not sure how this page would be affected. @kilfoyle, would you have any thoughts on this, please?
1 parent 5e94e2a commit 60045e6

File tree

10 files changed

+115
-8
lines changed

10 files changed

+115
-8
lines changed

manage-data/data-store/data-streams.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,16 @@ A data stream acts as a layer of abstraction over a set of indices that are opti
1717

1818
You can submit indexing and search requests directly to a data stream. The stream automatically routes the request to backing indices that store the stream’s data. You can use [{{ilm}} ({{ilm-init}})](../lifecycle/index-lifecycle-management.md) to automate the management of these backing indices. For example, you can use {{ilm-init}} to automatically move older backing indices to less expensive hardware and delete unneeded indices. {{ilm-init}} can help you reduce costs and overhead as your data grows.
1919

20+
You can also use a [data stream lifecycle](../lifecycle/data-stream.md) to automate lifecycle management according to your retention requirements.
21+
22+
:::{admonition} Managing data streams with Streams
23+
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}
24+
25+
Starting with {{stack}} version 9.2, the [**Streams**](/solutions/observability/streams/streams.md) page provides a centralized interface for managing your data in {{kib}}. It consolidates common data management tasks and eliminates the need for manual configuration of multiple applications and components. A stream maps directly to an {{es}} data stream, for example `logs-myapp-default`. Any changes that you make on the **Streams** page are automatically propagated to the associated data stream.
26+
27+
For more information, refer to [Manage data streams with Streams](/manage-data/data-store/data-streams/manage-data-stream.md#manage-data-streams-with-streams).
28+
29+
:::
2030

2131
## Should you use a data stream? [should-you-use-a-data-stream]
2232

manage-data/data-store/data-streams/failure-store.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,12 @@ PUT _data_stream/my-datastream-existing/_options
9090

9191
1. Redirecting failed documents into the failure store will now be disabled.
9292

93+
:::{tip}
94+
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}
95+
96+
You can also enable the data stream failure store in {{kib}}. Locate the data stream on the **Streams** page, where a stream maps directly to a data stream. Select a stream to view its details and go to the **Retention** tab where you can find the **Enable failure store** option.
97+
:::
98+
9399
### Enable failure store via cluster setting [set-up-failure-store-cluster-setting]
94100

95101
If you have a large number of existing data streams you may want to enable their failure stores in one place. Instead of updating each of their options individually, set `data_streams.failure_store.enabled` to a list of index patterns in the [cluster settings](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). Any data streams that match one of these patterns will operate with their failure store enabled.

manage-data/data-store/data-streams/manage-data-stream.md

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,4 +21,25 @@ In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB ind
2121

2222
* To view more information about a data stream, such as its generation or its current index lifecycle policy, click the stream’s name. From this view, you can navigate to **Discover** to further explore data within the data stream.
2323
* To view information about the stream’s backing indices, click the number in the **Indices** column.
24-
* To modify the data retention value, select a data stream, open the **Manage** menu, and click **Edit data retention**.
24+
* To modify the data retention value, select a data stream, open the **Manage** menu, and click **Edit data retention**.
25+
26+
## Manage data streams with Streams [manage-data-streams-with-streams]
27+
```{applies_to}
28+
serverless: ga
29+
stack: preview 9.1, ga 9.2
30+
```
31+
Starting with {{stack}} version 9.2, the [**Streams**](/solutions/observability/streams/streams.md) page provides a centralized interface for managing your data in {{kib}}. It consolidates common data management tasks and eliminates the need for manual configuration of multiple applications and components. A stream maps directly to an {{es}} data stream, for example `logs-myapp-default`. Any changes that you make on the **Streams** page are automatically propagated to the associated data stream.
32+
33+
:::{image} /manage-data/images/data-stream-management-streams.png
34+
:alt: The Streams page
35+
:screenshot:
36+
:::
37+
38+
39+
You can perform the following data management tasks on the **Streams** page:
40+
* [define parsing and field extraction logic](/solutions/observability/streams/management/extract.md)
41+
* [configure data retention policies](/solutions/observability/streams/management/retention.md)
42+
* [manually adjust index settings](/solutions/observability/streams/management/advanced.md)
43+
* [manage and update field mappings](/solutions/observability/streams/management/schema.md) {applies_to}`stack: unavailable 9.1`
44+
* [identify failed and degraded documents](/solutions/observability/streams/management/data-quality.md) {applies_to}`stack: unavailable 9.1`
45+
* [partition data into child streams](/solutions/observability/streams/management/partitioning.md) {applies_to}`stack: preview 9.2` {applies_to}`serverless: preview`

manage-data/data-store/data-streams/modify-data-stream.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,11 @@ To add a mapping for a new field to a data stream, following these steps:
9191
```
9292

9393

94+
:::{tip}
95+
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}
96+
97+
You can also add a new field mapping to a data stream in {{kib}}. Locate the data stream on the **Streams** page where a stream maps directly to a data stream. Select a stream to view its details and go to the **Schema** tab to add a new field.
98+
:::
9499

95100
### Change an existing field mapping in a data stream [change-existing-field-mapping-in-a-data-stream]
96101

@@ -168,6 +173,12 @@ The documentation for each [mapping parameter](elasticsearch://reference/elastic
168173
```
169174

170175

176+
:::{tip}
177+
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}
178+
179+
You can also update an existing field's mapping in {{kib}}. Locate the data stream on the **Streams** page where a stream maps directly to a data stream. Select the stream to view its details and go to the **Schema** tab to make your updates. For more information, refer to [](/solutions/observability/streams/management/schema.md).
180+
:::
181+
171182
Except for supported mapping parameters, we don’t recommend you change the mapping or field data type of existing fields, even in a data stream’s matching index template or its backing indices. Changing the mapping of an existing field could invalidate any data that’s already indexed.
172183

173184
If you need to change the mapping of an existing field, create a new data stream and reindex your data into it. See [Use reindex to change mappings or settings](../data-streams/modify-data-stream.md#data-streams-use-reindex-to-change-mappings-settings).
@@ -218,6 +229,14 @@ To change the `index.lifecycle.name` setting, first use the [remove policy API](
218229
::::
219230

220231

232+
:::{tip}
233+
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}
234+
235+
You can also update certain dynamic index settings for a data stream in {{kib}}, such as the number of shards, replicas, and the refresh interval used.
236+
237+
Locate the data stream on the **Streams** page where a stream maps directly to a data stream. Select the stream to view its details and go to the **Advanced** tab to make your adjustments. For more information, refer to [](/solutions/observability/streams/management/advanced.md#streams-advanced-index-config).
238+
:::
239+
221240
### Change a static index setting for a data stream [change-static-index-setting-for-a-data-stream]
222241

223242
[Static index settings](elasticsearch://reference/elasticsearch/index-settings/index.md) can only be set when a backing index is created. You cannot update static index settings using the [update index settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-settings).

manage-data/data-store/index-basics.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,12 @@ In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB ind
153153
* To modify the data retention value, select a data stream, open the **Manage** menu, and click **Edit data retention**. On {{stack}}, this action is only available if your data stream is not managed by an ILM policy.
154154
* To view more information about a data stream including it's lifecycle settings, click the stream's name.
155155

156+
:::{admonition} Streams
157+
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}
158+
159+
Starting with {{stack}} version 9.2, the [**Streams**](/solutions/observability/streams/streams.md) page provides a centralized interface for common data management tasks in {{kib}}, including tasks such as [modifying data retention](/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md#data-retention-streams) values.
160+
:::
161+
156162
### Manage index templates [index-management-manage-index-templates]
157163

158164
An index template is a type of [template](/manage-data/data-store/templates.md) that tells {{es}} how to configure an index when it is created.
169 KB
Loading

manage-data/lifecycle/data-stream/tutorial-data-stream-retention.md

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ products:
1010

1111
# Tutorial: Configure data stream retention [tutorial-manage-data-stream-retention]
1212

13-
In this tutorial, we are going to go over the data stream lifecycle retention; we will define it, go over how it can be configured and how it can gets applied. Keep in mind, the following options apply only to data streams that are managed by the data stream lifecycle.
13+
This tutorial demonstrates lifecycle retention, showing how to define, configure, and apply it. Keep in mind that the following options apply only to data streams that are managed by the data stream lifecycle.
1414

1515
1. [What is data stream retention?](#what-is-retention)
1616
2. [How to configure retention?](#retention-configuration)
@@ -42,6 +42,11 @@ The result should look like this:
4242
2. Ensure that the lifecycle is enabled, meaning this should be `true`.
4343

4444

45+
:::{tip}
46+
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}
47+
48+
You can also review how a data stream is managed by locating it on the **Streams** page in {{kib}}. A stream directly corresponds to a data stream. Select a stream to view its details and go to the **Retention** tab.
49+
:::
4550

4651
## What is data stream retention? [what-is-retention]
4752

@@ -67,9 +72,9 @@ Global default and max retention do not apply to data streams internal to elasti
6772

6873
## How to configure retention? [retention-configuration]
6974

70-
* By setting the `data_retention` on the data stream level. This retention can be configured in two ways:
75+
* Configure data retention at the data stream level using the `data_retention` setting. You can do this in two ways:
7176

72-
For new data streams, it can be defined in the index template that would be applied during the data stream’s creation. You can use the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template), for example:
77+
* For a new data stream, the `data_retention` setting can be included in the index template that is applied when the data stream is created. You can use the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template), for example:
7378

7479
```console
7580
PUT _index_template/template
@@ -88,7 +93,7 @@ Global default and max retention do not apply to data streams internal to elasti
8893
}
8994
```
9095

91-
For an existing data stream, it can be set via the [PUT lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle).
96+
* For an existing data stream, the `data_retention` setting can be configured using the [PUT lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle).
9297

9398
```console
9499
PUT _data_stream/my-data-stream/_lifecycle
@@ -99,7 +104,13 @@ Global default and max retention do not apply to data streams internal to elasti
99104

100105
1. The retention period of this data stream is set to 30 days.
101106

102-
* By setting the global retention via the `data_streams.lifecycle.retention.default` and/or `data_streams.lifecycle.retention.max` that are set on a cluster level. You can be set via the [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). For example:
107+
:::{tip}
108+
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}
109+
110+
To adjust the retention period of a data stream in {{kib}}, locate a data stream on the **Streams** page. A stream maps directly to a data stream. Next, select a stream to view its details and review the **Retention** tab to find out how it's managed before making your adjustments.
111+
:::
112+
113+
* By setting the global retention via the `data_streams.lifecycle.retention.default` and/or `data_streams.lifecycle.retention.max` that are applied on a cluster level. You can set these via the [update cluster settings API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-settings). For example:
103114

104115
```console
105116
PUT /_cluster/settings

manage-data/lifecycle/data-stream/tutorial-migrate-ilm-managed-data-stream-to-data-stream-lifecycle.md

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,14 @@ products:
1313
In this tutorial we’ll look at migrating an existing data stream from [Index Lifecycle Management ({{ilm-init}})](../index-lifecycle-management.md) to [data stream lifecycle](../data-stream.md). The existing {{ilm-init}} managed backing indices will continue to be managed by {{ilm-init}} until they age out and get deleted by {{ilm-init}}; however, the new backing indices will be managed by data stream lifecycle. This way, a data stream is gradually migrated away from being managed by {{ilm-init}} to being managed by data stream lifecycle. As we’ll see, {{ilm-init}} and data stream lifecycle can co-manage a data stream; however, an index can only be managed by one system at a time.
1414

1515

16-
To migrate a data stream from {{ilm-init}} to data stream lifecycle we’ll have to execute two steps:
16+
:::{admonition} Configure data retention policies for Streams
17+
:applies_to: {"stack": "ga 9.2, preview 9.1", "serverless": "ga"}
18+
19+
Starting with {{stack}} version 9.2, the [**Streams**](/solutions/observability/streams/streams.md) page provides a centralized interface for common data management tasks in {{kib}}, including tasks such as configuring data retention policies. You can choose to retain your data indefinitely, for a custom period, or by following an existing ILM policy. For more information, refer to [Manage data retention in Streams](/manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md#data-retention-streams).
20+
21+
:::
22+
23+
To migrate a data stream from {{ilm-init}} to data stream lifecycle using APIs you need to run two steps:
1724

1825
1. Update the index template that’s backing the data stream to set [prefer_ilm](elasticsearch://reference/elasticsearch/configuration-reference/data-stream-lifecycle-settings.md#index-lifecycle-prefer-ilm) to `false`, and to configure data stream lifecycle.
1926
2. Configure the data stream lifecycle for the *existing* data stream using the [lifecycle API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-data-lifecycle).

manage-data/lifecycle/data-stream/tutorial-update-existing-data-stream.md

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ Follow these steps to configure or remove data stream lifecycle settings for an
1414

1515
- [Set a data stream’s lifecycle](#set-lifecycle)
1616
- [Remove the lifecycle for a data stream](#delete-lifecycle)
17+
- [Manage data retention with Streams](#data-retention-streams)
1718

1819
Note that these steps are for data stream lifecycle only. For the steps to configure {{ilm}}, refer to the [{{ilm-init}} documentation](/manage-data/lifecycle/index-lifecycle-management.md). For a comparison between the two, refer to [](/manage-data/lifecycle.md).
1920

@@ -212,4 +213,20 @@ GET .ds-my-data-stream-*/_lifecycle/explain
212213
3. The name of another backing index.
213214
4. Indication that the index is not managed by the data stream lifecycle.
214215
:::
215-
:::::
216+
:::::
217+
218+
## Manage data retention with Streams [data-retention-streams]
219+
```{applies_to}
220+
serverless: ga
221+
stack: preview 9.1, ga 9.2
222+
```
223+
224+
Starting with {{stack}} version 9.2, the **Streams** page provides a centralized interface for common data management tasks in {{kib}}, including getting insight into how your data streams retain data.
225+
226+
1. Go to the **Streams** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
227+
1. Each stream maps directly to an {{es}} data stream. Select a stream to view its details.
228+
1. Go to the **Retention** tab to set how long your stream retains data and to get insight into your stream's data ingestion and storage size.
229+
1. Select **Edit data retention** and choose to retain your data indefinitely, for a custom period, or by following an existing ILM policy. You can also use the data retention configuration that's set in the index template by enabling the **Inherit from index template** option. If you use this option, you don't need to specify a custom retention period or policy.
230+
231+
For more information about the retention configuration options, refer to [](/solutions/observability/streams/management/retention.md).
232+

manage-data/lifecycle/index-lifecycle-management/policy-view-status.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,16 @@ To learn how to create a specialized ILM policy for any data stream, such as tho
4747
1. Select the data stream to view details. The flyout that opens includes direct links to the ILM policy and the index template.
4848

4949
![Data stream status page](/manage-data/images/elasticsearch-reference-datastream-status.png "")
50+
51+
52+
**To view the current lifecycle status for a datastream on the Streams page:** {applies_to}`"stack": "ga 9.2, preview 9.1"`
53+
54+
Starting with {{stack}} version 9.2, the [**Streams**](/solutions/observability/streams/streams.md) page provides a centralized interface for common data management tasks in {{kib}}.
55+
56+
1. Go to the **Streams** page using the navigation menu or the [global search field](/explore-analyze/find-and-organize/find-apps-and-objects.md).
57+
1. A stream maps directly to an {{es}} data stream. Select a stream to view its details.
58+
1. Go to the **Retention** tab to see how long your stream retains data and to get insight into your stream's data ingestion and storage size. A stream can retain the data indefinitely, for a custom period, or by following an existing ILM policy. For more information, refer to [](/solutions/observability/streams/management/retention.md).
59+
5060
::::
5161

5262
:::{tab-item} API

0 commit comments

Comments
 (0)