[Index Management] Support data retention on Data Streams tab #165263

alisonelizabeth · 2023-08-30T18:46:04Z

This PR adds support for viewing the configured data retention value per data stream on the Data Streams tab in Index Management.

On serverless, a data stream can be managed by data stream lifecycle.

On stateful, a data stream can be managed by data stream lifecycle or ILM, or both.

Fixes #165134
Fixes #152720
Fixes #154259
Fixes #165143

Note: Skipping release note for now, however, we will want to add one once the remaining work is complete (meta: #154256)

How to test

Create a data stream managed via data stream lifecycle (docs). The configured data retention should appear on the list view and details flyout.
[Stateful only] Create a data stream managed via ILM (docs). There should be no data retention value. On the details flyout, there should be a link to the ILM policy.
[Stateful only] Create a data stream managed by ILM and data stream lifecycle. The configured data retention should appear on the list view and details flyout. The ILM policy name should also appear on the details flyout.

Screenshots

…t/data_stream_lifecycle

…-fix'

jacksonbroshear · 2023-09-06T00:23:45Z

Unsubscribe

…

On Tue, Sep 5, 2023, 1:53 PM kibana-ci ***@***.***> wrote: ⏳ Build in-progress, with failures - Buildkite Build <https://buildkite.com/elastic/kibana-pull-request/builds/156615> - Commit: c41885c <c41885c> - Interpreting CI Failures <https://www.elastic.co/guide/en/kibana/current/interpreting-ci-failures.html> Failed CI Steps - Checks <https://buildkite.com/elastic/kibana-pull-request/builds/156615#018a6711-dab6-4b4a-8d77-09c612838fa1> History - 💔 Build #156603 <https://buildkite.com/elastic/kibana-pull-request/builds/156603> failed 79cb0ab <79cb0ab> - 💔 Build #154982 <https://buildkite.com/elastic/kibana-pull-request/builds/154982> failed 1cd71e3 <1cd71e3> To update your PR or re-run it, just comment with: @elasticmachine merge upstream — Reply to this email directly, view it on GitHub <#165263 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABICVKDVHHT4LXRARKUTX3XY6GONANCNFSM6AAAAAA4E6IGPI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

…t/data_stream_lifecycle

alisonelizabeth · 2023-09-06T18:01:04Z

...nt/public/application/sections/home/data_stream_list/data_stream_table/data_stream_table.tsx

@@ -71,7 +71,6 @@ export const DataStreamTable: React.FunctionComponent<Props> = ({
    render: (health: DataStream['health']) => {
      return <DataHealth health={health} />;
    },
-    width: '100px',


This was causing some weird spacing issues in the table.

alisonelizabeth · 2023-09-06T18:04:46Z

x-pack/plugins/index_management/common/types/data_streams.ts

 interface TimestampFieldFromEs {
  name: string;
 }

 type TimestampField = TimestampFieldFromEs;

-interface MetaFromEs {


I refactored some of the server-side code to leverage the types from @elastic/elasticsearch rather than our own custom interfaces

alisonelizabeth · 2023-09-07T13:21:09Z

@elasticmachine merge upstream

…t/data_stream_lifecycle

elasticmachine · 2023-09-07T20:06:33Z

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

…t/data_stream_lifecycle

yuliacech

Thanks a lot for working on data retention, @alisonelizabeth!
Tested locally and the information is displayed as expected, code changes LGTM too 👍

gchaps

Indices appears twice in the flyout.

...ication/sections/home/data_stream_list/data_stream_detail_panel/data_stream_detail_panel.tsx

gchaps · 2023-09-12T20:56:17Z

...nt/public/application/sections/home/data_stream_list/data_stream_table/data_stream_table.tsx

+      <EuiToolTip
+        content={i18n.translate('xpack.idxMgmt.dataStreamList.table.dataRetentionColumnTooltip', {
+          defaultMessage:
+            'The minimum amount of time the data stream will be stored for. Only applicable for data streams managed by data stream lifecycle.',


Some suggestions:

The amount of time that a data stream retains its data before it is automatically deleted. Only applies to data streams managed by a data stream lifecycle policy.

The amount of time to keep the data indexed in the data stream. Only applies for data streams managed by a data stream lifecycle.

The amount of time to keep the data stream. Only applies for data streams managed by a data stream lifecycle.

I've learned this value may also not apply to all the data depending on the configuration. From the ES team:

Meaning, they switched from one to the other at some point, so for example: backing indices 1, 2 and 3 are managed by ILM, and backing indices 4 and 5 and any future ones are managed by data stream lifecycle.

Do you think we could work this in as well?

…t/data_stream_lifecycle

alisonelizabeth · 2023-09-14T15:42:05Z

@gchaps I've addressed most of your feedback in f48949b. Let me know what you think about the final copy for the tooltip in the table. Here's the latest screenshot.

tylerperk · 2023-09-14T20:47:36Z

Let me know what you think about the final copy for the tooltip in the table. Here's the latest screenshot.

@alisonelizabeth Something that comes up is how well ILM and now DSL enable timely deletion for compliance reasons (GDPR etc). We don't delete the data exactly after 7 days (in the above example) so we should be careful to not give that impression. I wonder if we could improve the first sentence a little as the first time I read it I felt a bit like it implies a guarantee which we know won't be satisfied. It is more like "the minimum amount of time data will be available before it is automatically deleted" or "data will be be kept at least this long before it is automatically deleted" or ?

gchaps · 2023-09-14T22:33:27Z

Here's a suggestion based on @tylerperk's comment:

Data will be be kept at least this long before it is automatically deleted. Only applies to data streams managed by a data stream lifecycle. This value might not apply to all data if the data stream also has an index lifecycle policy.

…t/data_stream_lifecycle

alisonelizabeth · 2023-09-18T13:50:39Z

Thanks @gchaps! I updated the text in 264952a.

kibana-ci · 2023-09-18T15:01:01Z

💚 Build Succeeded

Buildkite Build
Commit: 264952a

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`indexManagement`	189	181	-8

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`indexManagement`	536.8KB	538.1KB	+1.4KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`indexManagement`	4	3	-1

Unknown metric groups

API count

id	before	after	diff
`indexManagement`	194	186	-8

ESLint disabled line counts

id	before	after	diff
`indexManagement`	14	13	-1

Total ESLint disabled count

id	before	after	diff
`indexManagement`	18	17	-1

History

💚 Build #159187 succeeded f48949b
💚 Build #157641 succeeded 26541e5
💔 Build #157415 failed a9cb16d
💔 Build #157401 failed bdd649f
💔 Build #157258 failed e9c01a9
💔 Build #157244 failed 7fb90b5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

alisonelizabeth and others added 4 commits August 30, 2023 14:39

add data retention to data streams table

1cd71e3

add data retention to details panel

d0226d6

Merge branch 'main' of github.com:elastic/kibana into index_managemen…

79cb0ab

…t/data_stream_lifecycle

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

c41885c

…-fix'

alisonelizabeth added 4 commits September 6, 2023 12:28

Merge branch 'main' of github.com:elastic/kibana into index_managemen…

5dc856a

…t/data_stream_lifecycle

fix details panel logic

71552af

fix jest tests

002ae7c

fix API integration tests

86a4468

alisonelizabeth commented Sep 6, 2023

View reviewed changes

add tooltip to table

de09ecc

kibanamachine and others added 7 commits September 7, 2023 09:21

Merge branch 'main' into index_management/data_stream_lifecycle

9ba7c0b

fix i18n

7fb90b5

remove unused translation

e9c01a9

Merge branch 'main' of github.com:elastic/kibana into index_managemen…

44e72f8

…t/data_stream_lifecycle

update readme

6a5c9e5

remove ternary

bdd649f

fix test

a9cb16d

alisonelizabeth marked this pull request as ready for review September 7, 2023 20:05

alisonelizabeth requested a review from a team as a code owner September 7, 2023 20:05

alisonelizabeth added release_note:skip Skip the PR/issue when compiling release notes Feature:Index Management Index and index templates UI Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more labels Sep 7, 2023

alisonelizabeth added the ui-copy Review of UI copy with docs team is recommended label Sep 7, 2023

alisonelizabeth requested a review from yuliacech September 8, 2023 12:53

alisonelizabeth added 2 commits September 8, 2023 09:46

fix test

d4c1f4d

Merge branch 'main' of github.com:elastic/kibana into index_managemen…

26541e5

…t/data_stream_lifecycle

alisonelizabeth requested a review from gchaps September 8, 2023 14:20

yuliacech approved these changes Sep 8, 2023

View reviewed changes

gchaps reviewed Sep 12, 2023

View reviewed changes

alisonelizabeth mentioned this pull request Sep 13, 2023

[Index Management] UX support when data stream lifecycle is disabled #166385

Closed

alisonelizabeth added 2 commits September 14, 2023 10:20

Merge branch 'main' of github.com:elastic/kibana into index_managemen…

cc150bc

…t/data_stream_lifecycle

address initial copy feedback

f48949b

alisonelizabeth added 2 commits September 18, 2023 09:44

Merge branch 'main' of github.com:elastic/kibana into index_managemen…

5a678dc

…t/data_stream_lifecycle

update tooltip text

264952a

alisonelizabeth merged commit be01217 into elastic:main Sep 18, 2023

alisonelizabeth deleted the index_management/data_stream_lifecycle branch September 18, 2023 16:37

kibanamachine added v8.11.0 backport:skip This commit does not require backporting labels Sep 18, 2023

alisonelizabeth mentioned this pull request Sep 27, 2023

[Index management] Data stream edit data retention #167006

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Index Management] Support data retention on Data Streams tab #165263

[Index Management] Support data retention on Data Streams tab #165263

alisonelizabeth commented Aug 30, 2023 •

edited

Loading

jacksonbroshear commented Sep 6, 2023 via email

alisonelizabeth Sep 6, 2023

alisonelizabeth Sep 6, 2023

alisonelizabeth commented Sep 7, 2023

elasticmachine commented Sep 7, 2023

yuliacech left a comment

gchaps left a comment •

edited

Loading

gchaps Sep 12, 2023

alisonelizabeth Sep 13, 2023

alisonelizabeth commented Sep 14, 2023

tylerperk commented Sep 14, 2023

gchaps commented Sep 14, 2023 •

edited

Loading

alisonelizabeth commented Sep 18, 2023

kibana-ci commented Sep 18, 2023

API count

ESLint disabled line counts

Total ESLint disabled count

[Index Management] Support data retention on Data Streams tab #165263

[Index Management] Support data retention on Data Streams tab #165263

Conversation

alisonelizabeth commented Aug 30, 2023 • edited Loading

How to test

Screenshots

jacksonbroshear commented Sep 6, 2023 via email

alisonelizabeth Sep 6, 2023

Choose a reason for hiding this comment

alisonelizabeth Sep 6, 2023

Choose a reason for hiding this comment

alisonelizabeth commented Sep 7, 2023

elasticmachine commented Sep 7, 2023

yuliacech left a comment

Choose a reason for hiding this comment

gchaps left a comment • edited Loading

Choose a reason for hiding this comment

gchaps Sep 12, 2023

Choose a reason for hiding this comment

alisonelizabeth Sep 13, 2023

Choose a reason for hiding this comment

alisonelizabeth commented Sep 14, 2023

tylerperk commented Sep 14, 2023

gchaps commented Sep 14, 2023 • edited Loading

alisonelizabeth commented Sep 18, 2023

kibana-ci commented Sep 18, 2023

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

Public APIs missing exports

API count

ESLint disabled line counts

Total ESLint disabled count

History

alisonelizabeth commented Aug 30, 2023 •

edited

Loading

gchaps left a comment •

edited

Loading

gchaps commented Sep 14, 2023 •

edited

Loading