[S&R] Support data streams #68078

jloleysens · 2020-06-03T13:14:01Z

Summary

Update SLM management to support data streams. Rest of PR description WIP.

How to test

Start Kibana & ES (start ES with a repo path: yarn es snapshot -E path.repo=/tmp/es-snaps)
Create a data stream (see section below for Console commands)
Go to snapshot and restore UI in management section
Create a repository (using the path provided when starting ES)
Create a new SLM policy (in the Policies tab) and select a subset of indices and the data stream we created (these should be in the dropdown with badges "Data stream" if you unselect "All indices and data streams" <- new copy)
Run the SLM policy, creating a new snapshot
Go to the Snapshots tab and select the snapshot that was just created
The details should indicate all indices + the data stream that was snapshotted
Restore just the data stream by selecting the Restore action on the snapshot
On the step logistics view of the restore process select just the data stream and provide a capture pattern "test-data-stream" and rename to "restored-test-data-stream"
Restore the snapshot
Go to Index Management and view the data streams tab (there should be 2 data streams).
(Optional) Go to the indices tab in Index Management and select "Show Hidden Indices" and the backing indices should show along other hidden indices.

Create a data stream

Create an index template

PUT /_index_template/test-data-stream
{
	"index_patterns": [
		"test-data-stream*"
	],
	"template": {
		"mappings": {
			"properties": {
				"@timestamp": {
					"type": "date"
				}
			}
		}
	},
	"data_stream": {
		"timestamp_field": "@timestamp"
	}
}

Create a data stream

PUT /_data_stream/test-data-stream

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

Screenshots

Data streams and indices

Snapshot details view

Restoring a snapshot

The server endpoint for policies now returns data streams and filters out backing indices from the indices array it returned previously

jloleysens · 2020-06-22T08:14:35Z

@elasticmachine merge upstream

jloleysens · 2020-06-23T07:48:20Z

@elasticmachine merge upstream

jloleysens · 2020-06-24T09:42:28Z

@elasticmachine merge upstream

…icely separated

Given the contraints on working with data streams and indices in policies at the moment the simplest implementation is to just include data streams with indices and have the user select them there for now. The way snapshotting behaviour is currently implemented relies entirely on what is specified inside of "indices", this is also where data streams must be placed. This unfortunately means that capture patterns defined in indices will capture entire data streams too.

jloleysens · 2020-06-26T12:52:40Z

@elasticmachine merge upstream

- added spacer between title and data streams callout - added copy to the restore settings tab to indicate that settings also apply to backing indices

jloleysens · 2020-07-01T11:58:57Z

@elasticmachine merge upstream

jloleysens · 2020-07-01T12:12:59Z

Thanks for the review @jrodewig , @gchaps and @cjcenizal .

I think I have addressed all of your points, it would be great to get another pass for the copy that I added for renaming in the restore flow: Ensure that a matching index template exists for renamed data streams. and in case I missed anything 😅

@cjcenizal I think I have addressed most of your points except for your last one:

Can we change the "Index" column to "Index or data stream"?

I think in this case only indices are available which makes this view correct. It is inconsistent with the other views, but it does not look like the data streams are returned for us to display here unfortunately - there may be some way to get this that I am not aware of. I will as ES side about this.

Otherwise, would you mind taking another look?

jrodewig

Left a couple of non-blocking nits. LGTM.

...steps/step_settings/fields/indices_and_data_streams_field/indices_and_data_streams_field.tsx

jrodewig · 2020-07-01T13:36:51Z

...components/restore_snapshot_form/steps/step_logistics/data_streams_global_state_call_out.tsx

+    body: () => (
+      <FormattedMessage
+        id="xpack.snapshotRestore.restoreForm.dataStreamsWarningCallOut.body"
+        defaultMessage="Each data stream requires a matching index template. Please ensure any restored data streams have a matching index template. You can restore index templates by restoring the global cluster state. However, this may overwrite existing templates, cluster settings, ingest pipelines, and lifecycle policies. {learnMoreLink} about restoring snapshots that contain data streams."


...snapshot_restore/public/application/components/restore_snapshot_form/steps/step_settings.tsx

jloleysens · 2020-07-01T13:47:16Z

@elasticmachine merge upstream

cjcenizal

Code looks great! Thanks for making those copy changes. I had a number of small suggestions but nothing blocking.

x-pack/plugins/snapshot_restore/__jest__/client_integration/helpers/mocks.tsx

x-pack/plugins/snapshot_restore/__jest__/client_integration/helpers/restore_snapshot.helpers.ts

x-pack/plugins/snapshot_restore/__jest__/client_integration/helpers/setup_environment.tsx

x-pack/plugins/snapshot_restore/common/lib/is_data_stream_backing_index.ts

...ot_restore/public/application/components/collapsible_lists/collapsible_data_streams_list.tsx

cjcenizal · 2020-07-01T23:29:26Z

...s/snapshot_restore/public/application/components/restore_snapshot_form/steps/step_review.tsx

@@ -73,8 +73,8 @@ export const RestoreSnapshotStepReview: React.FunctionComponent<StepProps> = ({
          <EuiDescriptionList textStyle="reverse">
            <EuiDescriptionListTitle>
              <FormattedMessage
-                id="xpack.snapshotRestore.restoreForm.stepReview.summaryTab.indicesLabel"
-                defaultMessage="Indices"
+                id="xpack.snapshotRestore.restoreForm.stepReview.summaryTab.dataStreamsAndIndicesLabel"


I think we should keep this ID the same, to make it easier on the translators. We also want to uncouple the ID from the content so we don't need to change the id every time the label content changes, e.g. if we decide to change this to "Index abstractions" or even back to "Indices" in the future. This suggestion applies to other i18n IDs in this PR if there are similar changes.

Hm, I was actually thinking this would make the new translation work more discoverable for the translators. They will need to revisit these translations at any rate and I think the JSON doc is pretty much auto-generated.

I'd expect at some point for a release after 7.9 we will revisit these, hopefully with data streams and indices separated. For now I think these are also nice markers in the code for where we will need to change future copy again.

Makes sense! Would you mind pinging the i18n team about this? You raise good points, I'm just curious what the domain owners advise.

...snapshot_restore/public/application/components/restore_snapshot_form/steps/step_settings.tsx

x-pack/plugins/snapshot_restore/public/application/services/http/policy_requests.ts

x-pack/plugins/snapshot_restore/server/types.ts

jloleysens · 2020-07-02T09:04:27Z

@elasticmachine merge upstream

…very in snaphots

kibanamachine · 2020-07-02T13:01:14Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 2e46e93

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
snapshotRestore	27	+2	25

History

💚 Build #58180 succeeded c2fd2da
💚 Build #57963 succeeded 04f60e1
💔 Build #57862 failed 4a93135
💔 Build #57831 failed edadf62
💔 Build #57791 failed c0a1a12

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Sort endpoint responses into indices and datastreams The server endpoint for policies now returns data streams and filters out backing indices from the indices array it returned previously * Refactor indices switch and field out of the step settings file * Fix indices field form behaviour * WiP on UI. Added the second table per mockup for add and edit. * add support for creating a policy that backs up data streams end to end * wip on restore flow - added data streams to server response * add logic for detecting whether an index is part of a data stream * fix public jest tests * fix server side jest tests * pivot to different solution in UI while we do not have data streams nicely separated * added data stream to snapshot summary details * move the data streams badge file closer to where it used * add data stream badge when restoring snapshots too * update restore copy * fix pattern specification in indices and data streams field * first iteration of complete policy UX * First iteration that is ready for review Given the contraints on working with data streams and indices in policies at the moment the simplest implementation is to just include data streams with indices and have the user select them there for now. The way snapshotting behaviour is currently implemented relies entirely on what is specified inside of "indices", this is also where data streams must be placed. This unfortunately means that capture patterns defined in indices will capture entire data streams too. * delete unused import * fix type issue in tests * added logic for rendering out previous selection as custom pattern * refactor indices fields to make component smaller * added CIT for data streams badge * Data streams > indices * updates to relevant pieces of copy * more copy updates * fix types and remove unused import * removed backing indices from restore view * Added data stream restore warning message * restore CITs * first round of copy feedback * refactor help text to provide clearer feedback, for both restore and policy forms * Restore updates - added spacer between title and data streams callout - added copy to the restore settings tab to indicate that settings also apply to backing indices * further copy refinements * second round of copy feedback * fix i18n * added comment to mock * line spacing fixes and created issue for tracking backing index discovery in snaphots * refactor collapsible list logic and tests * refactor editing managed policy check * refactor copy to be clearer about pluralisation of data streams * refactor file structure in components for data stream badge * added tests for indices and data streams field helper * refactored types and fixed i18n id per guidelines Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

* master: Update known-plugins.asciidoc (elastic#69370) [ML] Anomaly Explorer swim lane pagination (elastic#70063) [Ingest Manager] Update asset paths to use _ instead of - (elastic#70320) Fix discover, tsvb and Lens chart theming issues (elastic#69695) Allow Saved Object type mappings to set a field's `doc_values` property (elastic#70433) [S&R] Support data streams (elastic#68078) [Maps] Add styling and tooltip support to mapbox mvt vector tile sources (elastic#64488) redirect to default app if hash can not be forwarded (elastic#70417) [APM] Don't fetch dynamic index pattern in setupRequest (elastic#70308) [Security_Solution][Endpoint] Leveraging msearch and ancestry array for resolver (elastic#70134) Update docs for api authentication usage (elastic#66819) chore(NA): disable alerts_detection_rules cypress suites (elastic#70577) add getVisibleTypes API to SO type registry (elastic#70559)

* Sort endpoint responses into indices and datastreams The server endpoint for policies now returns data streams and filters out backing indices from the indices array it returned previously * Refactor indices switch and field out of the step settings file * Fix indices field form behaviour * WiP on UI. Added the second table per mockup for add and edit. * add support for creating a policy that backs up data streams end to end * wip on restore flow - added data streams to server response * add logic for detecting whether an index is part of a data stream * fix public jest tests * fix server side jest tests * pivot to different solution in UI while we do not have data streams nicely separated * added data stream to snapshot summary details * move the data streams badge file closer to where it used * add data stream badge when restoring snapshots too * update restore copy * fix pattern specification in indices and data streams field * first iteration of complete policy UX * First iteration that is ready for review Given the contraints on working with data streams and indices in policies at the moment the simplest implementation is to just include data streams with indices and have the user select them there for now. The way snapshotting behaviour is currently implemented relies entirely on what is specified inside of "indices", this is also where data streams must be placed. This unfortunately means that capture patterns defined in indices will capture entire data streams too. * delete unused import * fix type issue in tests * added logic for rendering out previous selection as custom pattern * refactor indices fields to make component smaller * added CIT for data streams badge * Data streams > indices * updates to relevant pieces of copy * more copy updates * fix types and remove unused import * removed backing indices from restore view * Added data stream restore warning message * restore CITs * first round of copy feedback * refactor help text to provide clearer feedback, for both restore and policy forms * Restore updates - added spacer between title and data streams callout - added copy to the restore settings tab to indicate that settings also apply to backing indices * further copy refinements * second round of copy feedback * fix i18n * added comment to mock * line spacing fixes and created issue for tracking backing index discovery in snaphots * refactor collapsible list logic and tests * refactor editing managed policy check * refactor copy to be clearer about pluralisation of data streams * refactor file structure in components for data stream badge * added tests for indices and data streams field helper * refactored types and fixed i18n id per guidelines Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

jloleysens added 4 commits June 3, 2020 12:15

Sort endpoint responses into indices and datastreams

d056b29

The server endpoint for policies now returns data streams and filters out backing indices from the indices array it returned previously

Refactor indices switch and field out of the step settings file

4af5a8d

Fix indices field form behaviour

8861ec2

WiP on UI. Added the second table per mockup for add and edit.

f219f48

jloleysens added v8.0.0 Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more release_note:skip Skip the PR/issue when compiling release notes Feature:Snapshot and Restore Elasticsearch snapshots and repositories UI v7.9.0 labels Jun 3, 2020

Merge branch 'master' into slm/support-data-streams

b4b3d34

elasticmachine and others added 3 commits June 23, 2020 01:48

Merge branch 'master' into slm/support-data-streams

b6394a8

add support for creating a policy that backs up data streams end to end

dcb1764

wip on restore flow - added data streams to server response

8db5cd5

elasticmachine and others added 13 commits June 24, 2020 03:42

Merge branch 'master' into slm/support-data-streams

aa6fdde

add logic for detecting whether an index is part of a data stream

86ea659

fix public jest tests

a153c7f

fix server side jest tests

848e76f

pivot to different solution in UI while we do not have data streams n…

45f22e9

…icely separated

added data stream to snapshot summary details

31768fa

move the data streams badge file closer to where it used

12a4e80

add data stream badge when restoring snapshots too

23cd98b

update restore copy

ad0b6c6

fix pattern specification in indices and data streams field

f57af3c

first iteration of complete policy UX

f513474

delete unused import

904fa39

Restore updates

0b15b33

- added spacer between title and data streams callout - added copy to the restore settings tab to indicate that settings also apply to backing indices

elasticmachine and others added 2 commits July 1, 2020 05:59

Merge branch 'master' into slm/support-data-streams

c6351a1

further copy refinements

edadf62

jrodewig approved these changes Jul 1, 2020

View reviewed changes

second round of copy feedback

8001615

elasticmachine and others added 2 commits July 1, 2020 07:47

Merge branch 'master' into slm/support-data-streams

4a93135

fix i18n

04f60e1

cjcenizal approved these changes Jul 1, 2020

View reviewed changes

elasticmachine and others added 9 commits July 2, 2020 03:04

Merge branch 'master' into slm/support-data-streams

c2fd2da

added comment to mock

f152da9

line spacing fixes and created issue for tracking backing index disco…

132506c

…very in snaphots

refactor collapsible list logic and tests

d2551b8

refactor editing managed policy check

ab8719d

refactor copy to be clearer about pluralisation of data streams

7d2feee

refactor file structure in components for data stream badge

65eb37e

added tests for indices and data streams field helper

2fd876f

refactored types and fixed i18n id per guidelines

2e46e93

jloleysens merged commit 0066c4b into elastic:master Jul 2, 2020

jloleysens deleted the slm/support-data-streams branch July 2, 2020 13:38

jloleysens mentioned this pull request Jul 2, 2020

[7.x] [S&R] Support data streams (#68078) #70595

Merged

cjcenizal mentioned this pull request Jul 29, 2020

Add data stream support to Snapshot Restore SLM policies #65132

Closed

jloleysens mentioned this pull request Aug 31, 2020

[Snapshot & Restore] Decode URIs and fix editing of a policy #76278

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[S&R] Support data streams #68078

[S&R] Support data streams #68078

jloleysens commented Jun 3, 2020 •

edited

Loading

jloleysens commented Jun 22, 2020

jloleysens commented Jun 23, 2020

jloleysens commented Jun 24, 2020

jloleysens commented Jun 26, 2020

jloleysens commented Jul 1, 2020

jloleysens commented Jul 1, 2020

jrodewig left a comment

jrodewig Jul 1, 2020

jloleysens commented Jul 1, 2020

cjcenizal left a comment

cjcenizal Jul 1, 2020

jloleysens Jul 2, 2020

cjcenizal Jul 2, 2020

jloleysens commented Jul 2, 2020

kibanamachine commented Jul 2, 2020

[S&R] Support data streams #68078

[S&R] Support data streams #68078

Conversation

jloleysens commented Jun 3, 2020 • edited Loading

Summary

How to test

Create a data stream

Create an index template

Create a data stream

Checklist

Screenshots

Data streams and indices

Snapshot details view

Restoring a snapshot

jloleysens commented Jun 22, 2020

jloleysens commented Jun 23, 2020

jloleysens commented Jun 24, 2020

jloleysens commented Jun 26, 2020

jloleysens commented Jul 1, 2020

jloleysens commented Jul 1, 2020

jrodewig left a comment

Choose a reason for hiding this comment

jrodewig Jul 1, 2020

Choose a reason for hiding this comment

jloleysens commented Jul 1, 2020

cjcenizal left a comment

Choose a reason for hiding this comment

cjcenizal Jul 1, 2020

Choose a reason for hiding this comment

jloleysens Jul 2, 2020

Choose a reason for hiding this comment

cjcenizal Jul 2, 2020

Choose a reason for hiding this comment

jloleysens commented Jul 2, 2020

kibanamachine commented Jul 2, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

History

jloleysens commented Jun 3, 2020 •

edited

Loading