[Snapshot & Restore] NP migration #59109

sebelga · 2020-03-03T07:07:06Z

This PR migrates the Snapshot & Restore app to the NP "plugins" folder.

How to test

Add a repository

Local file system

In order to test the snapshot feature locally we will need to have a folder to store the snapshots.

mkdir /tmp/es-backups

We can then launch ES and define this path as a known repository

// Launch ES
yarn es snapshot --license trial -E path.repo=/tmp/es-backups

// Launch Kibana
yarn start

Navigate to the Snapshot and Restore app, and to the "Repositories" tab.

Register a new repository
Give it a name, chose "Shared file system", click "Next"
In the "Location" field add "/tmp/es-backups"
Click "Register"

Create an index

We won't be able to restore system indices that are not closed so let's create an index for our snapshot. Go to Console and create an index

PUT my-index

Add a policy

Back to the Snapshot and restore app:

Create a new policy
Give it a name (eg. "my-daily-policy") and also to the snapshot (eg. "<my-snapshot-{now/d}>")
Check that the "Repository" field is selected on the repository we created above
Click "Next" and under "Snapshot settings" disable "All indices" and only select the index you just created in Console
Click "Next". Define in you wish a retention
Create the policy

Create a snapshot

In the policy list table, click the "Run policy" (play icon) button in the table actions column
Navigate to the "Snapshots" tab, you should see the snapshot that has been created
Click on it to see the detail

Restore from snapshot

In order to restore the snapshot we need to close the index we created above.

Navigate to the Index management app and close the "my-index" index
Go back to Snapshot and Restore and under the "Snapshots" tab, click the "Restore" button on the snapshot that has been created
Once the restore is finished, you should be taken to the "Restore Status" tab with the detail of the restore. You can click on the "expand" down arrow to see the detail of the restore

Disable the app

Shut down the Kibana server
Add xpack.snapshot_restore.enabled: false in your kibana.dev.yml file
Start Kibana
The app should not appear under "Management / Elasticsearch"

Disable Snapshot Lifecycle Management

Shut down the Kibana server
Comment out the xpack.snapshot_restore.enabled: false we just added in the kibana.dev.yml
Add xpack.snapshot_restore.slmUi.enabled: false in your kibana.dev.yml file
Start Kibana
Navigate to Snapshot and Restore app: the "Policies" tab should not be present

Now re-enable SLM

Shut down the Kibana server
Comment out the xpack.snapshot_restore.slmUi.enabled: false we just added
Start Kibana

Security

A user needs to have the following roles in order to access the Snapshot and Restore app

cluster: "cluster:admin/snapshot", "cluster:admin/repository"
index: "monitor"

Additionally, a user needs to have the

cluster: "cluster:manage_slm" AND "cluster:monitor/state" (this last one is new, I just discovered it when testing) privilege to access the "Policies" tab, even when it is enabled in Kibana config

Let's create a role that does not have the minimum privileges to access the app

Navigate to "Management/security/roles" and Create a new role "test-sr-security"
Under "Elasticsearch" cluster privileges add the "monitor" privilege
Under "Kibana" add "All" privileges
Save the role
Create a user "test-sr-security", give it a password and attach the role you just created
Log out of Kibana and log in with that user
Navigate to the app and you should have a "Missing cluster privileges" message.
Log out of Kibana and back with the "elastic" username
Edit the role we previously created and add MANUALLY (does not appear in the dropdown)
- "cluster:admin/snapshot" & "cluster:admin/repository" cluster privilege
- AND the "monitor" privilege to all indices (using the wildcard "*")
Log out and in with the "test-sr-security" user. You should now be able to access the app. But if you navigate to the "Policies" tab you should get a warning about missing privileges.
Log out of Kibana and back with the "elastic" username
Edit the role we previously created and add the "manage_slm" privilege to Elasticsearch
Log out and in with the "test-sr-security" user. The "Policies" tab should be visible.

Disable security

Shut down Kibana
Add xpack.security.enabled: false in the kibana.dev.yml file
Start Kibana
Without any user, the app and "Policies" tab should be visible

Add a plugin repository

AWS S3

Shut down Elasticsearch
cd .es/8.0.0
Install the s3 plugin

bin/elasticsearch-plugin install https://snapshots.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-8.0.0-SNAPSHOT.zip

Go to the AWS S3 management and create a bucket
Create an IAM user in AWS and give it access to S3 (Permissions "AmazonS3FullAccess")
Copy the "Access key" for the user and add it to the keystore

bin/elasticsearch-keystore add s3.client.default.access_key

Copy the "Secret key" for the user and add it to the keystore

bin/elasticsearch-keystore add s3.client.default.secret_key

Start Elasticsearch with the "bin" command (not yarn es snapshot!)

bin/elasticsearch -E path.repo=/tmp/es-backups

In Kibana you should be able to create a repository and select "S3" type. Set the bucket name you created and register the new repo.
Edit the policy created earlier and chose the S3 repository, then save the policy.
Create a Snapshot from the policy. You should see it created in your S3 bucket.
Restore from Snapshot should work as well.

sebelga · 2020-03-03T07:28:55Z

...apshot_restore/public/application/sections/home/restore_list/restore_table/restore_table.tsx

    [key: string]: React.ReactNode;
  }>({});

-  // On sorting and pagination change


I re-wrote this component as there was an infinite re-render due to a state being update inside a useEffect and with the new eslint rule the state being updated had to be put in the array of dependencies.

elasticmachine · 2020-03-03T07:39:58Z

Pinging @elastic/es-ui (Team:Elasticsearch UI)

kibanamachine · 2020-03-04T11:55:58Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: d2292f0

History

💚 Build #30627 succeeded 0bfe09a
💔 Build #30424 failed 6088592a2698a0ff70229f31bf9906ecc9bfc2cb
💔 Build #30278 failed 133588d02f129416fa07f4075f04c30f26a58594
💔 Build #30271 failed 8a7c34a24cc5bd1d3e67134fc6c2af1c8d0b9880

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

jloleysens · 2020-03-04T15:06:36Z

x-pack/plugins/snapshot_restore/public/application/services/http/restore_requests.ts

+export const setUiMetricServiceRestore = (_uiMetricService: UiMetricService) => {
+  uiMetricService = _uiMetricService;
+};
+// End hack


😂 🖐🏻 🎤

jloleysens

Great work @sebelga ! I did not find any merge blockers but left some comments for discussion.

jloleysens · 2020-03-04T15:12:05Z

x-pack/plugins/snapshot_restore/server/plugin.ts

+    this.license = new License();
+  }
+
+  public async setup(


Not sure when, but async setup is going to be deprecated in the future and it will become sync code only

Ok. Let's wait and see. I guess we'll all have to go and update all our plugins then.

jloleysens · 2020-03-04T15:36:53Z

x-pack/plugins/snapshot_restore/server/plugin.ts

+        isCloudEnabled: cloud !== undefined && cloud.isCloudEnabled,
+        isSlmEnabled: pluginConfig.slmUi.enabled,
+      },
+      lib: {


This is quite cool a cool idea! I hadn't thought of putting commonly used lib (pure) functions in here like this.

I also find it very useful. If there is a need for another lib, add it to the dependency object and it is available on all routes without needing to add imports. Or if we change the route handler file path, no need to worry to update its imports.

jloleysens · 2020-03-04T15:46:37Z

x-pack/plugins/snapshot_restore/server/test/helpers/router_mock.ts

@@ -0,0 +1,113 @@
+/*


I wonder if we can get something like this available from core for all tests like this.

It would also be cool to test validation in something like this, there is a similar kind of hackery going on elsewhere (like in upgrade assistant), but I think this iteration is quite clean 👍. Perhaps something we can open an issue for (if there is not one already).

Let see first if more apps need this mock. I had to find a solution for SR as the app had so many written unit tests (and no API integration tests). 👍

sebelga · 2020-03-05T09:36:20Z

Thanks for the review @jloleysens !
I will merge this as the only design change are .scss files that have moved. @elastic/kibana-design let me know if there is anything I overlooked and I will add it in a separate PR 👍

…re/files-and-filetree * 'master' of github.com:elastic/kibana: (254 commits) Convert discover_page to ts, remove redundunt methods (elastic#59312) [Fix for Vis Editor] Revert setting time field to empty string when it's undefined (elastic#58873) Delete legacy search endpoint (elastic#59341) [Uptime] Improve duration chart (elastic#58404) [Snapshot & Restore] NP migration (elastic#59109) [ML] Add support for date_nanos time field in anomaly job wizard (elastic#59017) Revert "Makes alerting and actions optional properties for interface RequestH… (elastic#59264)" Change remote_clusters ID to remoteClusters (elastic#59246) Makes alerting and actions optional properties for interface RequestH… (elastic#59264) Clean up date histogram agg type. (elastic#58805) [ML] Management: fix license unsubscribe (elastic#59365) Remove documentation for server.cors settings (elastic#59096) Edit alert flyout (elastic#58964) [SIEM] Fix rule delete/duplicate actions (elastic#59306) move mouse to close obstructing tooltip (elastic#59214) Reset page after deleting (elastic#59310) Make sure phrases input filter triggers autosuggestons (elastic#59299) Add loading count source for http requests (elastic#59245) Revert "[ML] Transforms: Deprecate custom KibanaContext. (elastic#59133)" Expose metrics service to public API (elastic#59294) ... # Conflicts: # src/plugins/console/public/application/containers/editor/legacy/console_editor/editor.tsx

kibanamachine · 2020-03-06T09:38:44Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

sebelga requested a review from a team as a code owner March 3, 2020 07:07

sebelga commented Mar 3, 2020

View reviewed changes

sebelga added Feature:NP Migration Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more v7.7.0 v8.0.0 labels Mar 3, 2020

sebelga added the release_note:skip Skip the PR/issue when compiling release notes label Mar 3, 2020

sebelga added 22 commits March 4, 2020 09:32

Refactor server plugin and add License service

09f9e59

Wip app.ts api route

9f9331b

Update snapshots api routes

cd9136e

Add optional cloud plugin dep + wrap_es_error lib to route deps

c14feb6

Enable repositories routes

57fc805

Update app.ts api routes

aeb1c63

Update repositorySchema validation

e9ebf0d

Update repositorySchema validation (2)

387ab93

Update policy api routes

ecefa47

Update restore api routes

e89bfe1

Add helpers to test routes in unit test (jest)

8713f43

Fix restore.test.ts

a15c4ac

Fix api route policy.test

0bf6b27

Fix api route restore.test

1190aac

Fix snapshot API route tests

58bef16

Fix repositories API route tests

8613d39

Fix policy test

2a06667

Move test helpers to dedicate folder

51efb14

Use custom elasticsearch client in route context

25db004

Use POST verb to create policy + silently fail when not finding policy

86ca670

Update policy payload schema

d20d4ac

Update tests to use custom ES client from context

9ccaef5

sebelga added 12 commits March 4, 2020 09:32

[Send request hook]: allow generic Error response interface

345977f

Update CustomError interface in sendRequest/useRequest

4622366

Refactor <RestoreTable /> component to avoid infinite re-render

67e1b41

Read slmUi enabled config from yaml file

66434af

Add optional "security" plugin dependency

5785a02

Fix client integration tests

fa6e0ad

[Send request hook]: fix generic response returned

37af086

Remove "usageCollection" from AppDependencies

c9a6a7a

Refactor: use "setup" instead of "init" in TextService

468d132

Small refactor

c7549f3

Fix TS issues

e2bc2d0

Fix TS issues (2)

0bfe09a

sebelga force-pushed the np-migration/snapshot-restore branch from 6088592 to 0bfe09a Compare March 4, 2020 04:03

Add missing SLM cluster privilege

d2292f0

jloleysens reviewed Mar 4, 2020

View reviewed changes

jloleysens approved these changes Mar 4, 2020

View reviewed changes

sebelga merged commit a05c321 into elastic:master Mar 5, 2020

sebelga deleted the np-migration/snapshot-restore branch March 5, 2020 09:37

alisonelizabeth mentioned this pull request Mar 5, 2020

[SR] Selected indices do not show when editing a policy created via console #55219

Closed

4 tasks

sebelga mentioned this pull request Mar 6, 2020

[7.x] [Snapshot & Restore] NP migration (#59109) #59520

Merged

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 6, 2020

sebelga added a commit that referenced this pull request Mar 6, 2020

[7.x] [Snapshot & Restore] NP migration (#59109) (#59520)

de404f5

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Mar 6, 2020

cjcenizal mentioned this pull request Mar 9, 2020

Fix slm_ui setting by changing camel case back to snake case. #59663

Merged

alisonelizabeth mentioned this pull request Apr 19, 2021

[SR] Error handling not working as expected for fetching repositories #56442

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Snapshot & Restore] NP migration #59109

[Snapshot & Restore] NP migration #59109

sebelga commented Mar 3, 2020 •

edited

Loading

sebelga Mar 3, 2020

elasticmachine commented Mar 3, 2020

kibanamachine commented Mar 4, 2020

jloleysens Mar 4, 2020

jloleysens left a comment

jloleysens Mar 4, 2020

sebelga Mar 5, 2020

jloleysens Mar 4, 2020

sebelga Mar 5, 2020

jloleysens Mar 4, 2020

sebelga Mar 5, 2020

sebelga commented Mar 5, 2020

kibanamachine commented Mar 6, 2020

[Snapshot & Restore] NP migration #59109

[Snapshot & Restore] NP migration #59109

Conversation

sebelga commented Mar 3, 2020 • edited Loading

How to test

Add a repository

Local file system

Create an index

Add a policy

Create a snapshot

Restore from snapshot

Disable the app

Disable Snapshot Lifecycle Management

Security

Disable security

Add a plugin repository

AWS S3

Choose a reason for hiding this comment

elasticmachine commented Mar 3, 2020

kibanamachine commented Mar 4, 2020

💚 Build Succeeded

History

Choose a reason for hiding this comment

jloleysens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sebelga commented Mar 5, 2020

kibanamachine commented Mar 6, 2020

sebelga commented Mar 3, 2020 •

edited

Loading