[Alerting] Export rules and connectors #98802

ymao1 · 2021-04-29T17:51:19Z

Resolves #94151

Summary

Adds ability to export rules and connectors from the Saved Objects Management UI.

Rules will be exported as disabled, with apiKey information cleared.
Connectors will be exported with isMissingSecrets: true unless
- Index and server log connectors have no secrets, so isMissingSecrets will not be set to true
- Email and webhook connectors where hasAuth: false no secrets, so isMissingSecrets will not be set to true
Auditing the export event
Added hidden saved object types that are importableAndExportable to the Saved Object Management feature privilege.

How Rules and Connectors show up on Saved Objects Management UI

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

… object feature privilege

…logging

…ing/export-rules-and-connectors

elasticmachine · 2021-04-29T22:49:39Z

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

pgayvallet

LGTM, a few remarks.

pgayvallet · 2021-04-30T07:15:07Z

x-pack/plugins/features/server/plugin.ts

+    const savedObjectImportableAndExportableHiddenTypes = registry
+      .getImportableAndExportableTypes()
+      .filter((t) => registry.isHidden(t.name))
+      .map((t) => t.name);
+
+    const savedObjectTypes = Array.from(
+      new Set([...savedObjectVisibleTypes, ...savedObjectImportableAndExportableHiddenTypes])
+    );


Good catch.

x-pack/plugins/alerting/server/saved_objects/transform_rule_for_export.ts

pgayvallet · 2021-04-30T07:43:26Z

x-pack/plugins/actions/server/saved_objects/index.ts

+      onExport<RawAction>(
+        context: SavedObjectsExportTransformContext,
+        objects: Array<SavedObject<RawAction>>


(Referring to the getTitle function on top, as comments are not allowed on non-diff lines...)

Ideally, we shouldn't need the object's type to be displayed in its associated title, as it introduces redundancy in some parts of the SOM UI.

Now, the main SOM table is only using an (EUI) icon to display the type associated with each object. Is there an icon associated with rules and connectors in the EUI library? If so, you could add them via the management.icon property of the types.

@pgayvallet Discussed with @mdefazio and we currently don't have dedicated icons for rules or connectors. If we're using the default icon, should we leave Rules: and Connectors: in the title?

I guess so. This is low impact and can be changed later anyway, so I think it's fine.

Thanks for being flexible. Our concern is associating icons with Rules and Connectors that may also be used elsewhere for different concepts. The viz type icons were created specifically for those types and aren't used for other things (or shouldn't be).

If we want an icon, we will just need to go the route of creating on specifically for this. However, this type column may quickly turn into a lot of different icons and likely need the title to be shown alongside them anyway.

So let's circle back to it in the future and determine if we want to only show the title.

…ing/export-rules-and-connectors

docs/management/action-types.asciidoc

docs/user/alerting/rule-management.asciidoc

YulNaumenko · 2021-05-03T21:15:44Z

x-pack/plugins/actions/server/saved_objects/transform_connectors_for_export.ts

+import { SavedObject } from 'kibana/server';
+import { RawAction } from '../types';
+
+const CONNECTORS_WITHOUT_SECRETS = ['.index', '.server-log'];


Maybe here we can replace the constants with the dynamic approach to check the action type schema?

@YulNaumenko Great idea! I updated this in this commit 58dfb8a

I ran into the issue where the email and webhook secrets validation does not take into account whether hasAuth is set to false inside the connector parameters so in this case, if the secrets validation fails, I'm still checking for the existence of hasAuth. Do you think that works?

…ing/export-rules-and-connectors

gchaps

LGTM

YulNaumenko

Changes LGTM!

ymao1 · 2021-05-04T16:35:59Z

@elasticmachine merge upstream

…ing/export-rules-and-connectors

….com/ymao1/kibana into alerting/export-rules-and-connectors

chrisronline · 2021-05-05T13:18:14Z

Looking good!

One thing I noticed:

In this scenario, I don't have the right license level to use the connector. Did we build a specific experience to tell the user this at import time? I think what we have now is okay for now (and we could file an issue to improve it in the future) but I wasn't sure if it was supposed to show more

chrisronline · 2021-05-05T13:24:45Z

Also, I have a connector that requires secrets, but I didn't export it. Instead, I just exported a server log connector and I see this:

{"attributes":{"actionTypeId":".server-log","config":{},"isMissingSecrets":true,"name":"Default"},"coreMigrationVersion":"8.0.0","id":"93c19030-ada4-11eb-8afc-57c69e570459","migrationVersion":{"action":"7.14.0"},"references":[],"type":"action","updated_at":"2021-05-05T13:19:53.025Z","version":"WzI1LDFd"}

ymao1 · 2021-05-05T13:28:12Z

In this scenario, I don't have the right license level to use the connector. Did we build a specific experience to tell the user this at import time? I think what we have now is okay for now (and we could file an issue to improve it in the future) but I wasn't sure if it was supposed to show more

I added a comment to the import issue: #94152

ymao1 · 2021-05-05T13:36:22Z

Also, I have a connector that requires secrets, but I didn't export it. Instead, I just exported a server log connector and I see this:

{"attributes":{"actionTypeId":".server-log","config":{},"isMissingSecrets":true,"name":"Default"},"coreMigrationVersion":"8.0.0","id":"93c19030-ada4-11eb-8afc-57c69e570459","migrationVersion":{"action":"7.14.0"},"references":[],"type":"action","updated_at":"2021-05-05T13:19:53.025Z","version":"WzI1LDFd"}

Should be fixed now! Unit tests actually caught that one too! 😅

chrisronline

LGTM! I'm not too familiar with the code yet so I just focused on the functionality. Looks great, nice work!

kibanamachine · 2021-05-05T15:26:29Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💔 Build #124404 failed 6d3d7ee
💛 Build #124187 was flaky c4d4bf4
💛 Build #124128 was flaky 58dfb8a
💚 Build #124110 succeeded 47ebbf5
💚 Build #123726 succeeded b5e04cd

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ymao1

* Adding importableAndExportable but hidden saved object types to saved object feature privilege * Adding helper function for transforming rule for export. Added audit logging * Adding helper function for transforming rule for export. Added audit logging * Adding unit test for transforming rules for export * Exporting connectors * Removing auditing during export * Adding import/export to docs * PR fixes * Using action type validation onExport * Fixing logic for connectors with optional secrets * Fixing logic for connectors with optional secrets Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-05-05T15:35:32Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

* Adding importableAndExportable but hidden saved object types to saved object feature privilege * Adding helper function for transforming rule for export. Added audit logging * Adding helper function for transforming rule for export. Added audit logging * Adding unit test for transforming rules for export * Exporting connectors * Removing auditing during export * Adding import/export to docs * PR fixes * Using action type validation onExport * Fixing logic for connectors with optional secrets * Fixing logic for connectors with optional secrets Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: ymao1 <ying.mao@elastic.co>

@nastasha-solomon

…148703) ## Summary - [x] Addresses #118774 - [x] Enable Security Rule to be **imported** even if one of its connectors has a missing secret - [x] Shows **Warning Callout** in the Import Modal when missing secrets connector is imported. - [x] Added Link `connectors` to the connectors page in the same tab, so that the user can fix imported connectors. - [x] Added `Overwrite existing connectors with conflicting action "id"` option to the Import Modal ## Cases: > **Export:** > - Export Rule(s) with connectors through `Export All` or `Bulk Actions` > > **Import:** > - Import Rule with correct connectors data > - Import Rule with missing secrets' connectors by showing a warning callout > - Re-Import connectors even if they were stored when overwrite is true > > **Error:** > - Showing an error message when the user has a Read Actions permission and stops the importing => ` You may not have actions privileges required to import rules with actions ...` > - Showing an error message when the user has an old imported rule missing all the connectors data **OR** these connectors were not in the user's env => `X connector is missing. Connector id missing is: X` > - Showing an error if the new connectors defined in the exported file are not corresponding to the actions array under the rules param => `X connector is missing. Connector id missing is: X` > - **Showing a ` conflict` error in case of existing connectors and re-importing again with an `overwrite` false => this won't happen in case of implementing the `Skipping action-connectors importing if all connectors have been imported/created before`** > > **Skip importing:** > - Skipping action-connectors importing if the `actions` array is empty, even if the user has exported-connectors in the file > - Skipping action-connectors importing if all connectors have been imported/created before > ### Screenshots > > **1. Importing Connectors successfully** > <img width="1219" alt="image" src="https://user-images.githubusercontent.com/12671903/216049657-a313033b-e45e-4c99-b6ca-ed3070f15a97.png"> > > **2. Importing Connectors with warnings** <img width="1208" alt="image" src="https://user-images.githubusercontent.com/12671903/216980057-b5cdfe38-da1b-479b-8cfd-81f16037ff1d.png"> **3.Connector Page** <img width="1701" alt="image" src="https://user-images.githubusercontent.com/12671903/216049911-da29abc8-e20c-49d2-a507-ab382372b4f6.png"> ## New text: @nastasha-solomon **1. Warning message** title => could be ` 1 connector imported` or `x connectors imported` message => ` 1 connector has sensitive information that requires updates. review in connectors` or `x connectors have sensitive information that requires updates. review in connectors` <img width="588" alt="image" src="https://user-images.githubusercontent.com/12671903/216103805-9946b080-07d3-4e8b-93aa-b5e1dcaa415d.png"> **2. New `Overwrite` checkbox** <img width="431" alt="image" src="https://user-images.githubusercontent.com/12671903/216106354-3d435d64-0fa5-467b-90f1-effb2c0aef2a.png"> **3. Success Toast message** <img width="434" alt="image" src="https://user-images.githubusercontent.com/12671903/216104454-2d83744b-efbc-40c1-9e69-7e8b0670dd19.png"> **4. Error messages** a. Missing import action privileges <img width="438" alt="image" src="https://user-images.githubusercontent.com/12671903/216116350-f306d744-eef4-4064-b4f8-e794db4ad78e.png"> b. Missing connectors <img width="353" alt="image" src="https://user-images.githubusercontent.com/12671903/216104979-370f6826-8150-45d5-8724-6ca50f99ad71.png"> <img width="357" alt="image" src="https://user-images.githubusercontent.com/12671903/216106067-e6132a93-d36e-4bdf-b1bf-e6ddd1cf8a4e.png"> - [x] References: Use **getImporter** and **getExporter** from Saved Object [Connectors SO import/export implementation](#98802) , [Kibana-Core confirmation](https://elastic.slack.com/archives/C5TQ33ND8/p1673275186013589 ) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

ymao1 added 6 commits April 28, 2021 14:58

Adding importableAndExportable but hidden saved object types to saved…

bd1ea5c

… object feature privilege

Adding helper function for transforming rule for export. Added audit …

ae7b219

…logging

Adding helper function for transforming rule for export. Added audit …

6188455

…logging

Merge branch 'master' of https://github.com/elastic/kibana into alert…

0f0d65e

…ing/export-rules-and-connectors

Adding unit test for transforming rules for export

9b84a07

Exporting connectors

c66de1f

ymao1 self-assigned this Apr 29, 2021

ymao1 added Feature:Alerting release_note:enhancement Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v7.14.0 v8.0.0 labels Apr 29, 2021

ymao1 marked this pull request as ready for review April 29, 2021 22:49

ymao1 requested review from a team as code owners April 29, 2021 22:49

pgayvallet approved these changes Apr 30, 2021

View reviewed changes

ymao1 added 2 commits April 30, 2021 19:01

Merge branch 'master' of https://github.com/elastic/kibana into alert…

2621aff

…ing/export-rules-and-connectors

Removing auditing during export

df73b5f

ymao1 mentioned this pull request Apr 30, 2021

[Alerting][Connectors] Import rules and connectors #94152

Closed

Adding import/export to docs

b5e04cd

ymao1 requested a review from gchaps April 30, 2021 23:21

gchaps reviewed May 3, 2021

View reviewed changes

docs/management/action-types.asciidoc Outdated Show resolved Hide resolved

docs/user/alerting/rule-management.asciidoc Outdated Show resolved Hide resolved

YulNaumenko reviewed May 3, 2021

View reviewed changes

ymao1 added 2 commits May 4, 2021 07:54

Merge branch 'master' of https://github.com/elastic/kibana into alert…

aa132b9

…ing/export-rules-and-connectors

PR fixes

47ebbf5

ymao1 requested a review from gchaps May 4, 2021 12:19

Using action type validation onExport

58dfb8a

gchaps approved these changes May 4, 2021

View reviewed changes

YulNaumenko approved these changes May 4, 2021

View reviewed changes

Merge branch 'master' into alerting/export-rules-and-connectors

c4d4bf4

chrisronline self-requested a review May 5, 2021 01:33

ymao1 added 3 commits May 5, 2021 08:18

Fixing logic for connectors with optional secrets

41e7e7a

Merge branch 'master' of https://github.com/elastic/kibana into alert…

6a0d61c

…ing/export-rules-and-connectors

Merge branch 'alerting/export-rules-and-connectors' of https://github…

6d3d7ee

….com/ymao1/kibana into alerting/export-rules-and-connectors

Fixing logic for connectors with optional secrets

3be3824

chrisronline approved these changes May 5, 2021

View reviewed changes

ymao1 added the auto-backport Deprecated - use backport:version if exact versions are needed label May 5, 2021

ymao1 merged commit 4ab86c7 into elastic:master May 5, 2021

kibanamachine mentioned this pull request May 5, 2021

[7.x] [Alerting] Export rules and connectors (#98802) #99361

Merged

spalger mentioned this pull request May 6, 2021

[sec/actions] move constants to a separate file to avoid circular deps #99439

Merged

ymao1 deleted the alerting/export-rules-and-connectors branch May 19, 2021 13:50

doakalexi mentioned this pull request Aug 4, 2022

[ResponseOps][Actions] Getting error about secrets not being saved when import a SO (Connector Saved Object) #138019

Merged

1 task

WafaaNasr mentioned this pull request Jan 18, 2023

[Security Solution][Platform] - Add connectors to import/export API #148703

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Alerting] Export rules and connectors #98802

[Alerting] Export rules and connectors #98802

ymao1 commented Apr 29, 2021 •

edited

Loading

elasticmachine commented Apr 29, 2021

pgayvallet left a comment

pgayvallet Apr 30, 2021

pgayvallet Apr 30, 2021

ymao1 Apr 30, 2021

pgayvallet May 2, 2021

mdefazio May 4, 2021

YulNaumenko May 3, 2021

ymao1 May 4, 2021

gchaps left a comment

YulNaumenko left a comment

ymao1 commented May 4, 2021

chrisronline commented May 5, 2021

chrisronline commented May 5, 2021

ymao1 commented May 5, 2021

ymao1 commented May 5, 2021

chrisronline left a comment

kibanamachine commented May 5, 2021

kibanamachine commented May 5, 2021

[Alerting] Export rules and connectors #98802

[Alerting] Export rules and connectors #98802

Conversation

ymao1 commented Apr 29, 2021 • edited Loading

Summary

Checklist

elasticmachine commented Apr 29, 2021

pgayvallet left a comment

Choose a reason for hiding this comment

pgayvallet Apr 30, 2021

Choose a reason for hiding this comment

pgayvallet Apr 30, 2021

Choose a reason for hiding this comment

ymao1 Apr 30, 2021

Choose a reason for hiding this comment

pgayvallet May 2, 2021

Choose a reason for hiding this comment

mdefazio May 4, 2021

Choose a reason for hiding this comment

YulNaumenko May 3, 2021

Choose a reason for hiding this comment

ymao1 May 4, 2021

Choose a reason for hiding this comment

gchaps left a comment

Choose a reason for hiding this comment

YulNaumenko left a comment

Choose a reason for hiding this comment

ymao1 commented May 4, 2021

chrisronline commented May 5, 2021

chrisronline commented May 5, 2021

ymao1 commented May 5, 2021

ymao1 commented May 5, 2021

chrisronline left a comment

Choose a reason for hiding this comment

kibanamachine commented May 5, 2021

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented May 5, 2021

💚 Backport successful

ymao1 commented Apr 29, 2021 •

edited

Loading