Add conditional installation for S3 integrations #1518

Swiddis · 2024-03-11T20:44:29Z

Description

Adds rudimentary support for toggling some parts of the integration installation process. There's still some parts that need to be updated:

The build process on the backend still can't check if it's supposed to load saved objects/which ones. We need to include workflows as part of the build request.
The UI for the checkbox still needs a nicer header.
- Since the UI is larger now it also is probably a good idea to fix the clipping with the bottom bar overlapping the content on smaller screens.
It'd be nice to support enabling workflows for different data source types, but this might be complex enough to warrant a different issue and PR.
Add workflows to remaining S3 integrations

Issues Resolved

Closes #1462

Check List

New functionality includes testing.
- All tests pass, including unit test, integration test and doctest
New functionality has been documented.
- New functionality has javadoc added
- New functionality has user manual doc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

codecov · 2024-03-11T20:58:00Z

Codecov Report

Attention: Patch coverage is 72.22222% with 15 lines in your changes are missing coverage. Please review.

Project coverage is 56.66%. Comparing base (d17cad3) to head (35a3bfc).

Files	Patch %	Lines
...ents/integrations/components/setup_integration.tsx	54.54%	15 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1518      +/-   ##
==========================================
+ Coverage   56.62%   56.66%   +0.03%     
==========================================
  Files         348      348              
  Lines       12656    12686      +30     
  Branches     3202     3271      +69     
==========================================
+ Hits         7167     7189      +22     
- Misses       5436     5443       +7     
- Partials       53       54       +1

Flag	Coverage Δ
dashboards-observability	`56.66% <72.22%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis · 2024-03-11T23:10:41Z

server/adaptors/integrations/integrations_builder.ts

+    return builtInstance;
+  }
+
+  getSavedObjectBundles(


This is the core of the backend changes -- There are several small type changes that lead to this method filtering assets based on what's actually included in the current bundles.

Swiddis · 2024-03-11T23:11:31Z

public/components/integrations/components/setup_integration.tsx

@@ -182,6 +184,38 @@ const runQuery = async (
  }
 };

+export function SetupWorkflowSelector({


This is the main selector for the frontend, most of the other changes are wrappers around this

Swiddis · 2024-03-11T23:12:00Z

server/adaptors/integrations/__data__/repository/nginx/nginx-1.0.0.json

@@ -8,6 +8,20 @@
  "labels": ["Observability", "Logs", "Flint S3"],
  "author": "OpenSearch",
  "sourceUrl": "https://github.com/opensearch-project/dashboards-observability/tree/main/server/adaptors/integrations/__data__/repository/nginx/info",
+  "workflows": [


One design choice is that workflows should be user-defined instead of something we hardcode.

Swiddis · 2024-03-11T23:13:07Z

public/components/integrations/components/setup_integration.tsx

    const res = await addIntegrationRequest(
      false,
      integration.name,
      config.displayName,
      integration,
      setCalloutLikeToast,
      config.displayName,
-      `flint_${config.connectionDataSource}_default_${config.connectionTableName}_mview`
+      `flint_${config.connectionDataSource}_default_${config.connectionTableName}_mview`,
+      config.enabledWorkflows


Intentionally only enabling conditional workflows for S3 integrations and not all usages of addIntegrationRequest: #1519

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

ps48 · 2024-03-12T23:10:10Z

public/components/integrations/components/create_integration_helpers.ts

@@ -282,7 +282,8 @@ export async function addIntegrationRequest(
  integration: IntegrationConfig,
  setToast: (title: string, color?: Color, text?: string | undefined) => void,
  name?: string,
-  dataSource?: string
+  dataSource?: string,
+  workflows?: string[]


will these be arbitrary strings?

They're tied to the workflow names from the integration. They're used for workflow inclusion checking and as react keys, but I think they can be arbitrary strings (barring what React does when validating keys for rendering).

* Add workflows to integration format Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Render integration workflows on frontend Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add ability to toggle workflows to frontend Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add workflows to integration build options Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add asset workflow filtering to builder Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add enabled workflows to setup request Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Don't allow integration setup if no workflows enabled Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Add workflows to other integrations Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Improve header for workflows section Signed-off-by: Simeon Widdis <sawiddis@amazon.com> * Update snapshots Signed-off-by: Simeon Widdis <sawiddis@amazon.com> --------- Signed-off-by: Simeon Widdis <sawiddis@amazon.com> (cherry picked from commit 8874c8c) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Swiddis added 4 commits March 9, 2024 15:03

Add workflows to integration format

f8b1517

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Render integration workflows on frontend

8ecf62c

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add ability to toggle workflows to frontend

f20a6bf

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Merge remote-tracking branch 'upstream/main' into feature/multi-install

11ee46f

Swiddis added enhancement New feature or request backport 2.x integrations Used to denote items related to the Integrations project labels Mar 11, 2024

Swiddis added 3 commits March 11, 2024 14:42

Add workflows to integration build options

f3ffaf5

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add asset workflow filtering to builder

095d960

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add enabled workflows to setup request

4e6567e

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis mentioned this pull request Mar 11, 2024

[FEATURE] Integrations Workflows: enable workflows for index-based integrations. #1519

Open

Swiddis and others added 4 commits March 11, 2024 15:23

Don't allow integration setup if no workflows enabled

154e4ed

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Add workflows to other integrations

7a8e49f

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Merge branch 'main' into feature/multi-install

dc1a333

Improve header for workflows section

1967a20

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

Swiddis marked this pull request as ready for review March 11, 2024 23:09

Swiddis requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, rupal-bq, mengweieric, vmmusings, penghuo, seankao-az, anirudha and paulstn as code owners March 11, 2024 23:09

Swiddis requested a review from sumukhswamy as a code owner March 11, 2024 23:09

Swiddis commented Mar 11, 2024

View reviewed changes

YANG-DB approved these changes Mar 11, 2024

View reviewed changes

Update snapshots

35a3bfc

Signed-off-by: Simeon Widdis <sawiddis@amazon.com>

ps48 reviewed Mar 12, 2024

View reviewed changes

ps48 approved these changes Mar 12, 2024

View reviewed changes

Swiddis merged commit 8874c8c into opensearch-project:main Mar 13, 2024
16 of 20 checks passed

Swiddis deleted the feature/multi-install branch March 13, 2024 17:14

opensearch-trigger-bot bot mentioned this pull request Mar 13, 2024

[Backport 2.x] Add conditional installation for S3 integrations #1528

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add conditional installation for S3 integrations #1518

Add conditional installation for S3 integrations #1518

Swiddis commented Mar 11, 2024 •

edited

Loading

codecov bot commented Mar 11, 2024 •

edited

Loading

Swiddis Mar 11, 2024

Swiddis Mar 11, 2024

Swiddis Mar 11, 2024

Swiddis Mar 11, 2024 •

edited

Loading

ps48 Mar 12, 2024

Swiddis Mar 13, 2024

Add conditional installation for S3 integrations #1518

Add conditional installation for S3 integrations #1518

Conversation

Swiddis commented Mar 11, 2024 • edited Loading

Description

Issues Resolved

Check List

codecov bot commented Mar 11, 2024 • edited Loading

Codecov Report

Swiddis Mar 11, 2024

Choose a reason for hiding this comment

Swiddis Mar 11, 2024

Choose a reason for hiding this comment

Swiddis Mar 11, 2024

Choose a reason for hiding this comment

Swiddis Mar 11, 2024 • edited Loading

Choose a reason for hiding this comment

ps48 Mar 12, 2024

Choose a reason for hiding this comment

Swiddis Mar 13, 2024

Choose a reason for hiding this comment

Swiddis commented Mar 11, 2024 •

edited

Loading

codecov bot commented Mar 11, 2024 •

edited

Loading

Swiddis Mar 11, 2024 •

edited

Loading