Add datasource picker to import saved object flyout when multiple data source is enabled #5781

BionIT · 2024-02-02T04:31:32Z

Description

When multiple data source plugin is enabled, user can connect to remote data sources in addition to the local cluster, to support importing saved objects from remote data source via file upload(#5712), we need to have an option in the flyout for user to select where the file comes from.

Via discussion in the issue, we are altering the order of the import option, and have the Create new objects with random IDs as the default choice. When multiple data source is enabled, we will have the data source picker under Import options section then the Conflict management section will contain the options to address conflicts.

Issues Resolved

Fixes #5712

Screenshot

5717.mp4

Testing the changes

In the video above, we tested the following scenarios
when multiple data source is disabled:

upload same ndjson with Create new objects with unique IDs option is successful when there are duplicate objects exists
upload same ndjson with Automatically overwrite conflicts option is successful when there are duplicate objects
upload same ndjson with Request action on conflict option will prompt user to handle on the conflict
upload same json with Automatically overwrite conflicts option twice is successful when there are duplicate objects exists
upload same json with Request action on conflict option will prompt user to handle on the conflict
when multiple data source is enabled:
upload ndjson with Create new objects with unique IDs option is successful
upload ndjson with Automatically overwrite conflicts option is successful when there are duplicate objects
upload same ndjson with Request action on conflict option will prompt user to handle on the conflict
upload same json with Automatically overwrite conflicts option twice is successful when there are duplicate objects exists
upload same json with Request action on conflict option will prompt user to handle on the conflict

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

…ata source is enabled Signed-off-by: Lu Yu <nluyu@amazon.com>

Signed-off-by: Lu Yu <nluyu@amazon.com>

codecov · 2024-02-02T04:45:04Z

Codecov Report

Attention: 9 lines in your changes are missing coverage. Please review.

Comparison is base (7c0bd9f) 67.01% compared to head (d55ed3e) 66.97%.

Files	Patch %	Lines
...gement_section/objects_table/components/flyout.tsx	75.00%	3 Missing ⚠️
...c/components/cluster_selector/cluster_selector.tsx	60.00%	2 Missing ⚠️
...saved_objects_management/public/lib/import_file.ts	0.00%	2 Missing ⚠️
...cts_management/public/lib/resolve_import_errors.ts	33.33%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5781      +/-   ##
==========================================
- Coverage   67.01%   66.97%   -0.05%     
==========================================
  Files        3296     3301       +5     
  Lines       63370    63434      +64     
  Branches    10093    10104      +11     
==========================================
+ Hits        42465    42482      +17     
- Misses      18456    18501      +45     
- Partials     2449     2451       +2

Flag	Coverage Δ
Linux_1	`35.23% <0.00%> (-0.01%)`	⬇️
Linux_2	`55.12% <ø> (ø)`
Linux_3	`43.93% <60.00%> (-0.01%)`	⬇️
Linux_4	`35.28% <60.71%> (-0.06%)`	⬇️
Windows_1	`35.25% <0.00%> (-0.01%)`	⬇️
Windows_2	`55.09% <ø> (ø)`
Windows_3	`43.93% <60.00%> (-0.01%)`	⬇️
Windows_4	`35.28% <60.71%> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ashwin-pc

We already have a datasource selector component in the data plugin. We should use that and not create a new picker.

bandinib-amzn · 2024-02-02T20:22:01Z

I'm coming here after watching video attached in screenshot. Would it be possible to consider adding a checkbox or an alternative option for selecting 'overwrite all,' instead of having to click 'overwrite' 15 times? This could potentially enhance the user experience. cc: @kgcreative

...ugins/saved_objects_management/public/management_section/objects_table/components/flyout.tsx

bandinib-amzn · 2024-02-02T21:33:44Z

We already have a datasource selector component in the data plugin. We should use that and not create a new picker.

Hi @ashwin-pc
I guess you are talking about this PR - #5167 and wondering why we don't use the same selector.

Let's consider the use case from above PR which is Datasource selector support in OpenSearch Dashboard Discover.
The Discover feature in OpenSearch Dashboards is designed for searching, exploring, and visualizing data stored in OpenSearch indices or Remote cluster indices.

The Dev Tool feature in OpenSearch Dashboard provides an environment for developers and administrators to interact with OpenSearch or Remote cluster with query DSL and API directly.

I took discover and dev tool here as an example to show the difference between two pickers here. They serve different purposes. #5167 has picker for selecting data source for searching, exploring data in the selected data source. Whereas data source picker in this PR is listing the data sources (basically remote cluster) to interact with them directly for fetch sample data, fetch indices to create index patterns etc.

IMO, they both serves different purpose, so I would prefer separate picker. @kgcreative I would like to here your thoughts as well.

BionIT · 2024-02-02T22:00:39Z

I'm coming here after watching video attached in screenshot. Would it be possible to consider adding a checkbox or an alternative option for selecting 'overwrite all,' instead of having to click 'overwrite' 15 times? This could potentially enhance the user experience. cc: @kgcreative

@bandinib-amzn The reason user has to act on the action multiple times is because Request action on conflict is selected. If user selects Automatically overwrite conflicts then all the conflict objects will be overwritten without any additional actions. This is the current behavior of the import function

kgcreative · 2024-02-02T22:43:22Z

I'm coming here after watching video attached in screenshot. Would it be possible to consider adding a checkbox or an alternative option for selecting 'overwrite all,' instead of having to click 'overwrite' 15 times? This could potentially enhance the user experience. cc: @kgcreative

Yes, this is PAINFUL today when there are many objects that have conflicts, however, i'm hesitant to add to the scope unless the dev effort is reasonable. Ideally, we should show a selectable list, with easy "Select all/Deselect all/multiselect" options. Presumably each modal in the current experience triggers only when there is a conflict, so this would require a re-think of the flow.

BionIT · 2024-02-03T00:09:18Z

We already have a datasource selector component in the data plugin. We should use that and not create a new picker.

As discussed with @ashwin-pc and @mengweieric , we will switch to use the data source selector in 2.13 when it supports the feature requested in #5790

ashwin-pc · 2024-02-03T00:12:39Z

IMO, they both serves different purpose, so I would prefer separate picker. @kgcreative I would like to here your thoughts as well.

Thanks for that example. That was a very useful distinction! Then i think my concern is around the confusion in naming. We are calling this and the other component both datasource pickers but have very different uses for each. This one is used specifically to select an OpenSearch cluster while the other is to select what is in thes clusters but also data from other sources. Can we rename this to be a cluster selector instead to disambiguate the two?

Clarified the distinction

BionIT · 2024-02-03T00:18:13Z

IMO, they both serves different purpose, so I would prefer separate picker. @kgcreative I would like to here your thoughts as well.

Thanks for that example. That was a very useful distinction! Then i think my concern is around the confusion in naming. We are calling this and the other component both datasource pickers but have very different uses for each. This one is used specifically to select an OpenSearch cluster while the other is to select what is in thes clusters but also data from other sources. Can we rename this to be a cluster selector instead to disambiguate the two?

@ashwin-pc I hear you, I don't have a strong opinion, since I will change to use the new data source selector in 2.13, should we go with this name for now? Happy to change it if we feel necessary

ashwin-pc

Can we also add some tests for the code added here?

ashwin-pc · 2024-02-03T00:20:51Z

test/functional/page_objects/management/saved_objects_page.ts

@@ -50,15 +50,22 @@ export function SavedObjectsPageProvider({ getService, getPageObjects }: FtrProv
 await this.waitTableIsLoaded();
 }

- async importFile(path: string, overwriteAll = true) {
+ async importFile(path: string, overwriteAll = true, isLegacy = false) {


nit: if we default this to true, you can avoid changes in the test files.

ashwin-pc · 2024-02-03T00:29:43Z

@ashwin-pc I hear you, I don't have a strong opinion, since I will change to use the new data source selector in 2.13, should we go with this name for now? Happy to change it if we feel necessary

based on @bandinib-amzn's reply i'm actually convinced that the two might not need to be the same thing if the only thing you are selecting is the cluster and not the index/indexPattern. But if thats true then we do need to rename this since the two wont be combined in 2.13 or beyond

bandinib-amzn · 2024-02-03T01:25:16Z

e import function

Yes, I completely agree. It doesn't need to be within the scope of this PR. It's an enhancement that we can address in the future. I have created an issue #5794 to track.

...ugins/saved_objects_management/public/management_section/objects_table/components/flyout.tsx

Signed-off-by: Lu Yu <nluyu@amazon.com>

BionIT · 2024-02-05T02:37:19Z

Can we also add some tests for the code added here?

Added tests in new commit

…a source is enabled (#5781) * add datasource picker to saved object management page when multiple data source is enabled Signed-off-by: Lu Yu <nluyu@amazon.com> * add changelog Signed-off-by: Lu Yu <nluyu@amazon.com> * change name to cluster selector and move to higher level Signed-off-by: Lu Yu <nluyu@amazon.com> --------- Signed-off-by: Lu Yu <nluyu@amazon.com> (cherry picked from commit 8e93c54) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md

…a source is enabled (opensearch-project#5781) * add datasource picker to saved object management page when multiple data source is enabled Signed-off-by: Lu Yu <nluyu@amazon.com> * add changelog Signed-off-by: Lu Yu <nluyu@amazon.com> * change name to cluster selector and move to higher level Signed-off-by: Lu Yu <nluyu@amazon.com> --------- Signed-off-by: Lu Yu <nluyu@amazon.com> Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

…a source is enabled (#5781) * add datasource picker to saved object management page when multiple data source is enabled Signed-off-by: Lu Yu <nluyu@amazon.com> * add changelog Signed-off-by: Lu Yu <nluyu@amazon.com> * change name to cluster selector and move to higher level Signed-off-by: Lu Yu <nluyu@amazon.com> --------- Signed-off-by: Lu Yu <nluyu@amazon.com> (cherry picked from commit 8e93c54) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md

…a source is enabled (#5781) (#5807) * add datasource picker to saved object management page when multiple data source is enabled Signed-off-by: Lu Yu <nluyu@amazon.com> * add changelog Signed-off-by: Lu Yu <nluyu@amazon.com> * change name to cluster selector and move to higher level Signed-off-by: Lu Yu <nluyu@amazon.com> --------- Signed-off-by: Lu Yu <nluyu@amazon.com> (cherry picked from commit 8e93c54) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

add datasource picker to saved object management page when multiple d…

bfb3411

…ata source is enabled Signed-off-by: Lu Yu <nluyu@amazon.com>

BionIT requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, kristenTian, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, BSFishy, curq, bandinib-amzn and SuZhou-Joe as code owners February 2, 2024 04:31

github-actions bot added the first-time-contributor label Feb 2, 2024

add changelog

3db3d74

Signed-off-by: Lu Yu <nluyu@amazon.com>

ashwin-pc previously requested changes Feb 2, 2024

View reviewed changes

bandinib-amzn reviewed Feb 2, 2024

View reviewed changes

BionIT closed this Feb 2, 2024

BionIT reopened this Feb 2, 2024

BionIT requested a review from ashwin-pc February 3, 2024 00:10

ashwin-pc reviewed Feb 3, 2024

View reviewed changes

BionIT mentioned this pull request Feb 3, 2024

[DOC] Add documentation about how to use import saved objects with enabled multiple data source feature opensearch-project/documentation-website#6345

Closed

4 tasks

Flyingliuhub reviewed Feb 3, 2024

View reviewed changes

...ugins/saved_objects_management/public/management_section/objects_table/components/flyout.tsx Show resolved Hide resolved

BionIT added 2 commits February 5, 2024 01:48

change name to cluster selector and move to higher level

8d39649

Signed-off-by: Lu Yu <nluyu@amazon.com>

Merge branch 'main' into issue-5712

d55ed3e

Flyingliuhub approved these changes Feb 5, 2024

View reviewed changes

ZilongX approved these changes Feb 5, 2024

View reviewed changes

Flyingliuhub added v2.12.0 multiple datasource multiple datasource project backport 2.x and removed first-time-contributor labels Feb 5, 2024

bandinib-amzn approved these changes Feb 5, 2024

View reviewed changes

ZilongX merged commit 8e93c54 into opensearch-project:main Feb 5, 2024
81 checks passed

opensearch-trigger-bot bot mentioned this pull request Feb 5, 2024

[Backport 2.x] Add datasource picker to import saved object flyout when multiple data source is enabled #5807

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add datasource picker to import saved object flyout when multiple data source is enabled #5781

Add datasource picker to import saved object flyout when multiple data source is enabled #5781

BionIT commented Feb 2, 2024 •

edited

Loading

codecov bot commented Feb 2, 2024 •

edited

Loading

ashwin-pc left a comment

bandinib-amzn commented Feb 2, 2024 •

edited

Loading

bandinib-amzn commented Feb 2, 2024

BionIT commented Feb 2, 2024

kgcreative commented Feb 2, 2024 •

edited

Loading

BionIT commented Feb 3, 2024

ashwin-pc commented Feb 3, 2024

BionIT commented Feb 3, 2024

ashwin-pc left a comment

ashwin-pc Feb 3, 2024

ashwin-pc commented Feb 3, 2024 •

edited

Loading

bandinib-amzn commented Feb 3, 2024 •

edited

Loading

BionIT commented Feb 5, 2024

Add datasource picker to import saved object flyout when multiple data source is enabled #5781

Add datasource picker to import saved object flyout when multiple data source is enabled #5781

Conversation

BionIT commented Feb 2, 2024 • edited Loading

Description

Issues Resolved

Screenshot

Testing the changes

Check List

codecov bot commented Feb 2, 2024 • edited Loading

Codecov Report

ashwin-pc left a comment

Choose a reason for hiding this comment

bandinib-amzn commented Feb 2, 2024 • edited Loading

bandinib-amzn commented Feb 2, 2024

BionIT commented Feb 2, 2024

kgcreative commented Feb 2, 2024 • edited Loading

BionIT commented Feb 3, 2024

ashwin-pc commented Feb 3, 2024

BionIT commented Feb 3, 2024

ashwin-pc left a comment

Choose a reason for hiding this comment

ashwin-pc Feb 3, 2024

Choose a reason for hiding this comment

ashwin-pc commented Feb 3, 2024 • edited Loading

bandinib-amzn commented Feb 3, 2024 • edited Loading

BionIT commented Feb 5, 2024

BionIT commented Feb 2, 2024 •

edited

Loading

codecov bot commented Feb 2, 2024 •

edited

Loading

bandinib-amzn commented Feb 2, 2024 •

edited

Loading

kgcreative commented Feb 2, 2024 •

edited

Loading

ashwin-pc commented Feb 3, 2024 •

edited

Loading

bandinib-amzn commented Feb 3, 2024 •

edited

Loading