Update import api to have data source id to allow import saved objects from uploading files to have data source #5777

yujin-emma · 2024-02-02T00:53:35Z

Description

Open Search launched Multiple Data Source at 2.4, this feature enables Dashboards to query data from multiple compatible OpenSearch endpoints. When this feature enabled, users can connect data source when adding sample data, querying data via dev tool. However, there is a limitation when user try to import saved objects by uploading a local file. First, we need to know, when multiple data source feature enabled, if you export objects, the data source information will be attached to each object, when using dashboard to visualize the object, each object already contains the data source info, therefore, the visualization tool knows where the object comes from; However, if user tried to import saved objects when they haven’t enabled multiple data source, or from another dashboards where not have the data source info, then the imported saved objects does not include data source, which can be a problem to be visualized correctly in the dashboards when enabled multiple data source.
When multiple data source feature is enabled, user can add sample data and use the dev tool with one connected data source. Import saved objects accepts uploaded files, but the API does not know the data source. These objects cannot display correctly since the it does not know where to fetch the data.
We want to provide user the ability to specify a data source when importing saved objects. When user import saved object without any data source information, the imported saved objects will attach the specified dataSourceId with the objectId; while user import saved objects different data source information ( e.g. user can connect with dataSource1 and some exported saved objects from dataSource2 ), then the objectId would already contains data source info ( dataSource2 ), for this case, we will replace the dataSourceId. The id schema are exactly same as when user add sample data and dev tools with multiple data source enabled. The behavior will keep consistent for users.

Issues Resolved

#5712

previous behavior

test1_createNewCopy.mov

resolveDataSourceConflict.mov

Screenshot

Testing the changes

Request: need to enable Multiple Data Source in the config, code pointer: https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/config/opensearch_dashboards.yml#L239
Follow the guide to start the OpenSearch-Dashboards, create data source you want to use to import saved objects
Create new data source : Dashboards Management > Data sources
Go to Dashboards Management > Saved objects > Import
At the flyout, choose one data source, you can choose different mode and upload a local file, for different mode, please refer to the video below:

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

test1_createNewCopy.mov

resolveDataSourceConflict.mov

checkConflictNoDataSourceConflict.mov

checkConflictAndAskForOverride.mov

checkConflictAndAskForOverrideWithDataSourceConflict.mov

codecov · 2024-02-02T01:08:34Z

Codecov Report

Attention: 33 lines in your changes are missing coverage. Please review.

Comparison is base (bbd40e1) 66.97% compared to head (99b6c68) 66.78%.

Files	Patch %	Lines
...erver/saved_objects/import/create_saved_objects.ts	28.00%	14 Missing and 4 partials ⚠️
src/core/server/saved_objects/routes/import.ts	0.00%	5 Missing ⚠️
...rver/saved_objects/routes/resolve_import_errors.ts	0.00%	5 Missing ⚠️
...d_objects/import/check_conflict_for_data_source.ts	85.18%	1 Missing and 3 partials ⚠️
...core/server/saved_objects/import/regenerate_ids.ts	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5777      +/-   ##
==========================================
- Coverage   66.97%   66.78%   -0.19%     
==========================================
  Files        3301     2591     -710     
  Lines       63437    48749   -14688     
  Branches    10107     8765    -1342     
==========================================
- Hits        42485    32559    -9926     
+ Misses      18499    14053    -4446     
+ Partials     2453     2137     -316

Flag	Coverage Δ
Linux_1	`?`
Linux_2	`55.11% <55.40%> (-0.01%)`	⬇️
Linux_3	`43.83% <1.35%> (-0.09%)`	⬇️
Linux_4	`35.21% <1.35%> (-0.07%)`	⬇️
Windows_1	`?`
Windows_2	`?`
Windows_3	`?`
Windows_4	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

BionIT

Left some comments and please let me know if any question

src/core/server/saved_objects/routes/resolve_import_errors.ts

src/core/server/saved_objects/saved_objects_type_registry.ts

src/core/server/saved_objects/import/regenerate_ids.ts

src/core/server/saved_objects/import/check_origin_conflicts.ts

src/core/server/saved_objects/import/check_conflict_for_data_source.ts

src/core/server/saved_objects/import/create_saved_objects.ts

src/core/server/saved_objects/import/collect_saved_objects.ts

src/core/server/saved_objects/import/check_conflicts.ts

bandinib-amzn · 2024-02-02T19:15:26Z

Can you please take a look at failed checks and try to resolve them? Also it looks like commit are signed.

src/core/server/saved_objects/import/check_conflict_for_data_source.test.ts

src/core/server/saved_objects/import/check_conflict_for_data_source.ts

src/core/server/saved_objects/import/create_saved_objects.ts

src/core/server/saved_objects/import/import_saved_objects.test.ts

bandinib-amzn · 2024-02-02T19:59:55Z

Couple of suggestions:

Utilize block comments where appropriate.
Instead of utilizing @ts-expect-error to suppress errors, consider addressing them.

src/core/server/saved_objects/import/check_conflict_for_data_source.ts

CHANGELOG.md

src/core/server/saved_objects/import/check_conflict_for_data_source.test.ts

BionIT

Added comments

src/core/server/saved_objects/routes/resolve_import_errors.ts

src/core/server/saved_objects/import/collect_saved_objects.ts

CHANGELOG.md

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

…ed new but display they are overriden Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

…hboards into dev-import-api

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

bandinib-amzn · 2024-02-06T20:34:53Z

config/opensearch_dashboards.yml

@@ -274,4 +274,4 @@
 # opensearchDashboards.survey.url: "https://survey.opensearch.org"

 # Set the value of this setting to true to enable plugin augmentation on Dashboard
-# vis_augmenter.pluginAugmentationEnabled: true
+# vis_augmenter.pluginAugmentationEnabled: true


nit: let's revert this file. It is not related to our change.

src/core/server/saved_objects/import/check_conflict_for_data_source.test.ts

src/core/server/saved_objects/import/check_conflict_for_data_source.ts

src/core/server/saved_objects/import/create_saved_objects.ts

bandinib-amzn · 2024-02-06T20:50:24Z

src/core/server/saved_objects/import/create_saved_objects.ts

+      if (object.type === 'visualization' || object.type === 'search') {
+        // @ts-expect-error
+        const searchSourceString = object.attributes?.kibanaSavedObjectMeta?.searchSourceJSON;
+        // @ts-expect-error


what error we are getting here?

src/core/server/saved_objects/import/create_saved_objects.ts

bandinib-amzn · 2024-02-06T20:55:05Z

src/core/server/saved_objects/import/import_saved_objects.test.ts

      test('with createNewCopies disabled', async () => {
-        const options = setupOptions();
+        const options = setupOptions(false, undefined);


why we are changing this? Default values are false and undefined?

bandinib-amzn · 2024-02-06T20:55:16Z

src/core/server/saved_objects/import/import_saved_objects.test.ts

@@ -410,7 +475,7 @@ describe('#importSavedObjectsFromStream', () => {

      test('with createNewCopies enabled', async () => {
        // however, we include it here for posterity
-        const options = setupOptions(true);
+        const options = setupOptions(true, undefined);


same as above

bandinib-amzn · 2024-02-06T20:55:50Z

src/core/server/saved_objects/import/import_saved_objects.test.ts

    });

    test('accumulates multiple errors', async () => {
-      const options = setupOptions();
+      const options = setupOptions(false, undefined);


don't need to change. They have setupOptions has default value.

bandinib-amzn · 2024-02-06T20:59:53Z

Although this PR has scope for improvement, functionality wise it is good to go. Let's please make sure to address all the comments and improve quality of this PR in followup PRs.

opensearch-trigger-bot · 2024-02-06T21:09:49Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-5777-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 510c7597e4efad952af8de902d1734e2ee726a9d
# Push it to GitHub
git push --set-upstream origin backport/backport-5777-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-5777-to-2.x.

yujin-emma requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, kristenTian, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, BSFishy, curq, bandinib-amzn and SuZhou-Joe as code owners February 2, 2024 00:53

github-actions bot added the first-time-contributor label Feb 2, 2024

BionIT reviewed Feb 2, 2024

View reviewed changes

src/core/server/saved_objects/import/check_conflicts.ts Outdated Show resolved Hide resolved

bandinib-amzn previously requested changes Feb 2, 2024

View reviewed changes

xinruiba reviewed Feb 2, 2024

View reviewed changes

src/core/server/saved_objects/import/check_conflict_for_data_source.ts Show resolved Hide resolved

BionIT mentioned this pull request Feb 3, 2024

[DOC] Add documentation about how to use import saved objects with enabled multiple data source feature opensearch-project/documentation-website#6345

Closed

4 tasks

Flyingliuhub reviewed Feb 3, 2024

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

Flyingliuhub reviewed Feb 3, 2024

View reviewed changes

src/core/server/saved_objects/import/check_conflict_for_data_source.test.ts Show resolved Hide resolved

seraphjiang added the v2.12.0 label Feb 5, 2024

seraphjiang assigned yujin-emma Feb 5, 2024

BionIT reviewed Feb 5, 2024

View reviewed changes

src/core/server/saved_objects/routes/resolve_import_errors.ts Show resolved Hide resolved

src/core/server/saved_objects/import/collect_saved_objects.ts Outdated Show resolved Hide resolved

CHANGELOG.md Outdated Show resolved Hide resolved

yujin-emma and others added 16 commits February 6, 2024 00:33

refactor the generate id logic

8b1a3f4

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

rename some test parameters

336b111

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

fix the bug that create new with data source conflict

8571a6f

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

fix the bug that create new with data source conflict

7a2b14a

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

fix when first check conflict and auto override, the result are creat…

b392d82

…ed new but display they are overriden Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

fix the overriden bug

9b0d78c

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

Merge branch 'opensearch-project:main' into dev-import-api

09996c3

refactor code

dccbe4c

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

Merge branch 'dev-import-api' of github.com:yujin-emma/OpenSearch-Das…

7714943

…hboards into dev-import-api

fix the overriden bug by refactoring check data source conflict

ddc0aa9

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

fix test

4826769

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

fix test

772a582

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

back to mainline yaml

08713d4

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

remove local yaml change

0abb801

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

add test

66c1901

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

remove unused var

951ce61

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

Flyingliuhub previously approved these changes Feb 6, 2024

View reviewed changes

remove unused console log

99b6c68

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>

yujin-emma dismissed Flyingliuhub’s stale review via 99b6c68 February 6, 2024 19:37

ZilongX approved these changes Feb 6, 2024

View reviewed changes

ZilongX added the backport 2.x label Feb 6, 2024

bandinib-amzn reviewed Feb 6, 2024

View reviewed changes

bandinib-amzn approved these changes Feb 6, 2024

View reviewed changes

bandinib-amzn merged commit 510c759 into opensearch-project:main Feb 6, 2024
70 of 71 checks passed

opensearch-trigger-bot bot added the failed backport label Feb 6, 2024

yujin-emma mentioned this pull request Feb 6, 2024

[Manual Backport 2.x]Update import api to have data source id to allow import saved objects from uploading files to have data source #5820

Merged

7 tasks

abbyhu2000 removed the failed backport label Feb 7, 2024

cwperks mentioned this pull request Feb 9, 2024

Add release notes for 2.12.0.0 opensearch-project/security-dashboards-plugin#1772

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update import api to have data source id to allow import saved objects from uploading files to have data source #5777

Update import api to have data source id to allow import saved objects from uploading files to have data source #5777

yujin-emma commented Feb 2, 2024 •

edited

Loading

codecov bot commented Feb 2, 2024 •

edited

Loading

BionIT left a comment

bandinib-amzn commented Feb 2, 2024

bandinib-amzn commented Feb 2, 2024

BionIT left a comment

bandinib-amzn Feb 6, 2024

bandinib-amzn Feb 6, 2024

bandinib-amzn Feb 6, 2024

bandinib-amzn Feb 6, 2024

bandinib-amzn Feb 6, 2024

bandinib-amzn commented Feb 6, 2024

opensearch-trigger-bot bot commented Feb 6, 2024

Update import api to have data source id to allow import saved objects from uploading files to have data source #5777

Update import api to have data source id to allow import saved objects from uploading files to have data source #5777

Conversation

yujin-emma commented Feb 2, 2024 • edited Loading

Description

Issues Resolved

Screenshot

Testing the changes

Check List

codecov bot commented Feb 2, 2024 • edited Loading

Codecov Report

BionIT left a comment

Choose a reason for hiding this comment

bandinib-amzn commented Feb 2, 2024

bandinib-amzn commented Feb 2, 2024

BionIT left a comment

Choose a reason for hiding this comment

bandinib-amzn Feb 6, 2024

Choose a reason for hiding this comment

bandinib-amzn Feb 6, 2024

Choose a reason for hiding this comment

bandinib-amzn Feb 6, 2024

Choose a reason for hiding this comment

bandinib-amzn Feb 6, 2024

Choose a reason for hiding this comment

bandinib-amzn Feb 6, 2024

Choose a reason for hiding this comment

bandinib-amzn commented Feb 6, 2024

opensearch-trigger-bot bot commented Feb 6, 2024

yujin-emma commented Feb 2, 2024 •

edited

Loading

codecov bot commented Feb 2, 2024 •

edited

Loading