Migrate old style queries stored in filters array #38945

stacey-gammon · 2019-06-13T22:36:43Z

Summary

Prior to version 6.0.0, advanced lucene query settings were stored on the dashboard object. This could cause you to see certain settings applied that were not actually a part of your current lucene query settings.

This fixes that behavior by migrating all older style queries into the newer format, and removes those advanced settings from the dashboard object. This behavior is considered a bug, but note that if you have old dashboards that have never been edited since 6.0.0, you may start to see different query settings applied. The settings will now match whatever you have set up in your advanced settings configuration and not what previously happened to be stored on the dashboard.

Fixes #38933.

Details

Migrates dashboards that used to have queries stored in the filter array on search source and moves it to the query part of search source.

5.x url test added, though in code migration shouldn't be needed for bwc support of URLs because in the url it's already in a different format.

Also fixes #38933.

More migrations will likely follow to further simplify some of the dashboard code.

elasticmachine · 2019-06-13T22:36:45Z

Pinging @elastic/kibana-app-arch

src/legacy/core_plugins/kibana/migrations/is_doc.ts

elasticmachine · 2019-06-13T23:07:39Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-06-14T01:37:58Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-06-14T12:19:35Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-06-14T14:19:00Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-06-14T19:43:45Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-06-14T20:56:36Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-06-17T13:50:00Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-06-18T12:50:27Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

lukasolson

This LGTM, but just wanted to throw out the question, do we want to do the same migrations for visualizations/saved searches? Seems odd to have dashboards migrated but not the others.

stacey-gammon · 2019-06-18T19:01:17Z

do we want to do the same migrations for visualizations/saved searches?

That sounds like a good idea to me. @timroes - mind if I leave that as a follow up task for the Kibana App team?

elasticmachine · 2019-06-18T19:39:24Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

Bargs · 2019-06-18T20:06:25Z

@lukasolson @stacey-gammon I don't think this migration is required for saved searches and visualizations. afaik dashboard was the only app to store queries in the filters array like this.

elasticmachine · 2019-06-18T20:15:00Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

Bargs

Had a couple questions but no blocking issues. Tested the functionality with an export from 5.6 and it worked.

Bargs · 2019-06-18T18:40:21Z

src/legacy/core_plugins/kibana/migrations/types.ts

+
+import { SavedObjectReference } from '../../../../legacy/server/saved_objects/routes/types';
+
+export interface SavedObjectAttributes {


Could these not just be the existing saved object types? (Same question for the other types.ts file under the dashboard directory)

kibana/src/core/server/saved_objects/service/saved_objects_client.ts

Lines 145 to 169 in 260d907

/**

*

* @public

*/

export interface SavedObjectAttributes {

[key: string]: SavedObjectAttributes | string | number | boolean | null;

}

/**

*

* @public

*/

export interface SavedObject<T extends SavedObjectAttributes = any> {

id: string;

type: string;

version?: string;

updated_at?: string;

error?: {

message: string;

statusCode: number;

};

attributes: T;

references: SavedObjectReference[];

migrationVersion?: SavedObjectsMigrationVersion;

}

I agree with you these types could all be refactored/improved. It's just not super straightforward. I started going down many typescript rabbit holes with this endeavor (started trying to type all of migrations.js), but they quickly grew out of scope.

For instance, we have interface SavedObjectDashboard extends SavedObject. That's actually not the SavedObject here though, that's actually the SavedObjectAttributes (but also with id... so maybe it's actually the saved object after a post processing step). This extends SavedObject type is also different than the raw SavedObject.

Another type that gets in the mix is the one before and after the extractReferences call. It actually changes the raw saved object type by injecting the panel ids into the panelsJSON and removing the outer references.

So I think I'd prefer for this PR and just focus on the migration plus minimal typing support. Ideally we should typescript all of migrations and start a pattern of adding types for each iteration of saved object.

src/legacy/core_plugins/kibana/public/dashboard/migrations/move_filters_to_query.ts

lukasolson · 2019-06-18T21:23:58Z

@Bargs The advanced settings are still stored as part of queries in 5.x, though, which I thought was the intention of this migration (to remove those).

…grate_search_source_filter_queries

elasticmachine · 2019-06-19T13:46:26Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

elasticmachine · 2019-06-19T15:07:01Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

Bargs · 2019-06-19T15:42:31Z

@lukasolson ah yeah I suppose so, I thought the main point of this PR was to get the query out of the filters array, and removing the advanced settings was more of a side effect.

stacey-gammon · 2019-06-19T15:51:38Z

The main point of the PR was actually to clean up dashboard logic :D both removing the query from the filters array and removing the advanced settings were side affects of that. :-P

* Migrate old query filters * Null check instead of undefined for more completeness * remove unnecessary undefined check * Use good defaults, not undefined, for brand new dashboards. * fix: typescript errors * be explicit instead of matchinline snapshot. * default to Kuery when there is no query given

stacey-gammon added the :AppArch label Jun 13, 2019

stacey-gammon changed the title ~~Migrate old style queries stored in filters array and remove now unne…~~ Migrate old style queries stored in filters array Jun 13, 2019

stacey-gammon force-pushed the 2019-06-13-migrate_search_source_filter_queries branch from 0e3063c to 94f9793 Compare June 13, 2019 22:44

streamich reviewed Jun 13, 2019

View reviewed changes

src/legacy/core_plugins/kibana/migrations/is_doc.ts Show resolved Hide resolved

stacey-gammon force-pushed the 2019-06-13-migrate_search_source_filter_queries branch 2 times, most recently from 6ef31eb to 9a00282 Compare June 14, 2019 00:20

stacey-gammon marked this pull request as ready for review June 14, 2019 12:21

stacey-gammon added release_note:skip Skip the PR/issue when compiling release notes Team:Visualizations Visualization editors, elastic-charts and infrastructure v7.3.0 v8.0.0 labels Jun 14, 2019

stacey-gammon requested review from Bargs and lukasolson June 14, 2019 13:11

stacey-gammon added release_note:fix and removed release_note:skip Skip the PR/issue when compiling release notes labels Jun 14, 2019

stacey-gammon force-pushed the 2019-06-13-migrate_search_source_filter_queries branch from cd349d4 to d873a9c Compare June 14, 2019 13:19

stacey-gammon added the review label Jun 14, 2019

stacey-gammon force-pushed the 2019-06-13-migrate_search_source_filter_queries branch from d873a9c to 552b610 Compare June 14, 2019 19:16

stacey-gammon force-pushed the 2019-06-13-migrate_search_source_filter_queries branch from d81c811 to 575ffe2 Compare June 17, 2019 12:53

stacey-gammon force-pushed the 2019-06-13-migrate_search_source_filter_queries branch 2 times, most recently from 7009f8b to e7d5215 Compare June 18, 2019 12:42

stacey-gammon added 5 commits June 18, 2019 14:38

Migrate old query filters

f586cfa

Null check instead of undefined for more completeness

c592e92

remove unnecessary undefined check

0f51773

Use good defaults, not undefined, for brand new dashboards.

afccf88

fix: typescript errors

3b40c2a

stacey-gammon force-pushed the 2019-06-13-migrate_search_source_filter_queries branch from 0b3a55a to 3b40c2a Compare June 18, 2019 18:38

lukasolson reviewed Jun 18, 2019

View reviewed changes

be explicit instead of matchinline snapshot.

6f3ab7e

stacey-gammon mentioned this pull request Jun 18, 2019

Flaky test: Code Manage Repositories Manage Repositories import a git:// repository #39203

Closed

Bargs approved these changes Jun 18, 2019

View reviewed changes

Merge branch 'master' of github.com:elastic/kibana into 2019-06-13-mi…

97575f4

…grate_search_source_filter_queries

default to Kuery when there is no query given

147addf

stacey-gammon merged commit 184111d into elastic:master Jun 19, 2019

stacey-gammon added the backport pending label Jun 19, 2019

stacey-gammon mentioned this pull request Jun 19, 2019

[7.x] Migrate old style queries stored in filters array (#38945) #39293

Merged

stacey-gammon added backported and removed backport pending labels Jun 19, 2019

tylersmalley mentioned this pull request Dec 10, 2019

Moving saved object management to an ES plugin #49764

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate old style queries stored in filters array #38945

Migrate old style queries stored in filters array #38945

stacey-gammon commented Jun 13, 2019 •

edited

Loading

elasticmachine commented Jun 13, 2019

elasticmachine commented Jun 13, 2019

elasticmachine commented Jun 14, 2019

elasticmachine commented Jun 14, 2019

elasticmachine commented Jun 14, 2019

elasticmachine commented Jun 14, 2019

elasticmachine commented Jun 14, 2019

elasticmachine commented Jun 17, 2019

elasticmachine commented Jun 18, 2019

lukasolson left a comment

stacey-gammon commented Jun 18, 2019

elasticmachine commented Jun 18, 2019

Bargs commented Jun 18, 2019 •

edited

Loading

elasticmachine commented Jun 18, 2019

Bargs left a comment

Bargs Jun 18, 2019

stacey-gammon Jun 19, 2019

lukasolson commented Jun 18, 2019

elasticmachine commented Jun 19, 2019

elasticmachine commented Jun 19, 2019

Bargs commented Jun 19, 2019

stacey-gammon commented Jun 19, 2019


		import { SavedObjectReference } from '../../../../legacy/server/saved_objects/routes/types';

		export interface SavedObjectAttributes {

	/**
	*
	* @public
	*/
	export interface SavedObjectAttributes {
	[key: string]: SavedObjectAttributes \| string \| number \| boolean \| null;
	}

	/**
	*
	* @public
	*/
	export interface SavedObject<T extends SavedObjectAttributes = any> {
	id: string;
	type: string;
	version?: string;
	updated_at?: string;
	error?: {
	message: string;
	statusCode: number;
	};
	attributes: T;
	references: SavedObjectReference[];
	migrationVersion?: SavedObjectsMigrationVersion;
	}

Migrate old style queries stored in filters array #38945

Migrate old style queries stored in filters array #38945

Conversation

stacey-gammon commented Jun 13, 2019 • edited Loading

Summary

Details

elasticmachine commented Jun 13, 2019

elasticmachine commented Jun 13, 2019

💔 Build Failed

elasticmachine commented Jun 14, 2019

💔 Build Failed

elasticmachine commented Jun 14, 2019

💚 Build Succeeded

elasticmachine commented Jun 14, 2019

💚 Build Succeeded

elasticmachine commented Jun 14, 2019

💔 Build Failed

elasticmachine commented Jun 14, 2019

💚 Build Succeeded

elasticmachine commented Jun 17, 2019

💚 Build Succeeded

elasticmachine commented Jun 18, 2019

💔 Build Failed

lukasolson left a comment

Choose a reason for hiding this comment

stacey-gammon commented Jun 18, 2019

elasticmachine commented Jun 18, 2019

💔 Build Failed

Bargs commented Jun 18, 2019 • edited Loading

elasticmachine commented Jun 18, 2019

💔 Build Failed

Bargs left a comment

Choose a reason for hiding this comment

Bargs Jun 18, 2019

Choose a reason for hiding this comment

stacey-gammon Jun 19, 2019

Choose a reason for hiding this comment

lukasolson commented Jun 18, 2019

elasticmachine commented Jun 19, 2019

💚 Build Succeeded

elasticmachine commented Jun 19, 2019

💚 Build Succeeded

Bargs commented Jun 19, 2019

stacey-gammon commented Jun 19, 2019

stacey-gammon commented Jun 13, 2019 •

edited

Loading

Bargs commented Jun 18, 2019 •

edited

Loading