New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up spaces tests by letting v2 migrations do less work #91829

Merged

rudolf merged 5 commits into elastic:master from rudolf:faster-v2-migrations

Feb 22, 2021

Contributor

rudolf commented Feb 18, 2021 •

edited

Loading

Summary

When I added v2 migrations to FTR (#89992) I kept all the esArchiver datasets as-is so that we're sure the behaviour is exactly the same between v1 and v2.

However, given the discussion in #91618 we are looking for ways to optimize CI runtime. One way to achieve better performance is to change esArchives so that the v2 migration algorithm has to do less work. Instead of a migration from a legacy or v1 index, the v2 migration will now see a v2 migrations index and will only update mappings and transform documents (skipping three expensive operations create index, clone, reindex).

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

rudolf added the project:ResilientSavedObjectMigrations label

rudolf added 2 commits

February 18, 2021 20:51


          Don't mess with v2 migration's index

a4a5eca


          Speed up spaces tests by letting v2 migrations do less work

e5fc7fe

rudolf force-pushed the faster-v2-migrations branch from 9fd63d9 to e5fc7fe Compare

February 18, 2021 19:52


          Add disabled mapping for spaces so that plugin can be disabled

b917aaa

Contributor Author

rudolf commented Feb 19, 2021

Brings CI down to 1h57 🎉

rudolf marked this pull request as ready for review

February 19, 2021 11:04

rudolf requested review from a team as code owners

February 19, 2021 11:04

rudolf added v7.12.0 v8.0.0 Team:Core labels

Contributor

elasticmachine commented Feb 19, 2021

Pinging @elastic/kibana-core (Team:Core)

rudolf added the release_note:skip label

jbudz approved these changes

View reviewed changes

jportner approved these changes

View reviewed changes

Contributor

jportner left a comment •

edited

Loading

This is great, what a simple change!

Brings CI down to 1h57 🎉

What's the delta? 😄
Edit: In the linked issue, Spencer said "the overall build time changed from ~1h45m to ~2h10m". So I'm assuming 1h57m is down from 2h10m. Pretty good!

QQ: I noticed that x-pack/test/saved_object_api_integration is not listed in CODEOWNERS. Do you want to add @elastic/kibana-security as the code owner for that?

rudolf added 2 commits

February 22, 2021 12:47


          Merge branch 'master' into faster-v2-migrations

51a1a72


          Add security as codeowners of xpack saved object api integration suite

9d4f61c

rudolf enabled auto-merge (squash)

February 22, 2021 11:50

Contributor

kibanamachine commented Feb 22, 2021

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💚 Build #107926 succeeded b917aaa
💔 Build #107815 failed e5fc7fe
💔 Build #107639 failed 9fd63d9a68f581980aabdd86b945a2f9d6011196

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

rudolf merged commit 42b814e into elastic:master

rudolf deleted the faster-v2-migrations branch

February 22, 2021 13:47

kibanamachine added the backport missing label

Contributor

kibanamachine commented Feb 24, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

4 similar comments

Contributor

kibanamachine commented Feb 25, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

Contributor

kibanamachine commented Feb 26, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

Contributor

kibanamachine commented Mar 1, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

Contributor

kibanamachine commented Mar 2, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

Contributor

kibanamachine commented Mar 3, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

5 similar comments

Contributor

kibanamachine commented Mar 4, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

Contributor

kibanamachine commented Mar 5, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

Contributor

kibanamachine commented Mar 8, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

Contributor

kibanamachine commented Mar 9, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

Contributor

kibanamachine commented Mar 10, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

rudolf mentioned this pull request

EsArchiver: support for injecting kibana version and use it for spaces tests #94420

Merged

9 tasks

Contributor

kibanamachine commented Mar 11, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

2 similar comments

Contributor

kibanamachine commented Mar 12, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

Contributor

kibanamachine commented Mar 15, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

Contributor

spalger commented Mar 15, 2021

@rudolf I'm sorry, I know we've talked about this but what's the status of this backport?

Contributor Author

rudolf commented Mar 15, 2021

I was blocked on #94420 but can now backport this to 7.x

Contributor

kibanamachine commented Mar 16, 2021

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 91829 or prevent reminders by adding the backport:skip label.

rudolf mentioned this pull request

[7.x] Speed up spaces tests by letting v2 migrations do less work (#91829) #94797

Merged

rudolf added a commit to rudolf/kibana that referenced this pull request


          Speed up spaces tests by letting v2 migrations do less work (elastic#…

62c3c2f

…91829)

* Don't mess with v2 migration's index

* Speed up spaces tests by letting v2 migrations do less work

* Add disabled mapping for spaces so that plugin can be disabled

* Add security as codeowners of xpack saved object api integration suite
# Conflicts:
#	.github/CODEOWNERS
#	packages/kbn-es-archiver/src/lib/indices/kibana_index.ts
#	x-pack/test/functional/es_archives/saved_objects_management/spaces_integration/mappings.json
#	x-pack/test/saved_object_api_integration/common/fixtures/es_archiver/saved_objects/spaces/mappings.json

rudolf mentioned this pull request

[7.12] Speed up spaces tests by letting v2 migrations do less work (#91829) #94801

Merged

rudolf added a commit to rudolf/kibana that referenced this pull request


          Speed up spaces tests by letting v2 migrations do less work (elastic#…

9d80ad4

…91829)

* Don't mess with v2 migration's index

* Speed up spaces tests by letting v2 migrations do less work

* Add disabled mapping for spaces so that plugin can be disabled

* Add security as codeowners of xpack saved object api integration suite
# Conflicts:
#	.github/CODEOWNERS
#	x-pack/test/functional/es_archives/saved_objects_management/spaces_integration/mappings.json
#	x-pack/test/saved_object_api_integration/common/fixtures/es_archiver/saved_objects/spaces/mappings.json

Contributor

kibanamachine commented Mar 17, 2021

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

1 similar comment

Contributor

kibanamachine commented Mar 18, 2021

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

rudolf added a commit that referenced this pull request


          [7.x] Speed up spaces tests by letting v2 migrations do less work (#9…

70446af

…1829) (#94797)

* Speed up spaces tests by letting v2 migrations do less work (#91829)

* Don't mess with v2 migration's index

* Speed up spaces tests by letting v2 migrations do less work

* Add disabled mapping for spaces so that plugin can be disabled

* Add security as codeowners of xpack saved object api integration suite
# Conflicts:
#	.github/CODEOWNERS
#	packages/kbn-es-archiver/src/lib/indices/kibana_index.ts
#	x-pack/test/functional/es_archives/saved_objects_management/spaces_integration/mappings.json
#	x-pack/test/saved_object_api_integration/common/fixtures/es_archiver/saved_objects/spaces/mappings.json

* Apply suggestions from code review

* Fix backport for ES _types

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

Contributor

kibanamachine commented Mar 19, 2021

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.

kibanamachine removed the backport missing label

rudolf added a commit that referenced this pull request


          [7.12] Speed up spaces tests by letting v2 migrations do less work (#…

92e7cf6

…91829) (#94801)

* Speed up spaces tests by letting v2 migrations do less work (#91829)

* Don't mess with v2 migration's index

* Speed up spaces tests by letting v2 migrations do less work

* Add disabled mapping for spaces so that plugin can be disabled

* Add security as codeowners of xpack saved object api integration suite
# Conflicts:
#	.github/CODEOWNERS
#	x-pack/test/functional/es_archives/saved_objects_management/spaces_integration/mappings.json
#	x-pack/test/saved_object_api_integration/common/fixtures/es_archiver/saved_objects/spaces/mappings.json

* Fix backport for ES _types

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

jbudz mentioned this pull request

v2 migrations adds significant CI overhead #91618

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

project:ResilientSavedObjectMigrations release_note:skip Team:Core v7.12.0 v8.0.0