Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Saved Objects] Compatible mappings PR check #148656

Merged
merged 72 commits into from
Apr 27, 2023
Merged

[Saved Objects] Compatible mappings PR check #148656

merged 72 commits into from
Apr 27, 2023

Conversation

jloleysens
Copy link
Contributor

@jloleysens jloleysens commented Jan 10, 2023 •
edited
Loading

Summary

This PR adds a technical control to prevent incompatible mappings changes. These include:

  1. Removing mapped fields. For the foreseeable future we require that teams only introduce new fields - in short: this avoids the "reindex" step in our migrations.
  2. Changing the type of a field. We leverage ES to determine whether a given set of mappings can be applied "on top" of another. Similarly, this avoids the "reindex" step in migrations.

The above checks depend on a snapshot of the mappings from main, these are the "current" mappings and are extracted from plugin code. This PR will bootstrap main with an initial set of mappings extracted from plugins (bulk of new lines added).

The new CLI

See the added README.md for details on how the CLI works.

How will it work?

Any new PR that introduces compatible mappings changes will result in a new snapshot being captured, then merged to main for other PRs to merge and run the same checks against (currently committing new snapshots happens in the CI check so there is no manual step of maintaining the snapshot).

Additional

We should consider combining this CI check with the existing check in src/core/server/integration_tests/saved_objects/migrations/check_registered_types.test.ts. Hopefully we can automate the check such that no manual review is needed from Core, not sure how we might cover the hash of the non-mappings related fields. We could consider narrowing the Jest test to exclude mappings.

Checklist

@jloleysens jloleysens added Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc Feature:Saved Objects release_note:skip Skip the PR/issue when compiling release notes auto-backport Deprecated - use backport:version if exact versions are needed Feature:Migrations v8.7.0 labels Jan 10, 2023
kibanamachine and others added 13 commits January 10, 2023 16:54
@jloleysens jloleysens changed the title [Saved Objects] Compatible mappings check [Saved Objects] Compatible mappings PR check Jan 11, 2023
jloleysens and others added 6 commits February 10, 2023 11:10
@jloleysens
Merge branch 'main' into compatible-schema-change-check
a9b2e06 
* main: (115 commits)
  [Custom branding] Add custom logo to space selector (elastic#150284)
  [api-docs] 2023-02-10 Daily api_docs build (elastic#150831)
  [ci] build next docs in PRs when relevant files change (elastic#149991)
  [codeowners] allow overrides to take higher precedence (elastic#150821)
  [docs] Remove kibDevDocsOpsPluginDiscovery (elastic#150788)
  [Fleet] Fix max 20 installed integrations returned from Fleet API (elastic#150780)
  [maps] fix Changing resolutions on Heat map layer throws error in console (elastic#150761)
  fixes Failing ES Promotion: X-Pack API Integration Tests x-pack/test/api_integration/apis/maps/get_grid_tile.js (elastic#150768)
  [Synthetics] adjust overview scrolling e2e (elastic#150774)
  [Security Solution] Fixes bulk close alerts from exception flyout type bug (elastic#150765)
  Upgrade EUI to v74.1.0 (elastic#150235)
  [skip ci] Fix labeling for Infrastructure UI (elastic#150571)
  [Enterprise Search] Move pipelines modal to flyout (elastic#150727)
  [Security Solution] fix flaky endpoint tests (elastic#150652)
  Fixes the space selector page layout  (elastic#150503)
  [Dashboard] [Navigation] Fix mount point bug (elastic#150507)
  [Infrastructure UI] Track host cloud provider on table entry click (elastic#150685)
  [Dashboard Usability] Moves scrollbar to panel section (elastic#145628)
  [Maps] fixes Kibana maps shows MVT borders if the geometry border style is greater than 1 (elastic#150497)
  [Cloud Posture][Dashboard] dashboard re-design enhancements (elastic#150394)
  ...
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/generate codeowners'
1aeaa94
@jloleysens
Merge branch 'main' into compatible-schema-change-check
6e9640c
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/generate codeowners'
0aa561c
@jloleysens
Merge branch 'main' into compatible-schema-change-check
12b922b
@jloleysens
Merge branch 'main' into compatible-schema-change-check
6639be6
@cla-checker-service
Copy link

cla-checker-service bot commented Feb 22, 2023 •
edited
Loading

💚 CLA has been signed

@pgayvallet
Copy link
Contributor

@jloleysens you may need to rebase the PR to remove @spalger's commit and re-add it from your account (or ask him to sign the contributors agreement 😅 )

@spalger
Copy link
Contributor

spalger commented Feb 27, 2023

@jloleysens signed 🙃

@spalger
Copy link
Contributor

spalger commented Feb 27, 2023

Looks like I can't trigger a CLA check anymore :) but if someone in the org comments cla/check the bot should revalidate

@rudolf
Copy link
Contributor

rudolf commented Mar 2, 2023

cla/check

jloleysens and others added 5 commits March 7, 2023 11:35
@jloleysens
Merge branch 'main' into compatible-schema-change-check
481ac10
@jloleysens
Merge branch 'main' into compatible-schema-change-check
ab5ff42 
* main: (1294 commits)
  [SecuritySolution] Refactor security packages (elastic#155365)
  [Discover] Show "Temporary" badge for ad-hoc data views in Alerts flyout (elastic#155717)
  [RAM] Conditional actions feedback on pr review (elastic#155804)
  [Files] Adds bulk delete method (elastic#155628)
  [Lens] Use proper way to generate absolute short URL (elastic#155512)
  [Guided onboarding] Use Kibana features to grant access (elastic#155065)
  [Index Management] Fix duped mock (elastic#155844)
  [Lens] Enhance visualization modifier popup with layer palette (elastic#155280)
  Fix flaky combobox tests on role management screen (elastic#155711)
  [Infrastructure UI] Create InventoryViewsService and InventoryViewsClient (elastic#155126)
  [Fleet] always create agent upload write indices (elastic#155729)
  [Fleet] [Cloud Security Posture] Add CloudFormation agent install method (elastic#155045)
  Add tech preview label for search applications (elastic#155649)
  [ML] AIOps: Stabilize flaky functional tests. (elastic#155710)
  [ES UI Shared] Migrate JsonEditor to monaco (elastic#155610)
  [Security Solution] Fixes security_solution storybooks always rendering in a flyout (elastic#155814)
  [Synthetics] Make error popover disappear `onMouseLeave` of metric item card (elastic#155800)
  Remove Exploratory View components from Observability (elastic#155629)
  [Discover] Remove redundant "Filter was added" toast (elastic#155645)
  [RAM][Security Solution][Alerts] Support the ability to trigger a rule action per alert generated (elastic#153611) (elastic#155384)
  ...
@jloleysens
fix snapshot and update code
f0dc206
@jloleysens
remove eslint-disable?
b34f0aa
@kibanamachine
[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…
6432cb5 
…fix'
@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id before after diff
enterpriseSearch 17 19 +2
securitySolution 399 402 +3
total +5

Total ESLint disabled count

id before after diff
enterpriseSearch 18 20 +2
securitySolution 479 482 +3
total +5

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jloleysens

@rudolf rudolf added v8.9.0 and removed v8.8.0 labels Apr 27, 2023
@rudolf rudolf merged commit e9197ad into elastic:main Apr 27, 2023
@kibanamachine kibanamachine added the backport:skip This commit does not require backporting label Apr 27, 2023
@jloleysens jloleysens deleted the compatible-schema-change-check branch April 27, 2023 18:18
@pgayvallet pgayvallet mentioned this pull request May 1, 2023
Merged
pgayvallet added a commit that referenced this pull request May 2, 2023
@pgayvallet
Adapt the ZDT migration algorithm to support v2 migrations (#155981)
9cddb60 
## Summary

Adapt the `zdt` migration algorithm to run the v2 migrations in addition
to the model version transformations.

The intent is to be able to use the current migration system in a
zero-downtime upgrade friendly-ish way

### Technicals  

#148656 was a precondition, as the
zdt algo requires that mapping changes are compatible (given we keep the
same index).

Technically, it means we're now storing the `virtualVersion` instead of
the `modelVersion` in the index's meta (`mappingVersions` and
`docVersions`), to allow keeping track of mixed stack and model versions
per type.

Note that switching to model versioning is still a one way,
non-revertible action. Once using model versioning (by specifying
`switchToModelVersionAt` on your type), there is no going back.
pgayvallet added a commit to pgayvallet/kibana that referenced this pull request May 2, 2023
@pgayvallet
Adapt the ZDT migration algorithm to support v2 migrations (elastic#1…
57933aa 
…55981)

## Summary

Adapt the `zdt` migration algorithm to run the v2 migrations in addition
to the model version transformations.

The intent is to be able to use the current migration system in a
zero-downtime upgrade friendly-ish way

### Technicals  

elastic#148656 was a precondition, as the
zdt algo requires that mapping changes are compatible (given we keep the
same index).

Technically, it means we're now storing the `virtualVersion` instead of
the `modelVersion` in the index's meta (`mappingVersions` and
`docVersions`), to allow keeping track of mixed stack and model versions
per type.

Note that switching to model versioning is still a one way,
non-revertible action. Once using model versioning (by specifying
`switchToModelVersionAt` on your type), there is no going back.
pgayvallet added a commit to pgayvallet/kibana that referenced this pull request May 2, 2023
@pgayvallet
Adapt the ZDT migration algorithm to support v2 migrations (elastic#1…
3cfb847 
…55981)

## Summary

Adapt the `zdt` migration algorithm to run the v2 migrations in addition
to the model version transformations.

The intent is to be able to use the current migration system in a
zero-downtime upgrade friendly-ish way

### Technicals  

elastic#148656 was a precondition, as the
zdt algo requires that mapping changes are compatible (given we keep the
same index).

Technically, it means we're now storing the `virtualVersion` instead of
the `modelVersion` in the index's meta (`mappingVersions` and
`docVersions`), to allow keeping track of mixed stack and model versions
per type.

Note that switching to model versioning is still a one way,
non-revertible action. Once using model versioning (by specifying
`switchToModelVersionAt` on your type), there is no going back.
@rudolf rudolf mentioned this pull request May 22, 2023
Closed
@rudolf rudolf added the Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects label Sep 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@TinaHeiligers TinaHeiligers TinaHeiligers left review comments

@rudolf rudolf rudolf approved these changes

@spalger spalger spalger approved these changes

@pgayvallet pgayvallet pgayvallet approved these changes

@jbudz jbudz jbudz approved these changes

Assignees

@jloleysens jloleysens

Labels
auto-backport Deprecated - use backport:version if exact versions are needed backport:skip This commit does not require backporting Epic:ScaleMigrations Scale upgrade migrations to millions of saved objects Feature:Migrations Feature:Saved Objects release_note:skip Skip the PR/issue when compiling release notes Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc v8.9.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
9 participants
@jloleysens @elasticmachine @pgayvallet @spalger @rudolf @kibana-ci @jbudz @TinaHeiligers @kibanamachine