[Fleet][EPM] Don't roll back on saved objects conflict errors. #85131
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
skh
added
Feature:EPM
|
Pinging @elastic/ingest-management (Feature:EPM) |
skh
changed the title
skh
changed the title
skh
commented
Dec 7, 2020
|
@elasticmachine merge upstream |
jonathan-buttner
approved these changes
Dec 7, 2020
neptunian
requested changes
Dec 7, 2020
neptunian
approved these changes
Dec 8, 2020
skh
force-pushed
the
84651-check-for-saved-object-version-conflict
branch
from
846781c
to
69d125c
Compare
skh
added a commit
that referenced
this pull request
Dec 14, 2020
⏳ Build in-progress, with failures
Failed CI Steps
History
To update your PR or re-run it, just comment with: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This refines the implementation of #84190 and implements #84651 . See also #84656 for a bit of discussion.
This changes the behavior of
_installPackage()so thatConcurrentInstallOperationErroris thrown (instead of returning a list of installed assets which may or may not be complete)install*()methods called by_installPackage(), this is also wrapped in aConcurrentInstallOperationErrorConcurrentInstallOperationErrorwill not trigger a rollback. This fixes the bug that occurs when a second installation/upgrade operation aborts because of a saved object version conflict, and therefore rolls back the installation that a first installation operation just completed successfully, potentially resulting in follow-up errors and a broken installationConcurrentInstallOperationErrorwill cause the handler to return a409HTTP response with a message stating on which package the concurrent installation was detected, and that the operation was aborted.This is still a rather optimistic way of handling this situation: when a concurrent installation is detected, the running installation is aborted and no attempts are made to clean up after it. This is possible because the
install*()methods (installing kibana assets, pipelines, templates etc.) are idempotent. Indeed it is still perfectly possible that two parallel installations run successfully, installing everything twice, or that they only run into the saved object conflict at the very end, after almost everything was installed twice.This may have effects on other users of the install package code flow, namely endpoint security. (cc @jonathan-buttner )
How to test this
Try to get the installed package into a broken state. To do that, try to trigger a race condition by installing the same package several times at once, and observe if the race condition is handled correctly. https://gist.github.com/skh/cc695952031c9e349874b898c7066e42 may be helpful for this -- I had to set
WAIT_TIME_REINSTALLin that script to0and run it a few times.Try to break it in any other way.