[Ingest Manager] Implement concurrency control for package configs #70680

jen-huang · 2020-07-03T00:32:59Z

Summary

Resolves #69992. This PR enables concurrency control on package config saved objects by:

Surface version field provided by saved objects to get package config(s) requests
Send version field as part of the payload for update package config requests, where the saved object client will perform version check and throw a 409 if there is a conflict
Display a toast message in the UI for 409 responses

Testing:

Open Edit integration in two tabs
Submit the first tab, then attempt to submit the second tab
First tab should be successful whereas the second tab should display an error toast message

…ackage config update

elasticmachine · 2020-07-03T00:33:02Z

Pinging @elastic/ingest-management (Team:Ingest Management)

jfsiii

LGTM. I can submit a follow up PR which uses a custom error instead of status code

jen-huang · 2020-07-06T14:06:19Z

@elasticmachine merge upstream

…70680) (#70835) * Send SO version field as part of package configs, enforce it during package config update * Fix typings, extend response error to include optional status code * Revert unnecessary version fields in tests, fix schema Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

kibanamachine · 2020-07-06T20:32:25Z

💛 Build succeeded, but was flaky

continuous-integration/kibana-ci/pull-request
Commit: 27b1ee9
Flaky suites:
- xpack-kibana-ciGroup7

Test Failures

Kibana Pipeline / kibana-xpack-agent / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/dashboard/_async_dashboard·ts.dashboard sample data dashboard "before all" hook for "should launch sample flights data set dashboard"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 2 times on tracked branches: https://github.com/elastic/kibana/issues/70715

[00:00:00]       │
[00:00:00]         └-: dashboard
[00:00:00]           └-> "before all" hook
[00:09:41]           └-: sample data dashboard
[00:09:41]             └-> "before all" hook
[00:09:41]             └-> "before all" hook
[00:09:41]               │ debg ... sleep(5000) start
[00:09:46]               │ debg ... sleep(5000) end
[00:09:46]               │ debg navigateToActualUrl http://localhost:6181/app/home#/tutorial_directory/sampleData
[00:09:47]               │ debg browser[INFO] http://localhost:6181/app/home?_t=1594063947833#/tutorial_directory/sampleData 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:09:47]               │
[00:09:47]               │ debg browser[INFO] http://localhost:6181/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:09:47]               │ debg currentUrl = http://localhost:6181/app/home#/tutorial_directory/sampleData
[00:09:47]               │          appUrl = http://localhost:6181/app/home#/tutorial_directory/sampleData
[00:09:47]               │ debg TestSubjects.find(kibanaChrome)
[00:09:47]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:09:48]               │ debg isGlobalLoadingIndicatorVisible
[00:09:48]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:09:48]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:09:48]               │ debg browser[INFO] http://localhost:6181/34337/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 452:106112 "INFO: 2020-07-06T19:32:29Z
[00:09:48]               │        Adding connection to http://localhost:6181/elasticsearch
[00:09:48]               │
[00:09:48]               │      "
[00:09:49]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:09:49]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:09:49]               │ debg TestSubjects.exists(addSampleDataSetflights)
[00:09:49]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="addSampleDataSetflights"]') with timeout=2500
[00:09:49]               │ debg TestSubjects.click(addSampleDataSetflights)
[00:09:49]               │ debg Find.clickByCssSelector('[data-test-subj="addSampleDataSetflights"]') with timeout=10000
[00:09:49]               │ debg Find.findByCssSelector('[data-test-subj="addSampleDataSetflights"]') with timeout=10000
[00:09:49]               │ debg TestSubjects.find(sampleDataSetCardflights)
[00:09:49]               │ debg Find.findByCssSelector('[data-test-subj="sampleDataSetCardflights"]') with timeout=10000
[00:09:50]               │ debg TestSubjects.exists(addSampleDataSetflights)
[00:09:50]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="addSampleDataSetflights"]') with timeout=2500
[00:09:50]               │ info Taking screenshot "/dev/shm/workspace/kibana/x-pack/test/functional/screenshots/failure/dashboard sample data dashboard _before all_ hook.png"
[00:09:50]               │ info Current URL is: http://localhost:6181/app/home#/tutorial_directory/sampleData
[00:09:51]               │ info Saving page source to: /dev/shm/workspace/kibana/x-pack/test/functional/failure_debug/html/dashboard sample data dashboard _before all_ hook.html
[00:09:51]               └- ✖ fail: "dashboard sample data dashboard "before all" hook for "should launch sample flights data set dashboard""
[00:09:51]               │

Stack Trace

Error: expected false to equal true
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.be.Assertion.equal (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:227:8)
    at Assertion.be (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:69:22)
    at Context.before (test/functional/apps/dashboard/_async_dashboard.ts:39:30)
    at process._tickCallback (internal/process/next_tick.js:68:7)

Kibana Pipeline / kibana-xpack-agent / Chrome X-Pack UI Functional Tests.x-pack/test/functional/apps/dashboard/_async_dashboard·ts.dashboard sample data dashboard "after all" hook for "toggle from Discover to Dashboard attempt 2"

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 15 times on tracked branches: https://github.com/elastic/kibana/issues/65949

[00:00:00]       │
[00:00:00]         └-: dashboard
[00:00:00]           └-> "before all" hook
[00:09:41]           └-: sample data dashboard
[00:09:41]             └-> "before all" hook
[00:09:41]             └-> "before all" hook
[00:09:41]               │ debg ... sleep(5000) start
[00:09:46]               │ debg ... sleep(5000) end
[00:09:46]               │ debg navigateToActualUrl http://localhost:6181/app/home#/tutorial_directory/sampleData
[00:09:47]               │ debg browser[INFO] http://localhost:6181/app/home?_t=1594063947833#/tutorial_directory/sampleData 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:09:47]               │
[00:09:47]               │ debg browser[INFO] http://localhost:6181/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:09:47]               │ debg currentUrl = http://localhost:6181/app/home#/tutorial_directory/sampleData
[00:09:47]               │          appUrl = http://localhost:6181/app/home#/tutorial_directory/sampleData
[00:09:47]               │ debg TestSubjects.find(kibanaChrome)
[00:09:47]               │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:09:48]               │ debg isGlobalLoadingIndicatorVisible
[00:09:48]               │ debg TestSubjects.exists(globalLoadingIndicator)
[00:09:48]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:09:48]               │ debg browser[INFO] http://localhost:6181/34337/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 452:106112 "INFO: 2020-07-06T19:32:29Z
[00:09:48]               │        Adding connection to http://localhost:6181/elasticsearch
[00:09:48]               │
[00:09:48]               │      "
[00:09:49]               │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:09:49]               │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:09:49]               │ debg TestSubjects.exists(addSampleDataSetflights)
[00:09:49]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="addSampleDataSetflights"]') with timeout=2500
[00:09:49]               │ debg TestSubjects.click(addSampleDataSetflights)
[00:09:49]               │ debg Find.clickByCssSelector('[data-test-subj="addSampleDataSetflights"]') with timeout=10000
[00:09:49]               │ debg Find.findByCssSelector('[data-test-subj="addSampleDataSetflights"]') with timeout=10000
[00:09:49]               │ debg TestSubjects.find(sampleDataSetCardflights)
[00:09:49]               │ debg Find.findByCssSelector('[data-test-subj="sampleDataSetCardflights"]') with timeout=10000
[00:09:50]               │ debg TestSubjects.exists(addSampleDataSetflights)
[00:09:50]               │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="addSampleDataSetflights"]') with timeout=2500
[00:09:50]               │ info Taking screenshot "/dev/shm/workspace/kibana/x-pack/test/functional/screenshots/failure/dashboard sample data dashboard _before all_ hook.png"
[00:09:50]               │ info Current URL is: http://localhost:6181/app/home#/tutorial_directory/sampleData
[00:09:51]               │ info Saving page source to: /dev/shm/workspace/kibana/x-pack/test/functional/failure_debug/html/dashboard sample data dashboard _before all_ hook.html
[00:09:51]               └- ✖ fail: "dashboard sample data dashboard "before all" hook for "should launch sample flights data set dashboard""
[00:09:51]               │
[00:09:51]               └-> "after all" hook
[00:09:51]                 │ debg navigateToActualUrl http://localhost:6181/app/home#/tutorial_directory/sampleData
[00:09:51]                 │ debg browser[INFO] http://localhost:6181/app/home?_t=1594063951999#/tutorial_directory/sampleData 341 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:09:51]                 │
[00:09:51]                 │ debg browser[INFO] http://localhost:6181/bundles/app/core/bootstrap.js 42:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:09:51]                 │ debg currentUrl = http://localhost:6181/app/home#/tutorial_directory/sampleData
[00:09:51]                 │          appUrl = http://localhost:6181/app/home#/tutorial_directory/sampleData
[00:09:51]                 │ debg TestSubjects.find(kibanaChrome)
[00:09:51]                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:09:52]                 │ debg isGlobalLoadingIndicatorVisible
[00:09:52]                 │ debg TestSubjects.exists(globalLoadingIndicator)
[00:09:52]                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:09:52]                 │ debg browser[INFO] http://localhost:6181/34337/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js 452:106112 "INFO: 2020-07-06T19:32:33Z
[00:09:52]                 │        Adding connection to http://localhost:6181/elasticsearch
[00:09:52]                 │
[00:09:52]                 │      "
[00:09:53]                 │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:09:53]                 │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:09:53]                 │ debg TestSubjects.find(removeSampleDataSetflights)
[00:09:53]                 │ debg Find.findByCssSelector('[data-test-subj="removeSampleDataSetflights"]') with timeout=10000
[00:10:03]                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
[00:10:03]                 │      Wait timed out after 10036ms
[00:10:04]                 │ debg TestSubjects.find(removeSampleDataSetflights)
[00:10:04]                 │ debg Find.findByCssSelector('[data-test-subj="removeSampleDataSetflights"]') with timeout=10000
[00:10:14]                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
[00:10:14]                 │      Wait timed out after 10043ms
[00:10:14]                 │ debg TestSubjects.find(removeSampleDataSetflights)
[00:10:14]                 │ debg Find.findByCssSelector('[data-test-subj="removeSampleDataSetflights"]') with timeout=10000
[00:10:24]                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
[00:10:24]                 │      Wait timed out after 10059ms
[00:10:25]                 │ debg TestSubjects.find(removeSampleDataSetflights)
[00:10:25]                 │ debg Find.findByCssSelector('[data-test-subj="removeSampleDataSetflights"]') with timeout=10000
[00:10:35]                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
[00:10:35]                 │      Wait timed out after 10034ms
[00:10:35]                 │ debg TestSubjects.find(removeSampleDataSetflights)
[00:10:35]                 │ debg Find.findByCssSelector('[data-test-subj="removeSampleDataSetflights"]') with timeout=10000
[00:10:45]                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
[00:10:45]                 │      Wait timed out after 10013ms
[00:10:46]                 │ debg TestSubjects.find(removeSampleDataSetflights)
[00:10:46]                 │ debg Find.findByCssSelector('[data-test-subj="removeSampleDataSetflights"]') with timeout=10000
[00:10:56]                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
[00:10:56]                 │      Wait timed out after 10035ms
[00:10:56]                 │ debg TestSubjects.find(removeSampleDataSetflights)
[00:10:56]                 │ debg Find.findByCssSelector('[data-test-subj="removeSampleDataSetflights"]') with timeout=10000
[00:11:06]                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
[00:11:06]                 │      Wait timed out after 10013ms
[00:11:07]                 │ debg TestSubjects.find(removeSampleDataSetflights)
[00:11:07]                 │ debg Find.findByCssSelector('[data-test-subj="removeSampleDataSetflights"]') with timeout=10000
[00:11:17]                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
[00:11:17]                 │      Wait timed out after 10012ms
[00:11:17]                 │ debg TestSubjects.find(removeSampleDataSetflights)
[00:11:17]                 │ debg Find.findByCssSelector('[data-test-subj="removeSampleDataSetflights"]') with timeout=10000
[00:11:27]                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
[00:11:27]                 │      Wait timed out after 10058ms
[00:11:28]                 │ debg TestSubjects.find(removeSampleDataSetflights)
[00:11:28]                 │ debg Find.findByCssSelector('[data-test-subj="removeSampleDataSetflights"]') with timeout=10000
[00:11:38]                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
[00:11:38]                 │      Wait timed out after 10015ms
[00:11:38]                 │ debg TestSubjects.find(removeSampleDataSetflights)
[00:11:38]                 │ debg Find.findByCssSelector('[data-test-subj="removeSampleDataSetflights"]') with timeout=10000
[00:11:48]                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
[00:11:48]                 │      Wait timed out after 10055ms
[00:11:49]                 │ debg TestSubjects.find(removeSampleDataSetflights)
[00:11:49]                 │ debg Find.findByCssSelector('[data-test-subj="removeSampleDataSetflights"]') with timeout=10000
[00:11:59]                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
[00:11:59]                 │      Wait timed out after 10050ms
[00:11:59]                 │ info Taking screenshot "/dev/shm/workspace/kibana/x-pack/test/functional/screenshots/failure/dashboard sample data dashboard _after all_ hook.png"
[00:12:00]                 │ info Current URL is: http://localhost:6181/app/home#/tutorial_directory/sampleData
[00:12:00]                 │ info Saving page source to: /dev/shm/workspace/kibana/x-pack/test/functional/failure_debug/html/dashboard sample data dashboard _after all_ hook.html
[00:12:00]                 └- ✖ fail: "dashboard sample data dashboard "after all" hook for "toggle from Discover to Dashboard attempt 2""
[00:12:00]                 │

Stack Trace

Error: retry.tryForTime timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="removeSampleDataSetflights"])
Wait timed out after 10050ms
    at /dev/shm/workspace/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at process._tickCallback (internal/process/next_tick.js:68:7)
    at onFailure (/dev/shm/workspace/kibana/test/common/services/retry/retry_for_success.ts:28:9)
    at retryForSuccess (/dev/shm/workspace/kibana/test/common/services/retry/retry_for_success.ts:68:13)

Build metrics

✅ unchanged

History

💚 Build #58965 succeeded 27b1ee9
💚 Build #58574 succeeded 8a83e27

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

jen-huang added 3 commits July 2, 2020 16:40

Send SO version field as part of package configs, enforce it during p…

69a2228

…ackage config update

Fix typings, extend response error to include optional status code

87e6362

Revert unnecessary version fields in tests, fix schema

8a83e27

jen-huang added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.9.0 Team:Fleet Team label for Observability Data Collection Fleet team labels Jul 3, 2020

jen-huang requested a review from a team July 3, 2020 00:33

jen-huang self-assigned this Jul 3, 2020

jfsiii approved these changes Jul 6, 2020

View reviewed changes

Merge branch 'master' into ingest/pkg-config-concurrency

27b1ee9

jen-huang merged commit cbd39d9 into elastic:master Jul 6, 2020

jen-huang deleted the ingest/pkg-config-concurrency branch July 6, 2020 16:45

jen-huang mentioned this pull request Jul 6, 2020

[7.x] [Ingest Manager] Implement concurrency control for package configs (#70680) #70835

Merged

madirey mentioned this pull request Jul 6, 2020

[Security Solution][Endpoint] User Manifest Cleanup + Artifact Compression #70759

Merged

21 tasks

jen-huang mentioned this pull request Jul 23, 2020

[Ingest Manager] Fix broken copy agent config #73128

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Manager] Implement concurrency control for package configs #70680

[Ingest Manager] Implement concurrency control for package configs #70680

jen-huang commented Jul 3, 2020 •

edited

Loading

elasticmachine commented Jul 3, 2020

jfsiii left a comment

jen-huang commented Jul 6, 2020

kibanamachine commented Jul 6, 2020

Standard Out

Stack Trace

Standard Out

Stack Trace

[Ingest Manager] Implement concurrency control for package configs #70680

[Ingest Manager] Implement concurrency control for package configs #70680

Conversation

jen-huang commented Jul 3, 2020 • edited Loading

Summary

elasticmachine commented Jul 3, 2020

jfsiii left a comment

Choose a reason for hiding this comment

jen-huang commented Jul 6, 2020

kibanamachine commented Jul 6, 2020

💛 Build succeeded, but was flaky

Test Failures

Standard Out

Stack Trace

Standard Out

Stack Trace

Build metrics

History

jen-huang commented Jul 3, 2020 •

edited

Loading