Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[index patterns] Fleep app - Keep saved object field list until field caps provides fields #85370

Merged

Conversation

mattkime
Copy link
Contributor

@mattkime mattkime commented Dec 9, 2020

Summary

The Fleet app needs an index pattern field list to exist before documents do so that the dashboards it installs don't show errors before there's data. It provides a field list via the index pattern saved object. Index patterns need to use it until field caps provides a complete response.

Plans

  • logs-* and metrics-* index patterns provided by Fleet will have a flag that will ignore field caps 404 errors.
  • If an index pattern has fields in the field list, they'll be kept until the field caps request returns a response with at least one field.
  • Create migration for index patterns with ids metrics-* and logs-*

Checklist

Delete any items that are not applicable to this PR.

@mattkime
Copy link
Contributor Author

@elasticmachine merge upstream

@mattkime
Copy link
Contributor Author

@elasticmachine merge upstream

@mattkime
Copy link
Contributor Author

@elasticmachine merge upstream

@neptunian
Copy link
Contributor

@jonathan-buttner do you think the failing endpoint test is related to our changes?

@mattkime mattkime added the Feature:Data Views Data Views code and UI - index patterns before 8.0 label Dec 15, 2020
@mattkime mattkime requested a review from a team December 16, 2020 03:03
@mattkime mattkime requested review from a team as code owners December 16, 2020 03:03
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-app-services (Team:AppServices)

@mattkime mattkime added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes labels Dec 16, 2020
@botelastic botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Dec 16, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/ingest-management (Team:Ingest Management)

Copy link
Contributor

@jonathan-buttner jonathan-buttner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Endpoint changes LGTM ;)

Copy link
Member

@lukeelmers lukeelmers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM (note: I did not test the migration locally)

  • Should we be applying the migration for 7.11.1 at this point?
  • Super nit: I think requireIndex would be easier to grok than allowNoIndex, but that's a bit of a subjective opinion.

@mattkime
Copy link
Contributor Author

@elasticmachine merge upstream

@kibanamachine
Copy link
Contributor

💛 Build succeeded, but was flaky


Test Failures

X-Pack Reporting API Integration Tests.x-pack/test/reporting_api_integration/reporting_and_security/network_policy·ts.Reporting APIs Network Policy should fail job when page voilates the network policy

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has not failed recently on tracked branches

[00:00:00]       │
[00:00:00]         └-: Reporting APIs
[00:00:00]           └-> "before all" hook
[00:01:31]           └-: Network Policy
[00:01:31]             └-> "before all" hook
[00:01:31]             └-> "before all" hook
[00:01:31]               │ info [reporting/canvas_disallowed_url] Loading "mappings.json"
[00:01:31]               │ info [reporting/canvas_disallowed_url] Loading "data.json.gz"
[00:01:31]               │ info [o.e.c.m.MetadataDeleteIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1608402935274764996] [.kibana_2/vhZFVplKSv-o0NVZpUJrQQ] deleting index
[00:01:31]               │ info [o.e.c.m.MetadataDeleteIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1608402935274764996] [.kibana_1/LehxTq6CQNaN9uIWYlHwqQ] deleting index
[00:01:31]               │ info [reporting/canvas_disallowed_url] Deleted existing index [".kibana_2",".kibana_1"]
[00:01:31]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1608402935274764996] [.kibana_1] creating index, cause [api], templates [], shards [1]/[0]
[00:01:31]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1608402935274764996] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_1][0]]])." previous.health="YELLOW" reason="shards started [[.kibana_1][0]]"
[00:01:31]               │ info [reporting/canvas_disallowed_url] Created index ".kibana_1"
[00:01:31]               │ debg [reporting/canvas_disallowed_url] ".kibana_1" settings {"index":{"auto_expand_replicas":"0-1","number_of_replicas":"0","number_of_shards":"1"}}
[00:01:31]               │ info [reporting/canvas_disallowed_url] Indexed 2 docs into ".kibana_1"
[00:01:31]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1608402935274764996] [.kibana_1/4tzUCDlnSIS6tsDiL4A5DQ] update_mapping [_doc]
[00:01:31]               │ debg Migrating saved objects
[00:01:31]               │ proc [kibana]   log   [20:00:26.887] [info][savedobjects-service] Detected mapping change in "dynamic"
[00:01:31]               │ proc [kibana]   log   [20:00:26.889] [info][savedobjects-service] Creating index .kibana_2.
[00:01:31]               │ info [o.e.c.m.MetadataCreateIndexService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1608402935274764996] [.kibana_2] creating index, cause [api], templates [], shards [1]/[1]
[00:01:31]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1608402935274764996] updating number_of_replicas to [0] for indices [.kibana_2]
[00:01:31]               │ info [o.e.c.r.a.AllocationService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1608402935274764996] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana_2][0]]])." previous.health="YELLOW" reason="shards started [[.kibana_2][0]]"
[00:01:31]               │ proc [kibana]   log   [20:00:26.965] [info][savedobjects-service] Migrating .kibana_1 saved objects to .kibana_2
[00:01:31]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1608402935274764996] [.kibana_2/Xi4JF-BYRVys0XdGoefmoA] update_mapping [_doc]
[00:01:31]               │ info [o.e.c.m.MetadataMappingService] [kibana-ci-immutable-ubuntu-16-tests-xxl-1608402935274764996] [.kibana_2/Xi4JF-BYRVys0XdGoefmoA] update_mapping [_doc]
[00:01:31]               │ proc [kibana]   log   [20:00:27.093] [info][savedobjects-service] Pointing alias .kibana to .kibana_2.
[00:01:31]               │ proc [kibana]   log   [20:00:27.132] [info][savedobjects-service] Finished in 245ms.
[00:01:31]             └-> should fail job when page voilates the network policy
[00:01:31]               └-> "before each" hook: global before each
[00:01:31]               │ debg ReportingAPI.postJob(/api/reporting/generate/printablePdf?jobParams=(layout:(dimensions:(height:720,width:1080),id:preserve_layout),objectType:'canvas%20workpad',relativeUrls:!(%2Fapp%2Fcanvas%23%2Fexport%2Fworkpad%2Fpdf%2Fworkpad-e7464259-0b75-4b8c-81c8-8422b15ff201%2Fpage%2F1),title:'My%20Canvas%20Workpad'))
[00:01:31]               │ proc [kibana]   log   [20:00:27.198] [info][plugins][queue-job][reporting] Scheduled PDF report: kiw4m0im5mfibcdb6d39a2f3
[00:01:34]               │ proc [kibana]   log   [20:00:29.291] [info][esqueue][plugins][queue-worker][reporting] kiw4k1hx5mfibcdb6d2ko31o - Job marked as claimed: /.reporting-2020-12-13/kiw4m0im5mfibcdb6d39a2f3
[00:01:34]               │ proc [kibana]   log   [20:00:29.292] [info][esqueue][plugins][queue-worker][reporting] kiw4k1hx5mfibcdb6d2ko31o - Starting job
[00:01:34]               │ proc [kibana]   log   [20:00:29.957] [info][browser-driver][execute-job][kiw4m0im5mfibcdb6d39a2f3][kiw4m0im5mfibcdb6d39a2f3][plugins][printable_pdf][reporting] Creating browser page driver
[00:01:35]               │ proc [kibana]   log   [20:00:30.373] [info][execute-job][kiw4m0im5mfibcdb6d39a2f3][kiw4m0im5mfibcdb6d39a2f3][plugins][printable_pdf][reporting] opening url http://localhost:6161/app/canvas#/export/workpad/pdf/workpad-e7464259-0b75-4b8c-81c8-8422b15ff201/page/1?forceNow=2020-12-19T20%3A00%3A27.166Z
[00:01:35]               │ proc [kibana]   log   [20:00:30.511] [error][browser-driver][execute-job][headless-browser-console][kiw4m0im5mfibcdb6d39a2f3][kiw4m0im5mfibcdb6d39a2f3][plugins][printable_pdf][reporting] Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:01:35]               │ proc [kibana] 
[00:01:36]               │ proc [kibana]   log   [20:00:32.093] [error][browser-driver][execute-job][headless-browser-console][kiw4m0im5mfibcdb6d39a2f3][kiw4m0im5mfibcdb6d39a2f3][plugins][printable_pdf][reporting] JSHandle@error
[00:01:36]               │ proc [kibana]   log   [20:00:32.124] [error][execute-job][kiw4m0im5mfibcdb6d39a2f3][kiw4m0im5mfibcdb6d39a2f3][plugins][printable_pdf][reporting] Reporting encountered an error on the page: Error: Error: No trigger [triggerId = SELECT_RANGE_TRIGGER] exists, for attaching action [actionId = ACTION_SELECT_RANGE].
[00:01:36]               │ proc [kibana]     at ui_actions_service_UiActionsService.attachAction (http://localhost:6161/39152/bundles/plugin/uiActions/uiActions.plugin.js:1:60155)
[00:01:36]               │ proc [kibana]     at ui_actions_service_UiActionsService.addTriggerAction (http://localhost:6161/39152/bundles/plugin/uiActions/uiActions.plugin.js:1:60970)
[00:01:36]               │ proc [kibana]     at plugin_DataPublicPlugin.setup (http://localhost:6161/39152/bundles/plugin/data/data.plugin.js:8:231843)
[00:01:36]               │ proc [kibana]     at _callee$ (http://localhost:6161/39152/bundles/core/core.entry.js:13:251689)
[00:01:36]               │ proc [kibana]     at l (http://localhost:6161/39152/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968491)
[00:01:36]               │ proc [kibana]     at Generator._invoke (http://localhost:6161/39152/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968244)
[00:01:36]               │ proc [kibana]     at Generator.forEach.e.<computed> [as next] (http://localhost:6161/39152/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968848)
[00:01:36]               │ proc [kibana]     at plugin_asyncGeneratorStep (http://localhost:6161/39152/bundles/core/core.entry.js:13:249793)
[00:01:36]               │ proc [kibana]     at _next (http://localhost:6161/39152/bundles/core/core.entry.js:13:250110)
[00:01:36]               │ proc [kibana]   log   [20:00:32.129] [error][error][esqueue][plugins][queue-worker][reporting] kiw4k1hx5mfibcdb6d2ko31o - Failure occurred on job kiw4m0im5mfibcdb6d39a2f3: Reporting encountered an error on the page: Error: Error: No trigger [triggerId = SELECT_RANGE_TRIGGER] exists, for attaching action [actionId = ACTION_SELECT_RANGE].
[00:01:36]               │ proc [kibana]     at ui_actions_service_UiActionsService.attachAction (http://localhost:6161/39152/bundles/plugin/uiActions/uiActions.plugin.js:1:60155)
[00:01:36]               │ proc [kibana]     at ui_actions_service_UiActionsService.addTriggerAction (http://localhost:6161/39152/bundles/plugin/uiActions/uiActions.plugin.js:1:60970)
[00:01:36]               │ proc [kibana]     at plugin_DataPublicPlugin.setup (http://localhost:6161/39152/bundles/plugin/data/data.plugin.js:8:231843)
[00:01:36]               │ proc [kibana]     at _callee$ (http://localhost:6161/39152/bundles/core/core.entry.js:13:251689)
[00:01:36]               │ proc [kibana]     at l (http://localhost:6161/39152/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968491)
[00:01:36]               │ proc [kibana]     at Generator._invoke (http://localhost:6161/39152/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968244)
[00:01:36]               │ proc [kibana]     at Generator.forEach.e.<computed> [as next] (http://localhost:6161/39152/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968848)
[00:01:36]               │ proc [kibana]     at plugin_asyncGeneratorStep (http://localhost:6161/39152/bundles/core/core.entry.js:13:249793)
[00:01:36]               │ proc [kibana]     at _next (http://localhost:6161/39152/bundles/core/core.entry.js:13:250110)
[00:01:36]               │ proc [kibana]   log   [20:00:32.130] [warning][esqueue][plugins][queue-worker][reporting] kiw4k1hx5mfibcdb6d2ko31o - Failing job kiw4m0im5mfibcdb6d39a2f3
[00:01:36]               │ proc [kibana]   log   [20:00:32.143] [error][execute-job][kiw4m0im5mfibcdb6d39a2f3][kiw4m0im5mfibcdb6d39a2f3][plugins][printable_pdf][reporting] Error: Protocol error (Runtime.callFunctionOn): Target closed.
[00:01:36]               │ proc [kibana]     at /dev/shm/workspace/kibana-build-xpack-6/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:208:63
[00:01:36]               │ proc [kibana]     at new Promise (<anonymous>)
[00:01:36]               │ proc [kibana]     at CDPSession.send (/dev/shm/workspace/kibana-build-xpack-6/node_modules/puppeteer/lib/cjs/puppeteer/common/Connection.js:207:16)
[00:01:36]               │ proc [kibana]     at ExecutionContext._evaluateInternal (/dev/shm/workspace/kibana-build-xpack-6/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:200:50)
[00:01:36]               │ proc [kibana]     at ExecutionContext.evaluateHandle (/dev/shm/workspace/kibana-build-xpack-6/node_modules/puppeteer/lib/cjs/puppeteer/common/ExecutionContext.js:151:21)
[00:01:36]               │ proc [kibana]     at WaitTask.rerun (/dev/shm/workspace/kibana-build-xpack-6/node_modules/puppeteer/lib/cjs/puppeteer/common/DOMWorld.js:528:37)
[00:01:36]               │ proc [kibana]     at runMicrotasks (<anonymous>)
[00:01:36]               │ proc [kibana]     at processTicksAndRejections (internal/process/task_queues.js:93:5)
[00:01:36]               │ proc [kibana]   log   [20:00:32.155] [info][esqueue][plugins][queue-worker][reporting] kiw4k1hx5mfibcdb6d2ko31o - Job marked as failed: /.reporting-2020-12-13/kiw4m0im5mfibcdb6d39a2f3
[00:01:37]               └- ✖ fail: Reporting APIs Network Policy should fail job when page voilates the network policy
[00:01:37]               │      Error: expected 'Reporting generation failed: Reporting encountered an error on the page: Error: Error: No trigger [triggerId = SELECT_RANGE_TRIGGER] exists, for attaching action [actionId = ACTION_SELECT_RANGE].\n    at ui_actions_service_UiActionsService.attachAction (http://localhost:6161/39152/bundles/plugin/uiActions/uiActions.plugin.js:1:60155)\n    at ui_actions_service_UiActionsService.addTriggerAction (http://localhost:6161/39152/bundles/plugin/uiActions/uiActions.plugin.js:1:60970)\n    at plugin_DataPublicPlugin.setup (http://localhost:6161/39152/bundles/plugin/data/data.plugin.js:8:231843)\n    at _callee$ (http://localhost:6161/39152/bundles/core/core.entry.js:13:251689)\n    at l (http://localhost:6161/39152/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968491)\n    at Generator._invoke (http://localhost:6161/39152/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968244)\n    at Generator.forEach.e.<computed> [as next] (http://localhost:6161/39152/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968848)\n    at plugin_asyncGeneratorStep (http://localhost:6161/39152/bundles/core/core.entry.js:13:249793)\n    at _next (http://localhost:6161/39152/bundles/core/core.entry.js:13:250110)' to match /Reporting generation failed: Error:/
[00:01:37]               │       at Assertion.assert (/dev/shm/workspace/parallel/6/kibana/packages/kbn-expect/expect.js:100:11)
[00:01:37]               │       at Assertion.match (/dev/shm/workspace/parallel/6/kibana/packages/kbn-expect/expect.js:354:8)
[00:01:37]               │       at Context.<anonymous> (test/reporting_api_integration/reporting_and_security/network_policy.ts:48:31)
[00:01:37]               │       at Object.apply (/dev/shm/workspace/parallel/6/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)
[00:01:37]               │ 
[00:01:37]               │ 

Stack Trace

Error: expected 'Reporting generation failed: Reporting encountered an error on the page: Error: Error: No trigger [triggerId = SELECT_RANGE_TRIGGER] exists, for attaching action [actionId = ACTION_SELECT_RANGE].\n    at ui_actions_service_UiActionsService.attachAction (http://localhost:6161/39152/bundles/plugin/uiActions/uiActions.plugin.js:1:60155)\n    at ui_actions_service_UiActionsService.addTriggerAction (http://localhost:6161/39152/bundles/plugin/uiActions/uiActions.plugin.js:1:60970)\n    at plugin_DataPublicPlugin.setup (http://localhost:6161/39152/bundles/plugin/data/data.plugin.js:8:231843)\n    at _callee$ (http://localhost:6161/39152/bundles/core/core.entry.js:13:251689)\n    at l (http://localhost:6161/39152/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968491)\n    at Generator._invoke (http://localhost:6161/39152/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968244)\n    at Generator.forEach.e.<computed> [as next] (http://localhost:6161/39152/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:321:968848)\n    at plugin_asyncGeneratorStep (http://localhost:6161/39152/bundles/core/core.entry.js:13:249793)\n    at _next (http://localhost:6161/39152/bundles/core/core.entry.js:13:250110)' to match /Reporting generation failed: Error:/
    at Assertion.assert (/dev/shm/workspace/parallel/6/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.match (/dev/shm/workspace/parallel/6/kibana/packages/kbn-expect/expect.js:354:8)
    at Context.<anonymous> (test/reporting_api_integration/reporting_and_security/network_policy.ts:48:31)
    at Object.apply (/dev/shm/workspace/parallel/6/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:84:16)

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
fleet 963.8KB 963.8KB -2.0B

Distributable file count

id before after diff
default 47141 47901 +760

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
data 1002.5KB 1003.1KB +597.0B
fleet 376.8KB 376.5KB -350.0B
total +247.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@mattkime mattkime changed the title [index patterns] Keep saved object field list until field caps provides fields [index patterns] Fleep app - Keep saved object field list until field caps provides fields Dec 20, 2020
@mattkime mattkime merged commit c129f93 into elastic:master Dec 20, 2020
mattkime added a commit to mattkime/kibana that referenced this pull request Dec 20, 2020
mattkime added a commit to mattkime/kibana that referenced this pull request Dec 20, 2020
… caps provides fields (elastic#85370)

# Conflicts:
#	src/plugins/data/public/public.api.md
mattkime added a commit that referenced this pull request Dec 20, 2020
mattkime added a commit that referenced this pull request Dec 20, 2020
gmmorris added a commit to gmmorris/kibana that referenced this pull request Dec 21, 2020
* master: (48 commits)
  Fix request with disabled aggregation (elastic#85696)
  [Security Solution][Detections][Threshold Rules] Threshold Rule Bug Fixes (elastic#84918)
  Removed a possibility to define two different names for Alert types on API and UI level. (elastic#86236)
  Bump Node.js from version 14.15.2 to 14.15.3 (elastic#86593)
  [index patterns] Fleep app - Keep saved object field list until field caps provides fields (elastic#85370)
  [Security Solutions] fix timeline tabs + layout (elastic#86581)
  Upgrade to hapi version 20 (elastic#85406)
  App Services: Remove remaining uiActions, expressions, data, embeddable circular dependencies. (elastic#82791)
  Rename chartLibrary setting to legacyChartsLibrary (elastic#86529)
  [CI] TeamCity updates (elastic#85843)
  [Maps] Use Json for mvt-tests (elastic#86492)
  [Rollup Jobs] Added autofocus to cron editor (elastic#86324)
  [Monitoring][Alerting] CCR read exceptions alert (elastic#85908)
  [CI] Bump memory for main CI workers (elastic#86541)
  Explicitly set Elasticsearch heap size during CI and local development (elastic#86513)
  [App Search] Updates to results on the documents view (elastic#86181)
  [Discover] Change default sort handling  (elastic#85561)
  [App Search] Convert DocumentCreationModal to DocumentCreationFlyout (elastic#86508)
  [App Search] Sample Engines should have access to the Crawler (elastic#86502)
  Fixed duplication of create new modal (elastic#86489)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
blocker Feature:Data Views Data Views code and UI - index patterns before 8.0 release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.11.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants