Rename GenAI to OpenAI Preview #12
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Documentation preview: |
stephmilovic
commented
Sep 26, 2023
| @@ -13,7 +13,7 @@ export const OPEN_AI_TITLE = i18n.translate( | |||
| defaultMessage: 'OpenAI', | |||
| } | |||
| ); | |||
| export const GEN_AI_CONNECTOR_ID = '.gen-ai'; | |||
| export const OPEN_AI_CONNECTOR_ID = '.gen-ai'; | |||
There was a problem hiding this comment.
Keeping .gen-ai so migrations do not break
stephmilovic
commented
Sep 26, 2023
| @@ -143,7 +143,7 @@ export const getDashboard = (dashboardId: string): SavedObject<DashboardAttribut | |||
| '475e8ca0-e78e-454a-8597-a5492f70dce3': { | |||
| columns: { | |||
| '0f9814ec-0964-4efa-93a3-c7f173df2483': { | |||
| label: 'GenAI Completion Tokens', | |||
| label: 'OpenAI Completion Tokens', | |||
| dataType: 'number', | |||
| operationType: 'sum', | |||
| sourceField: 'kibana.action.execution.gen_ai.usage.completion_tokens', | |||
There was a problem hiding this comment.
keeping these fields the same
Closed
…lastic#167111) ## Summary Save to library action should not be present in ES|QL panels. For now we only allow by value embeddables and we don't want them to be edited/created in Lens editor <img width="915" alt="image" src="https://github.com/elastic/kibana/assets/17003240/d6e5e8a7-459f-4b1d-8461-2863b45db882"> ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
The `chi2test` utils so fare were only used within data comparison view. We plan to use it with other plugins, so moving it so a separate package in this PR. `SIGNIFICANCE_LEVELS` was updated to include some more digits.
## Summary Closes elastic#165227. Removes configurable app targets. Only the standard logs app is registered (Note: [Infra is disabled in serverless](elastic#165289) observability / security projects currently). --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Updates the text for the info callouts which are displayed when anomaly detection jobs are waiting for a node. @szabosteve would you mind having a review of the text? **Jobs list**  **Creating a new job**  Fixes elastic#165472 --------- Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>
…lastic#167319) When exporting an anomaly detection job, it would be useful if the original `created_by` property was not removed from the job config. Related to elastic#167021 (comment) Related PR elastic#88898
`v88.3.0`⏩`v88.5.0` closes elastic#151514 --- ## [`88.5.0`](https://github.com/elastic/eui/tree/v88.5.0) - Updated `EuiCallOut` with a new `onDismiss` prop ([elastic#7156](elastic/eui#7156)) - Added a new `renderCustomToolbar` prop to `EuiDataGrid`, which allows custom rendering of the toolbar. ([elastic#7190](elastic/eui#7190)) - Added a new `allowResetButton` prop to `toolbarVisibility.showDisplaySelector` of `EuiDataGrid`, which allows hiding the "Reset to default" button from the display settings popover. ([elastic#7190](elastic/eui#7190)) - Added a new `additionalDisplaySettings` prop to `toolbarVisibility.showDisplaySelector` of `EuiDataGrid`, which allows rendering extra settings inside the display settings popover. ([elastic#7190](elastic/eui#7190)) - Updated `EuiDataGrid`'s toolbar display settings button icon ([elastic#7190](elastic/eui#7190)) - Updated `EuiTextTruncate` with significantly improved iteration performance. Removed `measurementRenderAPI` prop, as `EuiTextTruncation` now only uses more performant canvas render API ([elastic#7210](elastic/eui#7210)) - Updated `EuiPopover` with a new configurable `repositionToCrossAxis` prop ([elastic#7211](elastic/eui#7211)) - Updated `EuiDatePicker` to support `compressed` input styling ([elastic#7218](elastic/eui#7218)) - Added `gradient` and `palette` icon glyphs. ([elastic#7220](elastic/eui#7220)) **Bug fixes** - Fixed `EuiPopover`'s missing animations on popover close ([elastic#7211](elastic/eui#7211)) - Fixed `EuiInputPopover` anchoring to the wrong side and missing shadows on smaller screens ([elastic#7211](elastic/eui#7211)) - Fixed `EuiSuperDatePicker` icon spacing on the quick select button ([elastic#7217](elastic/eui#7217)) - Fixed a missing type in `EuiMarkdownEditor`'s default processing plugins ([elastic#7221](elastic/eui#7221)) ## [`88.4.1`](https://github.com/elastic/eui/tree/v88.4.1) **Bug fixes** - Fixed missing `className`s on `EuiTextTruncate` ([elastic#7212](elastic/eui#7212)) - Fixed `title`s on `EuiComboBox` dropdown options to always be present ([elastic#7212](elastic/eui#7212)) - Fixed `EuiComboBox` truncation issues when search is an empty space ([elastic#7212](elastic/eui#7212)) ## [`88.4.0`](https://github.com/elastic/eui/tree/v88.4.0) - Updated `EuiComboBox` to allow configuring text truncation behavior via `truncationProps`. These props can be set on the entire combobox as well as on on individual dropdown options. ([elastic#7028](elastic/eui#7028)) - Updated `EuiInMemoryTable` with a new `searchFormat` prop (defaults to `eql`). When setting this prop to `text`, the built-in search bar will ignore EQL syntax and allow searching for plain strings with special characters and symbols. ([elastic#7175](elastic/eui#7175)) **Bug fixes** - `EuiComboBox` now always shows the highlighted search text, even on truncated text ([elastic#7028](elastic/eui#7028)) - Fixed missing i18n in `EuiSearchBar`'s default placeholder and aria-label text ([elastic#7175](elastic/eui#7175)) - Fixed the inline compressed styles of `EuiDescriptionListTitle` to use a taller line-height for readability ([elastic#7185](elastic/eui#7185)) - Fixed `EuiComboBox` to correctly truncate selected items when displayed as pills and plain text ([elastic#7193](elastic/eui#7193)) **Accessibility** - Added `aria-current` attribute to `EuiTablePagination` ([elastic#7186](elastic/eui#7186)) **CSS-in-JS conversions** - Converted `EuiDroppable` and `EuiDraggable` to Emotion; Removed `$euiDragAndDropSpacing` Sass variables ([elastic#7187](elastic/eui#7187)) --------- Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com> Co-authored-by: Jan Monschke <jan.monschke@elastic.co> Co-authored-by: Thomas Watson <watson@elastic.co>
…s_roles`, etc) to be overwritten via `yarn es serverless` command (elastic#167087) ## Summary ### `@kbn/es` package - Introduces `--resources` option to the `es serverless` command - Allows for [`serverless_resources`](https://github.com/elastic/kibana/tree/main/packages/kbn-es/src/serverless_resources) to be overwritten ### Security Solution Plugin - Added customized `users`, `users_roles` and `roles.yml` files that includes: - Updated `roles.yml` file (sync'ed with project controller version) - Updated `users` / `users_roles` file that includes one one per security project role (name the same as the role) - New CLI script - `node x-pack/plugins/security_solution/scripts/endpoint/start_es_serverless_with_security_users.js` - that will start ES in serverless mode and inject the customized `--resources` into the command - Any existing option that can be passed to `es` or `es serverless` can still continue to be passed to this new script. I will pass them along. closes elastic#167092 __
…167393) elastic/security-team#7588 To meet the above requirements, I believe it is sufficient to replace only the URL with one that directly points to the security integration instead of the integrations list. This change does not appear to affect the flow of guided onboarding in any way. https://github.com/elastic/kibana/assets/29123534/4d1aa943-2dce-4d43-bab1-d7cc024695f8
## Summary This PR removes the feature flag and enables the new index details page by default. The index details page was implemented in following PRs: - elastic#163521 - elastic#163955 - elastic#164741 - elastic#165027 - elastic#165038 - elastic#165456 In this PR we completely remove now obsolete code for the old index details flyout: react components and corresponding redux code. All related tests are updated and cleaned up. The config value for Index Management plugin `xpack.index_management.dev.enableIndexDetailsPage` is deprecated as unused and can be removed in v9.0. ### How to test 1. Start ES and Kibana with `yarn es snapshot` and `yarn start` 3. Navigate to Index Management and create an index 4. Click the index name in the table and check the tabs of the details page ### Screenshots #### Stateful Overview <img width="1387" alt="Screenshot 2023-09-27 at 14 41 57" src="https://github.com/elastic/kibana/assets/6585477/e58b15e7-d10c-4473-873c-d0f128392404"> Mappings <img width="1392" alt="Screenshot 2023-09-27 at 14 42 05" src="https://github.com/elastic/kibana/assets/6585477/441157cb-5a26-47c3-8da0-b4df51ebec5d"> Settings <img width="1385" alt="Screenshot 2023-09-27 at 14 42 13" src="https://github.com/elastic/kibana/assets/6585477/da66a2eb-1f21-44c1-9356-484c66caab88"> Statistics <img width="1380" alt="Screenshot 2023-09-27 at 14 42 22" src="https://github.com/elastic/kibana/assets/6585477/ec93d85c-e754-4c21-88ab-0124dc114fc9"> Error loading data <img width="1333" alt="Screenshot 2023-09-26 at 19 05 37" src="https://github.com/elastic/kibana/assets/6585477/fc1804b3-6aa0-4019-bae6-e7bb40113b28"> <img width="1327" alt="Screenshot 2023-09-26 at 19 06 07" src="https://github.com/elastic/kibana/assets/6585477/ca711697-cc74-4ba8-b17c-ec9b01f3026e"> <img width="1329" alt="Screenshot 2023-09-26 at 19 06 28" src="https://github.com/elastic/kibana/assets/6585477/0cb46b09-8542-452a-8845-40d060057e95"> <img width="1331" alt="Screenshot 2023-09-26 at 19 06 48" src="https://github.com/elastic/kibana/assets/6585477/87de8d3d-b6e5-4e8f-b27c-18a1c6e950d8"> Error saving index settings <img width="1332" alt="Screenshot 2023-09-26 at 19 07 31" src="https://github.com/elastic/kibana/assets/6585477/e6e4b3d0-c237-4d0a-995a-4562bc78f88e"> ### Serverless Overview <img width="1336" alt="Screenshot 2023-09-26 at 19 51 47" src="https://github.com/elastic/kibana/assets/6585477/6c76c23b-4be6-4ab3-ae1d-c7ae751e100d"> Mappings <img width="1336" alt="Screenshot 2023-09-26 at 19 23 51" src="https://github.com/elastic/kibana/assets/6585477/625fa703-506f-4389-9df0-86441a655074"> Settings <img width="1332" alt="Screenshot 2023-09-26 at 19 24 02" src="https://github.com/elastic/kibana/assets/6585477/c496ab09-f2db-4c1b-9fb6-1e9b64b1c142"> # Release note Index details can now be viewed on a new index details page in Index Management. <img width="1387" alt="Screenshot 2023-09-27 at 14 41 57" src="https://github.com/elastic/kibana/assets/6585477/b90c706d-8b15-49e4-8f6a-cb66f3ed1822"> --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
…#161512) ## Summary Migrates the deprecated components in Logstash pipelines view based on [the gude](https://elastic.github.io/eui/#/layout/page-components). A list of deprecated components in Logstash pipelines can be found [here](https://github.com/search?q=repo%3Aelastic%2Fkibana+%2FEuiPage%5Ba-zA-Z%5D%2B_Deprecated%2F+path%3A%2F%5Ex-pack%5C%2Fplugins%5C%2Flogstash%2F&type=code). ### Tests Opened a dev tools and proceeded manual tests on local dev env (based on [local env setup guide](https://docs.elastic.dev/kibana-dev-docs/getting-started/setup-dev-env)): - create pipeline(s) - check list of pipeline(s) - delete pipeline(s) - check alerts when deleting the pipeline - check Elasticsearch .logstash_pipelines index (`GET _logstash/pipeline`) to check the data shape which Logstash polls and uses. - Closes: elastic#161417 ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) ~~- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials~~ - [x ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ~~- [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))~~ - [x] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) ~~- [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)~~ - [x] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [x] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) ### Risk Matrix Delete this section if it is not applicable to this PR. Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release. When forming the risk matrix, consider some of the following examples and how they may potentially impact the change: | Risk | Probability | Severity | Mitigation/Notes | |---------------------------|-------------|----------|-------------------------| | Multiple Spaces—unexpected behavior in non-default Kibana Space. | Low | High | Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces. | | Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. | High | Low | Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure. | | Code should gracefully handle cases when feature X or plugin Y are disabled. | Medium | High | Unit tests will verify that any feature flag or plugin combination still results in our service operational. | | [See more potential risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) | ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary We're breaking elastic#166813 up into smaller PRs in the interest of getting PRs through sooner for type fixes. These are the changes for `x-pack/test`. --------- Co-authored-by: Thomas Watson <watson@elastic.co> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Alex Szabo <alex.szabo@elastic.co>
…lastic#167126) Fixes elastic#164366 ### Acceptance Criteria - use `toMountPoint` from `@kbn/react-kibana-mount` for [kibana/x-pack/plugins/observability/public/pages/overview/components/header_menu/header_menu_portal.tsx](https://github.com/elastic/kibana/blob/fcf838e1f389cae6fcaeb9f7bce713123a01c939/x-pack/plugins/observability/public/pages/overview/components/header_menu/header_menu_portal.tsx#L10) --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
…ailable via `server` (elastic#167465) Follow up to elastic#167237. Part of elastic#167467. We plan to reuse some of the queries log pattern analysis does to use via log rate analysis too. Log pattern analysis mostly does queries from the client side, late rate analysis has its own API endpoint and does ES queries via Kibana server. In preparation for the use via log rate analysis, this moves the code we need to have available server side for log rate analysis to the `common` area of the plugin so it can be used both on server/client.
…oper wait logic (elastic#166936) ## Summary Hopefully closes elastic#167104 closes elastic#167130 closes elastic#167100 closes elastic#167013 closes elastic#166964 Fixing a few issues with login/logout: 1. Failed to login in "before" hook <img width="1336" alt="Screenshot 2023-09-25 at 12 37 45" src="https://github.com/elastic/kibana/assets/10977896/e3b2830e-7b0d-4467-9b90-261b385bf71e"> My theory is that we are loading `/login` route too soon while log out was not completed yet. When we navigate to `https://localhost:5620/logout` there are multiple url re-directions with final page being Cloud login form. This PR makes sure we wait for this form to be displayed + 2500 ms extra to avoid "immediate" /login navigation 2. Failed login on MKI: Updating login via UI for serverless to pass password valid for deployment: currently FTR uses `changeme` for both Kibana CI & MKI. 3. ES activate user profile call returning 500 We saw some login failures that are preceded with the following logs: ``` [00:03:27] │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000 [00:03:27] │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000 [00:03:27] │ debg Find.waitForDeletedByCssSelector('.kibanaWelcomeLogo') with timeout=10000 [00:03:27] │ proc [kibana] [2023-09-19T07:08:26.126+00:00][INFO ][plugins.security.routes] Logging in with provider "basic" (basic) [00:03:27] │ info [o.e.x.s.s.SecurityIndexManager] [ftr] security index does not exist, creating [.security-profile-8] with alias [.security-profile] [00:03:27] │ proc [kibana] [2023-09-19T07:08:26.140+00:00][ERROR][plugins.security.user-profile] Failed to activate user profile: {"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [1] shards, but this cluster currently has [27]/[27] maximum normal shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [1] shards, but this cluster currently has [27]/[27] maximum normal shards open;"},"status":400}. [00:03:27] │ proc [kibana] [2023-09-19T07:08:26.140+00:00][ERROR][http] 500 Server Error [00:03:27] │ warn browser[SEVERE] http://localhost:5620/internal/security/login - Failed to load resource: the server responded with a status of 500 (Internal Server Error) ``` User activation happens during `POST internal/security/login` call to Kibana server. ~~The only improvement that we can do from FTR perspective is to call this end-point via API to makes sure user is activated and only after proceed with UI login.~~ While working on issue #4 and talking to @jeramysoucy I believe retrying login via UI will work here as well. We are checking if we are still on login page (similar to incorrect password login), waiting 2500 ms and pressing login button again. 4. Failed to login with Kibana reporting UNEXPECTED_SESSION_ERROR and been re-directed to Cloud login page ``` proc [kibana] [2023-09-25T11:35:12.794+00:00][INFO ][plugins.security.authentication] Authentication attempt failed: UNEXPECTED_SESSION_ERROR ``` Temporary solution is to retry login from scratch (navigation to Kibana login page & re-login ) Flaky-test-runner for functional obtl tests 50x https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3215 This PR is not fixing random 401 response when user navigates to some apps with direct url --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
…lastic#167470) ## Summary This PR fix type check problems caused by the inclusion of a new item in the AlertStatus union type
… eslint rule. (elastic#167317) Implements elastic#153108. This enables the `@kbn/telemetry/event_generating_elements_should_be_instrumented` eslint rule for the `aiops` plugin to enforce `data-test-subj` attributes on actionable EUI components so they are auto-instrumented by telemetry. The ids were first auto-created using `node scripts/eslint --fix x-pack/plugins/aiops` and then adapted.
) ## Summary I had to change `waitForRender` since `page.waitForFunction` tries to run a script on page and it is not working due to CSP settings on Cloud. Instead of injecting a script, we use a classical API to find elements/attributes in the DOM. Since `PUT /internal/core/_settings` is merged in 8.11.0, journeys run on Cloud with on-fly labels update is supported starting deployments 8.11.0+. I added error message for 404 code just in case someone runs it on earlier version. `many_fields_discover` journey was update since on Cloud the data view used by scenario is not selected by default. How it works: Create a deployment with QAF and re-configure it for journey run: ``` export EC_DEPLOYMENT_NAME=my-run-8.11 qaf elastic-cloud deployments create --stack-version 8.11.0-SNAPSHOT --environment staging --region gcp-us-central1 qaf elastic-cloud deployments configure-for-performance-journeys ``` Run any journey, e.g. many_fields_discover ``` TEST_CLOUD=1 TEST_ES_URL=https://username:pswd@es_url:443 TEST_KIBANA_URL=https://username:pswd@kibana-ur_url node scripts/functional_test_runner --config x-pack/performance/journeys/many_fields_discover.ts ``` You should see a log about labels being updated: ``` Updating telemetry & APM labels: {"testJobId":"local-a3272047-6724-44d1-9a61-5c79781b06a1","testBuildId":"local-d8edbace-f441-4ba9-ac83-5909be3acf2a","journeyName":"many_fields_discover","ftrConfig":"x-pack/performance/journeys/many_fields_discover.ts"} ``` And then able to find APM logs for the journey in [Ops](https://kibana-ops-e2e-perf.kb.us-central1.gcp.cloud.es.io:9243/app/apm/services?comparisonEnabled=true&environment=ENVIRONMENT_ALL&kuery=labels.testJobId%20%3A%20%22local-d79a878c-cc7a-423b-b884-c9b6b1a8d781%22&latencyAggregationType=avg&offset=1d&rangeFrom=now-24h%2Fh&rangeTo=now&serviceGroup=&transactionType=request) cluster
Fixing bug introduced by a merge clash between elastic#166622 and elastic#166651 The `hideFrozenDataTierChoice` prop is no longer available in the `FullTimeRangeSelector` component and so the transforms plugin needs to set `showFrozenDataTierChoice` in the data picker context.
Closes elastic#156245 One test agent went from offline to inactive due to inactivity timeout and broke the tests, instead changed the `last_checkin` to now-6m so it stays in offline state.
## 📓 Summary Closes elastic#166848 This work adds a new tab to navigate Data View from the Log Explorer selector. In this first iteration, when the user selects a data view, we move into discovering preselecting and loading the data view of choice. **N.B.**: this recording is made on a setup where I have no installed integrations, so having the no integrations panel is the expected behaviour. https://github.com/elastic/kibana/assets/34506779/e8d1f622-86fb-4841-b4cc-4a913067d2cc ## Updated selector state machine <img width="1492" alt="Screenshot 2023-09-22 at 12 15 44" src="https://github.com/elastic/kibana/assets/34506779/c563b765-6c6c-41e8-b8cd-769c518932c3"> ## New DataViews state machine <img width="995" alt="Screenshot 2023-09-22 at 12 39 09" src="https://github.com/elastic/kibana/assets/34506779/e4e43343-c915-42d8-8660-a2ee89f8d595"> --------- Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
closes: elastic#166428 ## Summary This PR removes code that is no longer needed after replacing the Node Details View for Host with the Asset Details. ### TSVB The TSVB files were apparently only used to display charts in the Node Details view. Due to the Asset Details using Lens to power the charts, corresponding `host` TSVB formulas and configs are no longer needed. Therefore, `host*`, `hostK8s*`, and `hostDocker*` (the latter appears to have never been used) have been removed. Additionally, `aws*` from `required_metrics` was also removed because it was host-specific. ### FE Components The main change is in the `useMetadata` hook. I have changed the hook signature, making `requiredMetrics` optional. This parameter is used to process additional filtering and is only used in asset types that the old Node Details page supports. Not passing it is not expected to break other places that depend on this hook. ### Server Removing TSVB files has a direct impact on the `api/metrics/node_details` endpoint. This endpoint is only used to provide data to the Node Details page. It returns a 400 error if an invalid metric is passed - which will be the case for hosts **Example Request:** ```json POST kbn:api/metrics/node_details { "metrics": [ "hostK8sCpuCap", "hostSystemOverview" ], "nodeId": "gke-release-oblt-release-oblt-pool-c4163099-bcaj", "nodeType": "host", "timerange": { "from": 1695210522045, "to": 1695214122045, "interval": ">=1m" }, "cloudId": "6106013995483209805", "sourceId": "default" } ``` **Response:** ```json { "statusCode": 400, "error": "Bad Request", "message": "Failed to validate: \n in metrics/0: \"hostK8sCpuCap\" does not match expected type \"podOverview\" | \"podCpuUsage\" | \"podMemoryUsage\" | \"podLogUsage\" | \"podNetworkTraffic\" | \"containerOverview\" | \"containerCpuKernel\" | \"containerCpuUsage\" | \"containerDiskIOOps\" | \"containerDiskIOBytes\" | \"containerMemory\" | \"containerNetworkTraffic\" | \"containerK8sOverview\" | \"containerK8sCpuUsage\" | \"containerK8sMemoryUsage\" | \"nginxHits\" | \"nginxRequestRate\" | \"nginxActiveConnections\" | \"nginxRequestsPerConnection\" | \"awsEC2CpuUtilization\" | \"awsEC2NetworkTraffic\" | \"awsEC2DiskIOBytes\" | \"awsS3TotalRequests\" | \"awsS3NumberOfObjects\" | \"awsS3BucketSize\" | \"awsS3DownloadBytes\" | \"awsS3UploadBytes\" | \"awsRDSCpuTotal\" | \"awsRDSConnections\" | \"awsRDSQueriesExecuted\" | \"awsRDSActiveTransactions\" | \"awsRDSLatency\" | \"awsSQSMessagesVisible\" | \"awsSQSMessagesDelayed\" | \"awsSQSMessagesSent\" | \"awsSQSMessagesEmpty\" | \"awsSQSOldestMessage\" | \"custom\"\n in metrics/1: \"hostSystemOverview\" does not match expected type \"podOverview\" | \"podCpuUsage\" | \"podMemoryUsage\" | \"podLogUsage\" | \"podNetworkTraffic\" | \"containerOverview\" | \"containerCpuKernel\" | \"containerCpuUsage\" | \"containerDiskIOOps\" | \"containerDiskIOBytes\" | \"containerMemory\" | \"containerNetworkTraffic\" | \"containerK8sOverview\" | \"containerK8sCpuUsage\" | \"containerK8sMemoryUsage\" | \"nginxHits\" | \"nginxRequestRate\" | \"nginxActiveConnections\" | \"nginxRequestsPerConnection\" | \"awsEC2CpuUtilization\" | \"awsEC2NetworkTraffic\" | \"awsEC2DiskIOBytes\" | \"awsS3TotalRequests\" | \"awsS3NumberOfObjects\" | \"awsS3BucketSize\" | \"awsS3DownloadBytes\" | \"awsS3UploadBytes\" | \"awsRDSCpuTotal\" | \"awsRDSConnections\" | \"awsRDSQueriesExecuted\" | \"awsRDSActiveTransactions\" | \"awsRDSLatency\" | \"awsSQSMessagesVisible\" | \"awsSQSMessagesDelayed\" | \"awsSQSMessagesSent\" | \"awsSQSMessagesEmpty\" | \"awsSQSOldestMessage\" | \"custom\"" } ``` ### How to Test - Start a local Kibana instance pointing to an oblt cluster. - Navigate to `Infrastructure`. - Try different asset types and navigate to the Node Details view. --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
) Fixes elastic#162299 Fixes elastic#162540 ## Summary This PR unskips the custom threshold rule executor unit test and removes the warning implementation from the BE. ## 🧪 How to test - Create a custom threshold rule, it should work as before. (Warning implementation logic was already removed from FE; this PR only removes the BE implementation.)
## Summary This PR introduces the new color mapping feature into Lens. The color mapping feature is introduced as a standalone sharable component available from `@kbn/coloring`. The [README.md](https://github.com/elastic/kibana/blob/ddd216457d66912de43c7688ae99044c2c34bbd2/packages/kbn-coloring/src/shared_components/color_mapping/README.md) file describes the components and the logic behind it. The Color Mapping component is also connected to Lens and is available in the following charts: - XY (you can specify the mappings from a breakdown dimension - Partition (you can specify the mappings from the main slice/group by dimension) - Tag cloud (you can specify the mappings from the tags dimension) This MVP feature will be released under the Tech Preview flag. This PR needs to prove the user experience and the ease of use. UI styles, design improvements and embellishments will be released in subsequent PRs. The current MVP-provided palettes are just a placeholder. I'm coordinating with @gvnmagni for a final set of palettes. close elastic#155037 close elastic#6480 fix elastic#28618 fix elastic#96044 fix elastic#101942 fix elastic#112839 fix elastic#116634 ## Release note This feature introduces the ability to change and map colors to break down dimensions in Lens. The feature provides an improved way to specify colors and their association with categories by giving the user a predefined set of color choices or customized one that drives the user toward a correct color selection. It provides ways to pick new colors and generate gradients. This feature is in Tech Preview and is enabled by default on every new visualization but can be turned off at will. 
…67226) ## Summary Adds an error handler block to help debug test setup errors during fleet agent setup for endpoint API ftr tests. Also adds missing header API version. fixes elastic/issues/151854 **flaky ftr test runners** - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3223 x 150 (without header) - (1 fail) - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3224 x 50 (all 50 pass) - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3228 x 100 (2 fails) - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3231 x 100 ( 1 fail) - https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3241 x 50 ( 2 fails) ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
…tic#167485) This reverts commit d61a5a0 from elastic#166565 ## Summary Summarize your PR. If it involves visual changes include a screenshot or gif. ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) ### Risk Matrix Delete this section if it is not applicable to this PR. Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release. When forming the risk matrix, consider some of the following examples and how they may potentially impact the change: | Risk | Probability | Severity | Mitigation/Notes | |---------------------------|-------------|----------|-------------------------| | Multiple Spaces—unexpected behavior in non-default Kibana Space. | Low | High | Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces. | | Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. | High | Low | Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure. | | Code should gracefully handle cases when feature X or plugin Y are disabled. | Medium | High | Unit tests will verify that any feature flag or plugin combination still results in our service operational. | | [See more potential risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) | ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
…#167246) Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
## 📓 Summary When retrieving the CPU stats for containerized (or non-container) clusters, we were not considering a scenario where the user could run in a cgroup but without limits set. These changes re-write the conditions to determine whether we allow treating limitless containers as non-containerized, covering the case where a user run in a cgroup and for some reason hasn't set the limit. ## Testing > Taken from elastic#159351 since it reproduced the same behaviours There are 3 main states to test: No limit set but Kibana configured to use container stats. Limit changed during lookback period (to/from real value, to/from no limit). Limit set and CPU usage crossing threshold and then falling down to recovery **Note: Please also test the non-container use case for this rule to ensure that didn't get broken during this refactor** **1. Start Elasticsearch in a container without setting the CPU limits:** ``` docker network create elastic docker run --name es01 --net elastic -p 9201:9200 -e xpack.license.self_generated.type=trial -it docker.elastic.co/elasticsearch/elasticsearch:master-SNAPSHOT ``` (We're using `master-SNAPSHOT` to include a recent fix to reporting for cgroup v2) Make note of the generated password for the `elastic` user. **2. Start another Elasticsearch instance to act as the monitoring cluster** **3. Configure Kibana to connect to the monitoring cluster and start it** **4. Configure Metricbeat to collect metrics from the Docker cluster and ship them to the monitoring cluster, then start it** Execute the below command next to the Metricbeat binary to grab the CA certificate from the Elasticsearch cluster. ``` docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt . ``` Use the `elastic` password and the CA certificate to configure the `elasticsearch` module: ``` - module: elasticsearch xpack.enabled: true period: 10s hosts: - "https://localhost:9201" username: "elastic" password: "PASSWORD" ssl.certificate_authorities: "PATH_TO_CERT/http_ca.crt" ``` **5. Configure an alert in Kibana with a chosen threshold** OBSERVE: Alert gets fired to inform you that there looks to be a misconfiguration, together with reporting the current value for the fallback metric (warning if the fallback metric is below threshold, danger is if is above). **6. Set limit** First stop ES using `docker stop es01`, then set the limit using `docker update --cpus=1 es01` and start it again using `docker start es01`. After a brief delay you should now see the alert change to a warning about the limits having changed during the alert lookback period and stating that the CPU usage could not be confidently calculated. Wait for change event to pass out of lookback window. **7. Generate load on the monitored cluster** [Slingshot](https://github.com/elastic/slingshot) is an option. After you clone it, you need to update the `package.json` to match [this change](https://github.com/elastic/slingshot/blob/8bfa8351deb0d89859548ee5241e34d0920927e5/package.json#L45-L46) before running `npm install`. Then you can modify the value for `elasticsearch` in the `configs/hosts.json` file like this: ``` "elasticsearch": { "node": "https://localhost:9201", "auth": { "username": "elastic", "password": "PASSWORD" }, "ssl": { "ca": "PATH_TO_CERT/http_ca.crt", "rejectUnauthorized": false } } ``` Then you can start one or more instances of Slingshot like this: `npx ts-node bin/slingshot load --config configs/hosts.json` **7. Observe the alert firing in the logs** Assuming you're using a connector for server log output, you should see a message like below once the threshold is breached: ``` `[2023-06-13T13:05:50.036+02:00][INFO ][plugins.actions.server-log] Server log: CPU usage alert is firing for node e76ce10526e2 in cluster: docker-cluster. [View node](/app/monitoring#/elasticsearch/nodes/OyDWTz1PS-aEwjqcPN2vNQ?_g=(cluster_uuid:kasJK8VyTG6xNZ2PFPAtYg))` ``` The alert should also be visible in the Stack Monitoring UI overview page. At this point you can stop Slingshot and confirm that the alert recovers once CPU usage goes back down below the threshold. **8. Stop the load and confirm that the rule recovers.** --------- Co-authored-by: Marco Antonio Ghiani <marcoantonio.ghiani@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
) We realized that our 'real agent' tests in DW Cypress do not run locally, because FS is not being created (normally created in a before:run task). The `before:run` task in cypress was not run. According to the [docs](https://docs.cypress.io/api/plugins/before-run-api), it requires an experimental flag `experimentalInteractiveRunEvents` enabled and it was removed in [here](https://github.com/elastic/kibana/pull/166757/files#diff-c2e90a076b73f17a80dcb5bfaf96f143d4d0de4dfaa68f5085608621a86c612dL33)
…otations (elastic#167483) ## Summary Fixes elastic#167408 This PR reduces the amount of "visualization modifiers" in the specific case where ALL annotations in a layer are manual. The `Ignore global filters` will still be shown if at least one query-based annotation is defined in the layer. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
…tic#166349) ## Summary Part of elastic#165958 Replaces usage of the deprecated `alertFactory` with the new alerts client and adds alerts-as-data integration for Anomaly Detection alerting rule type. Alert instances are stored in `.alerts-ml.anomaly-detection.alerts-default` index and extends the common `AlertSchema`. <details> <summary>Result mappings</summary> ```json { ".internal.alerts-ml.anomaly-detection.alerts-default-000001": { "mappings": { "dynamic": "false", "_meta": { "namespace": "default", "kibana": { "version": "8.11.0" }, "managed": true }, "properties": { "@timestamp": { "type": "date" }, "event": { "properties": { "action": { "type": "keyword" }, "kind": { "type": "keyword" } } }, "kibana": { "properties": { "alert": { "properties": { "action_group": { "type": "keyword" }, "anomaly_score": { "type": "double" }, "anomaly_timestamp": { "type": "date" }, "case_ids": { "type": "keyword" }, "duration": { "properties": { "us": { "type": "long" } } }, "end": { "type": "date" }, "flapping": { "type": "boolean" }, "flapping_history": { "type": "boolean" }, "instance": { "properties": { "id": { "type": "keyword" } } }, "is_interim": { "type": "boolean" }, "job_id": { "type": "keyword" }, "last_detected": { "type": "date" }, "maintenance_window_ids": { "type": "keyword" }, "reason": { "type": "keyword" }, "rule": { "properties": { "category": { "type": "keyword" }, "consumer": { "type": "keyword" }, "execution": { "properties": { "uuid": { "type": "keyword" } } }, "name": { "type": "keyword" }, "parameters": { "type": "flattened", "ignore_above": 4096 }, "producer": { "type": "keyword" }, "revision": { "type": "long" }, "rule_type_id": { "type": "keyword" }, "tags": { "type": "keyword" }, "uuid": { "type": "keyword" } } }, "start": { "type": "date" }, "status": { "type": "keyword" }, "time_range": { "type": "date_range", "format": "epoch_millis||strict_date_optional_time" }, "top_influencers": { "type": "nested", "dynamic": "false", "properties": { "influencer_field_name": { "type": "keyword" }, "influencer_field_value": { "type": "keyword" }, "influencer_score": { "type": "double" }, "initial_influencer_score": { "type": "double" }, "is_interim": { "type": "boolean" }, "job_id": { "type": "keyword" }, "timestamp": { "type": "date" } } }, "top_records": { "type": "nested", "dynamic": "false", "properties": { "actual": { "type": "double" }, "by_field_name": { "type": "keyword" }, "by_field_value": { "type": "keyword" }, "detector_index": { "type": "integer" }, "field_name": { "type": "keyword" }, "function": { "type": "keyword" }, "initial_record_score": { "type": "double" }, "is_interim": { "type": "boolean" }, "job_id": { "type": "keyword" }, "over_field_name": { "type": "keyword" }, "over_field_value": { "type": "keyword" }, "partition_field_name": { "type": "keyword" }, "partition_field_value": { "type": "keyword" }, "record_score": { "type": "double" }, "timestamp": { "type": "date" }, "typical": { "type": "double" } } }, "url": { "type": "keyword", "index": false, "ignore_above": 2048 }, "uuid": { "type": "keyword" }, "workflow_status": { "type": "keyword" }, "workflow_tags": { "type": "keyword" } } }, "space_ids": { "type": "keyword" }, "version": { "type": "version" } } }, "tags": { "type": "keyword" } } } } } ``` </details> ### Checklist - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
…xecution (elastic#166283) resolves: elastic#158403 When conflicts are detected while updating alert docs after a rule runs, we'll try to resolve the conflict by `mget()`'ing the alert documents again, to get the updated OCC info `_seq_no` and `_primary_term`. We'll also get the current versions of "ad-hoc" updated fields (which caused the conflict), like workflow status, case assignments, etc. And then attempt to update the alert doc again, with that info, which should get it back up-to-date. Note that the rule registry was not touched here. During this PR's development, I added the retry support to it, but then my function tests were failing because there were never any conflicts happening. Turns out rule registry mget's the alerts before it updates them, to get the latest values. So they won't need this fix. It's also not clear to me if this can be exercised in serverless, since it requires the use of an alerting framework based AaD implementation AND the ability to ad-hoc update alerts. I think this can only be done with Elasticsearch Query and Index Threshold, and only when used in metrics scope, so it will show up in the metrics UX, which is where you can add the alerts to the case. ## manual testing It's hard! I've seen the conflict messages before, but it's quite difficult to get them to go off whenever you want. The basic idea is to get a rule that uses alerting framework AAD (not rule registry, which is not affected the same way with conflicts (they mget alerts right before updating them), set it to run on a `1s` interval, and probably also configure TM to run a `1s` interval, via the following configs: ``` xpack.alerting.rules.minimumScheduleInterval.value: "1s" xpack.task_manager.poll_interval: 1000 ``` You want to get the rule to execute often and generate a lot of alerts, and run for as long as possible. Then while it's running, add the generated alerts to cases. Here's the EQ rule definition I used:  I selected the alerts from the o11y alerts page, since you can't add alerts to cases from the stack page. Hmm. :-). Sort the alert list by low-high duration, so the newest alerts will be at the top. Refresh, select all the rules (set page to show 100), then add to case from the `...` menu. If you force a conflict, you should see something like this in the Kibana logs: ``` [ERROR] [plugins.alerting] Error writing alerts: 168 successful, 100 conflicts, 0 errors: [INFO ] [plugins.alerting] Retrying bulk update of 100 conflicted alerts [INFO ] [plugins.alerting] Retried bulk update of 100 conflicted alerts succeeded ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Will reopen against
mainonce elastic#166662 is merged