wip #14
Closed
wip #14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ymao1
force-pushed
the
alerting/flatten-aad
branch
from
5752c3f to
7255e9a
Compare
…-ref HEAD~1..HEAD --fix'
…:ymao1/kibana into alerting/handle-unflattened-esquery-docs
…-unflattened-esquery-docs
…c#167392) ## Summary This is hopefully the last batch of typescript issues to be fixed, related to elastic#166813. It's also re-enabling full typecheck, with this, we should be back in a clean, typechecked main branch. Blocked by elastic#167428 --------- Co-authored-by: Brad White <Ikuni17@users.noreply.github.com> Co-authored-by: Brad White <brad.white@elastic.co> Co-authored-by: Thomas Watson <watson@elastic.co> Co-authored-by: Patryk Kopyciński <contact@patrykkopycinski.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
…lastic#167208) ## Summary Updates the exceptions flyout UI `match_any` operator to accept numerous duplicate values that differ in case. Prior to this change, a user could not add a field value of `foo` and `FOO` - the UI would display that the value is a duplicate. We now will allow this as exceptions are case sensitive and this is a necessary use case for the current exceptions behavior. Cypress tests and FTR tests are added.
## Summary Limit available spaces to 1 for risk engine <img width="1483" alt="Screenshot 2023-09-25 at 08 15 47" src="https://github.com/elastic/kibana/assets/7609147/94c088fb-55f5-436c-8c39-428fbb8e1e8c"> --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary Implement risk score engine telemetry Here we use 2 types of telemetry: Event base telemetry: - Risk execution success. With parameters `scoresWritten`, `taskCompletionTimeSeconds`, `isRunMoreThanInteval` - Risk execution error Usage telemetry: - `unique_user_risk_score_total` and `unique_host_risk_score_total` - Total amount from latest transform index for host and users - `unique_user_risk_score_day` and `unique_host_risk_score_day` - Last day amount from the latest transform index for host and users - `all_host_risk_scores_total` and `all_user_risk_scores_total` - Total amount from datastream for all risk executions for host and users - `all_host_risk_scores_total_day` and `all_user_risk_scores_total_day` - Last day amount from datastream for all risk executions for host and users - `all_risk_scores_index_size` and `unique_risk_scores_index_size` - sizes of datastream of all risk scores and latest transform index --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary Fixes a bug in ES|QL mode where you: - have a query of index pattern 1 and select some fields - change the query by using a different index pattern - the selected columns do not reset **BUG**  **NOW**  ### Checklist - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
…5149) Improves the display of long descriptions of transforms in the Transform management page and when editing the description in the transform wizard or edit flyout. Previously If there was a long description, the text would not be wrapped in the table on the management page, and it would not be possible to view the full text in the text input when editing. This PR adds line wrapping for the description column, and uses a text area for editing the text. Part of elastic#163147
…ed but hidden element instead of non existant (elastic#167594)
## Summary This PR skips a flaky test that has been failing on main.
## Summary I decided to remove the backticks suggestion as it was a copy paste from SQL and is not going to solve ES|QL related errors. It also creates confusion for many users. <img width="1677" alt="image" src="https://github.com/elastic/kibana/assets/17003240/fe2a4fcb-c3e0-4d87-8568-32e7525f70d5">
…#166916) elastic#166040 Inform user that there are no responses associated with and alert instead of returning `null`. 
…tic#167410) ## Summary Closes elastic#167387 Replaced using kibana version when deciding if agent upgrade is available (only in serverless, in stateful kibana version is still returned as an available version). To verify locally: - [to test stateless] add this to `kibana.dev.yml`: `xpack.fleet.internal.onlyAllowAgentUpgradeToKnownVersions: true` - extract the `agent_versions_list.json` to local kibana folder `~/kibana/x-pack/plugins/fleet/target` [agent_versions_list.json.zip](https://github.com/elastic/kibana/files/12739519/agent_versions_list.json.zip) - verify that upgrade available warnings still work if agent is < latest agent version (8.10.2) - when trying to upgrade agent, verify that the default version is the latest agent version, and 8.11 is not in the list Agent list: <img width="1475" alt="image" src="https://github.com/elastic/kibana/assets/90178898/f06b7bc8-97e6-4ff9-b872-736ede5e969a"> Upgrade available filter - 1 agent on latest version, 9 upgradeable: <img width="1314" alt="image" src="https://github.com/elastic/kibana/assets/90178898/4ff5ac02-903b-493b-94df-68b1b7ad6846"> Agent details: <img width="1512" alt="image" src="https://github.com/elastic/kibana/assets/90178898/3ff6e1d5-2ccc-4814-83e5-c4760ad63722"> Agent on latest version has disable `Upgrade agent` action: <img width="1322" alt="image" src="https://github.com/elastic/kibana/assets/90178898/f461dbf5-04e5-4bcc-8801-48c2b1a90225"> Bulk action with one agent that is not upgradeable (already on latest version), expected error: <img width="1597" alt="image" src="https://github.com/elastic/kibana/assets/90178898/8bfa46ae-6684-4748-9fca-e908c142b642"> ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary Closes elastic#167561 Closes elastic#167552 FT runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3254 ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
…tic#166942) ## Summary Fixes elastic#165569 Fixes elastic#166617 Fixes elastic#166618 Fixes elastic#166619 Fixes elastic#166620
Closes elastic#163282 ## Summary This PR: * Adds a `featureFlags.metricsExplorerEnabled` property to the Infra plugin config to enable and disable Metrics Explorer depending on the offering type * Prevents `MetricsExplorerViewsService` initialization for serveless based on the feature flag * Prevents creating Metrics Explorer frontend routes when in serverless * Prevents registration of the MetricsExplorerViews saved object when in serverless * Prevents initialization of the `metrics_explorer_views` API routes when in serverless **Trying to access Metrics Explorer in serverless** <img width="1829" alt="CleanShot 2023-09-22 at 12 59 35@2x" src="https://github.com/elastic/kibana/assets/793851/2b039925-0f0b-4c07-be29-bbe910de7a34"> **Trying to access views API** <img width="1829" alt="CleanShot 2023-09-22 at 13 00 00@2x" src="https://github.com/elastic/kibana/assets/793851/15269ec2-becd-4ee3-9b5e-d916df28a7b8"> **`infra/metrics_explorer` API still works as per ticket requirements** <img width="1829" alt="CleanShot 2023-09-22 at 13 00 06@2x" src="https://github.com/elastic/kibana/assets/793851/fb23f912-c6fd-46c8-9084-c17c51e5b064"> ## How to test * Checkout locally * Enable Infra in `serverless.oblt.yml`: `xpack.infra.enabled: true` * Run Kibana in serverless mode * Try accessing `/app/metrics/explorer` route and make sure it's not available * Make sure other Infra routes (`/app/metrics/inventory` and `/app/metrics/hosts`) still load as expected * In Kibana dev console make sure you get 404 for `GET kbn:/api/infra/metrics_explorer_views` * Also check that you don't see `metrics-explorer-view` saved object in the response for `GET kbn:/api/kibana/management/saved_objects/_allowed_types` * Run Kibana in non-serverless mode and make sure Metrics Explorer is accessible and works as usual --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
…es across Kibana (elastic#166937) - Resolves elastic#158173 Based on PoC elastic#166260 ## Summary This PR adds a new "Saved Query Management" privilege with 2 options: - `All` will override any per app privilege and will allow users to save queries from any Kibana page - `None` will default to per app privileges (backward-compatible option) <img width="600" alt="Screenshot 2023-09-21 at 15 26 25" src="https://github.com/elastic/kibana/assets/1415710/6d53548e-5c5a-4d6d-a86a-1e639cb77202"> ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co> Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Stratoula Kalafateli <efstratia.kalafateli@elastic.co>
## Summary This PR adds the project controller settings for serverless Elasticsearch to the Kibana serverless FTR configs. This gets our local setup closer to what we have in MKI. ### Details Project controller settings for ES per project: * [Observability](https://github.com/elastic/project-controller/blob/main/internal/project/observability/config/elasticsearch.yml) * [Search](https://github.com/elastic/project-controller/blob/main/internal/project/esproject/config/elasticsearch.yml) * [Security](https://github.com/elastic/project-controller/blob/main/internal/project/security/config/elasticsearch.yml)
Reverts elastic#167591 since the skipped test was fixed in elastic#167594
…alone and Elastic agents in Docker (elastic#165415) ## Summary Let's automate E2E against Serverless Changelog: - updated certs to include additional dns names we are using for testing locally, `host.docker.internal`, `es01` - updated certs generation README to include changes related to `openssl@3` - added new certs for Fleet server - added fleet-server service token - added support for `ca_trusted_fingerprint` in fleet preconfig  --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Tomasz Ciecierski <ciecierskitomek@gmail.com> Co-authored-by: Tomasz Ciecierski <tomasz.ciecierski@elastic.co> Co-authored-by: Kevin Logan <kevin.logan@elastic.co>
## Summary This PR introduces grouped common configs in serverless project FTR tests. ### Details * With increasing number of added tests, we're running into issues with test run time (configurations are stopped at 40 minutes) * This PR moves the inclusion of `common` tests from the projects' main `config.ts` file to `common_configs/config.group1.ts`, which can easily be extended * As part of that, `common` tests in `api_integration/test_suites` and `functional/test_suites` are re-organized to no longer contain a top level index file * Created sub-directories and index files where needed * This makes it easier to group `common` tests when including them in project config files ### Additional changes * Add README files to `x-pack/test_serverless/[api_integration|functional]/test_suites/common` * Rename `security` directory in `common` tests to `platform_security` to avoid confusion with the `security` project type * Include sample data test suite in an index file (this suite wasn't included so far and didn't run at all) and prepared it for actually working in serverless * it's still failing and should be fixed soon - skipped it for now
…ss (elastic#166942)" This reverts commit 4c1ca7e.
…tened alerts docs (elastic#167439) Resolves elastic#166946 ## Summary The rule registry has traditionally written out AAD docs with flattened keys, like ``` { "kibana.alert.rule.name": "test" } ``` The framework alerts client has been writing out AAD docs as objects, like ``` { "kibana": { "alert": { "rule": { "name": "test" } } } } ``` We've identified a few places where we're updating the docs where having this divergence makes things more difficult, so this is to switch the framework to writing flattened alert docs before onboarding more rule types. This PR is targeted for 8.11, which is also when we onboarded the index threshold rule type to FAAD. The only other rule type using FAAD to write docs is ES query, which landed in 8.10 so there will be a followup issue to handle the case of updating unflattened ES query AAD docs from 8.10 ## To Verify ### ES Query and Index Threshold AaD Create these rules that trigger alerts and verify that their AaD docs are written out as flattened. For the ES Query rule type, select a Metrics/Logs consumer and verify that they appear on the O11y alerts table. ### ML alerts ML alerts added in elastic#166349 looked like: <details> <summary>Unflattened</summary> ``` { "kibana": { "alert": { "url": "/app/ml/explorer/?_g=(ml%3A(jobIds%3A!(rt-anomaly-mean-value))%2Ctime%3A(from%3A'2023-09-28T14%3A57%3A00.000Z'%2Cmode%3Aabsolute%2Cto%3A'2023-09-28T15%3A17%3A00.000Z'))&_a=(explorer%3A(mlExplorerFilter%3A(filterActive%3A!t%2CfilteredFields%3A!(key%2Cthird-key)%2CinfluencersFilterQuery%3A(bool%3A(minimum_should_match%3A1%2Cshould%3A!((match_phrase%3A(key%3Athird-key)))))%2CqueryString%3A'key%3A%22third-key%22')%2CmlExplorerSwimlane%3A()))", "reason": "Alerts are raised based on real-time scores. Remember that scores may be adjusted over time as data continues to be analyzed.", "job_id": "rt-anomaly-mean-value", "anomaly_score": 73.63508175828011, "is_interim": false, "anomaly_timestamp": 1695913620000, "top_records": [{ "job_id": "rt-anomaly-mean-value", "record_score": 73.63516446528412, "initial_record_score": 73.63516446528412, "detector_index": 0, "is_interim": false, "timestamp": 1695913620000, "partition_field_name": "key", "partition_field_value": "third-key", "function": "mean", "actual": [ 3 ], "typical": [ 4.187715468532429 ] }], "top_influencers": [{ "job_id": "rt-anomaly-mean-value", "influencer_field_name": "key", "influencer_field_value": "third-key", "influencer_score": 73.63508175828011, "initial_influencer_score": 73.63508175828011, "is_interim": false, "timestamp": 1695913620000 }], "action_group": "anomaly_score_match", "flapping": false, "flapping_history": [ true, false, false, false ], "instance": { "id": "rt-anomaly-mean-value" }, "maintenance_window_ids": [], "rule": { "category": "Anomaly detection alert", "consumer": "alerts", "execution": { "uuid": "e9e681d4-c8e4-43eb-82e5-a58bdf7ffe12" }, "name": "rt-ad-alert-influencer", "parameters": { "severity": 5, "resultType": "influencer", "includeInterim": false, "jobSelection": { "jobIds": [ "rt-anomaly-mean-value" ], "groupIds": [] }, "lookbackInterval": null, "topNBuckets": null }, "producer": "ml", "revision": 0, "rule_type_id": "xpack.ml.anomaly_detection_alert", "tags": [], "uuid": "9e1d6bc0-5e10-11ee-8416-3bf48cca0922" }, "status": "active", "uuid": "c9c1f075-9985-4c55-8ff8-22349cb30269", "workflow_status": "open", "duration": { "us": "99021000000" }, "start": "2023-09-28T15:07:12.868Z", "time_range": { "gte": "2023-09-28T15:07:12.868Z" } }, "space_ids": [ "default" ], "version": "8.11.0" }, "@timestamp": "2023-09-28T15:08:51.889Z", "event": { "action": "active", "kind": "signal" }, "tags": [] } ``` </details> Now they look like: <details> <summary>Flattened</summary> ``` { "kibana.alert.url": "/app/ml/explorer/?_g=(ml%3A(jobIds%3A!(rt-anomaly-mean-value))%2Ctime%3A(from%3A'2023-09-28T15%3A03%3A00.000Z'%2Cmode%3Aabsolute%2Cto%3A'2023-09-28T15%3A23%3A00.000Z'))&_a=(explorer%3A(mlExplorerFilter%3A(filterActive%3A!t%2CfilteredFields%3A!(key%2Cthird-key)%2CinfluencersFilterQuery%3A(bool%3A(minimum_should_match%3A1%2Cshould%3A!((match_phrase%3A(key%3Athird-key)))))%2CqueryString%3A'key%3A%22third-key%22')%2CmlExplorerSwimlane%3A()))", "kibana.alert.reason": "Alerts are raised based on real-time scores. Remember that scores may be adjusted over time as data continues to be analyzed.", "kibana.alert.job_id": "rt-anomaly-mean-value", "kibana.alert.anomaly_score": 72.75515452061356, "kibana.alert.is_interim": false, "kibana.alert.anomaly_timestamp": 1695913980000, "kibana.alert.top_records": [{ "job_id": "rt-anomaly-mean-value", "record_score": 72.75515452061356, "initial_record_score": 72.75515452061356, "detector_index": 0, "is_interim": false, "timestamp": 1695913980000, "partition_field_name": "key", "partition_field_value": "third-key", "function": "mean", "actual": [ 0.5 ], "typical": [ 4.138745343296527 ] }], "kibana.alert.top_influencers": [{ "job_id": "rt-anomaly-mean-value", "influencer_field_name": "key", "influencer_field_value": "third-key", "influencer_score": 72.75515452061356, "initial_influencer_score": 72.75515452061356, "is_interim": false, "timestamp": 1695913980000 }], "kibana.alert.rule.category": "Anomaly detection alert", "kibana.alert.rule.consumer": "alerts", "kibana.alert.rule.execution.uuid": "17fef3d3-d595-4362-837e-b2a73650169e", "kibana.alert.rule.name": "rt-ad-alert-influencer", "kibana.alert.rule.parameters": { "severity": 5, "resultType": "influencer", "includeInterim": false, "jobSelection": { "jobIds": [ "rt-anomaly-mean-value" ], "groupIds": [] }, "lookbackInterval": null, "topNBuckets": null }, "kibana.alert.rule.producer": "ml", "kibana.alert.rule.revision": 0, "kibana.alert.rule.rule_type_id": "xpack.ml.anomaly_detection_alert", "kibana.alert.rule.tags": [], "kibana.alert.rule.uuid": "757c7610-5e11-11ee-8bc6-a95c3ced4757", "kibana.space_ids": [ "default" ], "@timestamp": "2023-09-28T15:14:52.057Z", "event.action": "active", "event.kind": "signal", "kibana.alert.action_group": "anomaly_score_match", "kibana.alert.flapping": false, "kibana.alert.flapping_history": [ true, false, false, false ], "kibana.alert.instance.id": "rt-anomaly-mean-value", "kibana.alert.maintenance_window_ids": [], "kibana.alert.status": "active", "kibana.alert.uuid": "ac1f0d7c-461b-4fc6-b4c3-04416ac876d3", "kibana.alert.workflow_status": "open", "kibana.alert.duration.us": "99028000000", "kibana.alert.start": "2023-09-28T15:13:13.028Z", "kibana.alert.time_range": { "gte": "2023-09-28T15:13:13.028Z" }, "kibana.version": "8.11.0", "tags": [] } ``` </details>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Summarize your PR. If it involves visual changes include a screenshot or gif.
Checklist
Delete any items that are not applicable to this PR.
Risk Matrix
Delete this section if it is not applicable to this PR.
Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.
When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:
For maintainers