-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Risk Score Persistence API (#161503)
## Summary * Introduces a new API, POST `/api/risk_scores/calculate`, that triggers the code introduced here * As with the [preview route](#155966), this endpoint is behind the `riskScoringRoutesEnabled` feature flag * We intend to __REMOVE__ this endpoint before 8.10 release; it's mainly a convenience/checkpoint for testing the existing code. The next PR will introduce a scheduled Task Manager task that invokes this code periodically. * Updates to the /preview route: * `data_view_id` is now a required parameter on both endpoints. If a dataview is not found by that ID, the id is used as the general index pattern to the query. * Response has been updated to be more similar to the [ECS risk fields](elastic/ecs#2236) powering this data. * Mappings created by the [Data Client](#158422) have been updated to be aligned to the ECS risk fields (linked above) * Adds/updates the [OpenAPI spec](https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/server/lib/risk_engine/schema/risk_score_apis.yml) for these endpoints; useful starting point if you're trying to get oriented here. ## Things to review * [PR Demo environment](https://rylnd-pr-161503-risk-score-task-api.kbndev.co/app/home) * Preview API and related UI still works as expected * Calculation/Persistence API correctly bootstraps/persists data * correct mappings/ILM are created * things work in non-default spaces ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios ### Risk Matrix Delete this section if it is not applicable to this PR. Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release. When forming the risk matrix, consider some of the following examples and how they may potentially impact the change: | Risk | Probability | Severity | Mitigation/Notes | |---------------------------|-------------|----------|-------------------------| | Multiple Spaces—unexpected behavior in non-default Kibana Space. | Low | High | Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces. | | Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks. | High | Low | Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure. | | Code should gracefully handle cases when feature X or plugin Y are disabled. | Medium | High | Unit tests will verify that any feature flag or plugin combination still results in our service operational. | | [See more potential risk examples](https://github.com/elastic/kibana/blob/main/RISK_MATRIX.mdx) | ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- Loading branch information
Showing
45 changed files
with
2,378 additions
and
608 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
32 changes: 32 additions & 0 deletions
32
x-pack/plugins/security_solution/common/risk_engine/risk_score_calculation/request_schema.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import * as t from 'io-ts'; | ||
| import { DataViewId } from '../../api/detection_engine'; | ||
| import { afterKeysSchema } from '../after_keys'; | ||
| import { identifierTypeSchema } from '../identifier_types'; | ||
| import { riskWeightsSchema } from '../risk_weights/schema'; | ||
|
|
||
| export const riskScoreCalculationRequestSchema = t.exact( | ||
| t.intersection([ | ||
| t.type({ | ||
| data_view_id: DataViewId, | ||
| identifier_type: identifierTypeSchema, | ||
| range: t.type({ | ||
| start: t.string, | ||
| end: t.string, | ||
| }), | ||
| }), | ||
| t.partial({ | ||
| after_keys: afterKeysSchema, | ||
| debug: t.boolean, | ||
| filter: t.unknown, | ||
| page_size: t.number, | ||
| weights: riskWeightsSchema, | ||
| }), | ||
| ]) | ||
| ); |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,5 +11,5 @@ export enum RiskWeightTypes { | |
| } | ||
|
|
||
| export enum RiskCategories { | ||
| alerts = 'alerts', | ||
| category_1 = 'category_1', | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
23 changes: 23 additions & 0 deletions
23
...lugins/security_solution/server/lib/risk_engine/calculate_and_persist_risk_scores.mock.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import type { CalculateAndPersistScoresResponse } from './types'; | ||
|
|
||
| const buildResponseMock = ( | ||
| overrides: Partial<CalculateAndPersistScoresResponse> = {} | ||
| ): CalculateAndPersistScoresResponse => ({ | ||
| after_keys: { | ||
| host: { 'host.name': 'hostname' }, | ||
| }, | ||
| errors: [], | ||
| scores_written: 2, | ||
| ...overrides, | ||
| }); | ||
|
|
||
| export const calculateAndPersistRiskScoresMock = { | ||
| buildResponse: buildResponseMock, | ||
| }; |
51 changes: 51 additions & 0 deletions
51
...lugins/security_solution/server/lib/risk_engine/calculate_and_persist_risk_scores.test.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import type { ElasticsearchClient, Logger } from '@kbn/core/server'; | ||
| import { elasticsearchServiceMock, loggingSystemMock } from '@kbn/core/server/mocks'; | ||
|
|
||
| import { calculateAndPersistRiskScores } from './calculate_and_persist_risk_scores'; | ||
| import { calculateRiskScores } from './calculate_risk_scores'; | ||
| import { calculateRiskScoresMock } from './calculate_risk_scores.mock'; | ||
|
|
||
| jest.mock('./calculate_risk_scores'); | ||
|
|
||
| describe('calculateAndPersistRiskScores', () => { | ||
| let esClient: ElasticsearchClient; | ||
| let logger: Logger; | ||
|
|
||
| beforeEach(() => { | ||
| esClient = elasticsearchServiceMock.createScopedClusterClient().asCurrentUser; | ||
| logger = loggingSystemMock.createLogger(); | ||
| }); | ||
|
|
||
| describe('with no risk scores to persist', () => { | ||
| beforeEach(() => { | ||
| (calculateRiskScores as jest.Mock).mockResolvedValueOnce( | ||
| calculateRiskScoresMock.buildResponse({ scores: { host: [] } }) | ||
| ); | ||
| }); | ||
|
|
||
| it('returns an appropriate response', async () => { | ||
| const results = await calculateAndPersistRiskScores({ | ||
| afterKeys: {}, | ||
| identifierType: 'host', | ||
| esClient, | ||
| logger, | ||
| index: 'index', | ||
| pageSize: 500, | ||
| range: { start: 'now - 15d', end: 'now' }, | ||
| spaceId: 'default', | ||
| // @ts-expect-error not relevant for this test | ||
| riskEngineDataClient: { getWriter: jest.fn() }, | ||
| runtimeMappings: {}, | ||
| }); | ||
|
|
||
| expect(results).toEqual({ after_keys: {}, errors: [], scores_written: 0 }); | ||
| }); | ||
| }); | ||
| }); |
Oops, something went wrong.