Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RAM] Adds Bulk Edit API to rulesClient #126904

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
db16543
POC
vitaliidm Jan 11, 2022
23cc54a
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Feb 23, 2022
4c6bac3
POC update
vitaliidm Mar 8, 2022
6bb3489
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 9, 2022
d116acc
add bulkmark API keys invalidate
vitaliidm Mar 9, 2022
4fdc48d
Merge branch 'security-solution/bulk-update-rulesClient' of https://g…
vitaliidm Mar 9, 2022
7a151d2
actions refactoring
vitaliidm Mar 9, 2022
ba9a4ab
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 9, 2022
b020e11
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 10, 2022
42db655
move actions to bulk actions
vitaliidm Mar 10, 2022
82aeb49
encrypted + multi_terms
vitaliidm Mar 15, 2022
5d9580d
add validated mutated params
vitaliidm Mar 16, 2022
b61ca0f
revert changes
vitaliidm Mar 16, 2022
8de2546
updates
vitaliidm Mar 17, 2022
20250ca
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 17, 2022
d7edcb5
Merge branch 'main' into security-solution/bulk-update-rulesClient
kibanamachine Mar 17, 2022
6351fba
POC route
vitaliidm Mar 17, 2022
c1ca8f3
few updates
vitaliidm Mar 17, 2022
3f7dcf6
add alering route
vitaliidm Mar 17, 2022
6245f23
comment taskRunner actions
vitaliidm Mar 17, 2022
22699a3
FEW UPDATES
vitaliidm Mar 18, 2022
be2f05b
add tests for alerting routes
vitaliidm Mar 18, 2022
3d3a39d
fix typo
vitaliidm Mar 18, 2022
c9debba
unterface update
vitaliidm Mar 18, 2022
da313f1
original params for validatedParams
vitaliidm Mar 22, 2022
c180049
remove unnecessary encrypted method
vitaliidm Mar 22, 2022
a48a93c
aggr validation
vitaliidm Mar 22, 2022
07f1e42
add bulk_edit tests
vitaliidm Mar 23, 2022
589c96b
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 23, 2022
cafa5ff
mocked test cases
vitaliidm Mar 23, 2022
88574c6
add more tests
vitaliidm Mar 23, 2022
5b83028
add more tests
vitaliidm Mar 23, 2022
1295292
add more tests
vitaliidm Mar 23, 2022
bd18d5e
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 23, 2022
7554e0d
cleanup: remove security solutiion changes
vitaliidm Mar 23, 2022
802ff6c
fix test
vitaliidm Mar 24, 2022
1d60e84
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 24, 2022
e6e3235
fix failed cases tests
vitaliidm Mar 24, 2022
765d74a
cleanup of withSpan
vitaliidm Mar 24, 2022
27eb27d
add order to multi_terms
vitaliidm Mar 24, 2022
c9bdb85
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 24, 2022
21bbe74
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 24, 2022
83752df
add namespace
vitaliidm Mar 24, 2022
1ad1bcd
cleanup
vitaliidm Mar 24, 2022
6c32a63
code review changes
vitaliidm Mar 24, 2022
241eeea
fix tests
vitaliidm Mar 24, 2022
296d9b2
fix failed tests
vitaliidm Mar 28, 2022
563606e
make paramsModifier async
vitaliidm Mar 28, 2022
05974d2
add ids
vitaliidm Mar 28, 2022
64fcc96
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 28, 2022
2b61b4d
cleanup
vitaliidm Mar 28, 2022
a4fff45
fix typings
vitaliidm Mar 28, 2022
b323c4c
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 28, 2022
5f94542
add functional tests
vitaliidm Mar 29, 2022
5e96a39
move rule_rewrite to a separate lib util
vitaliidm Mar 30, 2022
70f1c32
add security_and_spaces tests
vitaliidm Mar 30, 2022
1862832
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 30, 2022
dc13dd4
CR: BulkEditFields rename
vitaliidm Mar 30, 2022
4f93236
Merge branch 'security-solution/bulk-update-rulesClient' of https://g…
vitaliidm Mar 30, 2022
ae171f0
CR: updates
vitaliidm Mar 30, 2022
59d41e2
CR: rename editActins to operations
vitaliidm Mar 31, 2022
fc677ff
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Mar 31, 2022
db0bc0e
more functional tests
vitaliidm Mar 31, 2022
ed7c877
introduce skeleton for occ on bulk edit
XavierM Apr 1, 2022
55931d6
add tests for encrypted
vitaliidm Apr 1, 2022
6ea3e6f
fix typo in encrypted method
vitaliidm Apr 4, 2022
6b34fd7
refactor bulkRetry
vitaliidm Apr 4, 2022
d46c11f
fix
vitaliidm Apr 4, 2022
7f2a03d
fix lint
vitaliidm Apr 4, 2022
beab1ac
add rules ids to KueryNode converter
vitaliidm Apr 4, 2022
35d6ce6
bulk update
vitaliidm Apr 4, 2022
d5b52f5
refactor bulkRetry
vitaliidm Apr 4, 2022
96cc3f5
fix unit tests
vitaliidm Apr 4, 2022
3d01c77
add retryIfBulkEditConflicts tests
vitaliidm Apr 5, 2022
c555b83
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Apr 5, 2022
7eb5947
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Apr 5, 2022
3270cbb
check typings
vitaliidm Apr 5, 2022
3f8bc4c
fix tyois
vitaliidm Apr 5, 2022
f17f005
add more tests
vitaliidm Apr 5, 2022
027bfca
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Apr 6, 2022
214b7fa
fix styles
vitaliidm Apr 6, 2022
a613a2d
Merge branch 'security-solution/bulk-update-rulesClient' of https://g…
vitaliidm Apr 6, 2022
a054e5f
add 'should handle bulk edit of rules appropriately when consumer is …
vitaliidm Apr 6, 2022
b15799e
add it('should handle update alert request appropriately when…
vitaliidm Apr 6, 2022
5ae088b
addditiona; tests
vitaliidm Apr 6, 2022
dcca12f
tests and typings
vitaliidm Apr 6, 2022
9b4794e
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Apr 6, 2022
80d5e75
schema changes
vitaliidm Apr 6, 2022
9a29fa7
alerts to rule renamimgs
vitaliidm Apr 6, 2022
50662bb
help text update
vitaliidm Apr 6, 2022
dc9bffc
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Apr 11, 2022
ca086f0
fix empty filter
vitaliidm Apr 12, 2022
77da27d
fix typings
vitaliidm Apr 12, 2022
0a3b0e3
Revert "fix typings"
vitaliidm Apr 12, 2022
a441597
fix typing in filter
vitaliidm Apr 12, 2022
8efd299
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Apr 13, 2022
ed64393
fix failed tests
vitaliidm Apr 13, 2022
5993835
Updating README
vitaliidm Apr 13, 2022
da6ffa1
export BUlkEdtotError
vitaliidm Apr 13, 2022
7f86c0b
Update x-pack/plugins/encrypted_saved_objects/server/mocks.ts
vitaliidm Apr 14, 2022
8fb30e7
Update x-pack/plugins/encrypted_saved_objects/README.md
vitaliidm Apr 14, 2022
d32fdbf
Update x-pack/plugins/encrypted_saved_objects/README.md
vitaliidm Apr 14, 2022
788a70c
- add concurrency limit to ESO
vitaliidm Apr 14, 2022
723e495
fix failing test
vitaliidm Apr 14, 2022
7b3ae88
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Apr 14, 2022
f4b76a4
add docs
vitaliidm Apr 20, 2022
55f6246
Merge branch 'security-solution/bulk-update-rulesClient' of https://g…
vitaliidm Apr 20, 2022
7f4eb30
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Apr 20, 2022
e094d8c
type fixes
vitaliidm Apr 20, 2022
cf7e22b
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine Apr 20, 2022
4427166
fix failed test
vitaliidm Apr 20, 2022
5e31374
Merge branch 'security-solution/bulk-update-rulesClient' of https://g…
vitaliidm Apr 20, 2022
800b306
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine Apr 20, 2022
aacb6f1
Security CR feedback
vitaliidm Apr 20, 2022
b6e7a88
Merge branch 'security-solution/bulk-update-rulesClient' of https://g…
vitaliidm Apr 20, 2022
4423c39
rename method
vitaliidm Apr 20, 2022
17b927c
add functional tests
vitaliidm Apr 20, 2022
ba906df
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Apr 21, 2022
61606ba
Merge branch 'main' into security-solution/bulk-update-rulesClient
kibanamachine Apr 21, 2022
74f11a2
remove missing export warning
vitaliidm Apr 21, 2022
bfdd96b
Merge branch 'main' into security-solution/bulk-update-rulesClient
kibanamachine Apr 25, 2022
dae1940
fix missing await in rulesClient context, introduced by recent changes
vitaliidm Apr 25, 2022
f594701
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm Apr 26, 2022
cfb3a56
Update x-pack/plugins/encrypted_saved_objects/server/saved_objects/in…
vitaliidm Apr 26, 2022
61f307b
Update x-pack/plugins/encrypted_saved_objects/server/saved_objects/in…
vitaliidm Apr 26, 2022
2cab36b
Update x-pack/test/encrypted_saved_objects_api_integration/tests/encr…
vitaliidm Apr 26, 2022
ebec27a
Update x-pack/plugins/encrypted_saved_objects/server/saved_objects/in…
vitaliidm Apr 26, 2022
722adbf
fix tests
vitaliidm Apr 26, 2022
712ff08
Merge branch 'main' into security-solution/bulk-update-rulesClient
vitaliidm May 9, 2022
19ca219
remove typo(?)
vitaliidm May 9, 2022
2b4a992
fix merge conflict issue
vitaliidm May 9, 2022
d9138c2
CR: add audit of authorization
vitaliidm May 9, 2022
4e74487
CR: interval and operations values
vitaliidm May 10, 2022
2e6d918
CR: mapped params
vitaliidm May 10, 2022
38428e1
CR: remove mark_api_key_for_invalidation
vitaliidm May 10, 2022
00020ed
CR: fix failed test
vitaliidm May 10, 2022
0b53e91
CR: move applyBulkOPeration to a separate file
vitaliidm May 11, 2022
9faef0b
CR: add missing _bulk_edit route error handlers
vitaliidm May 11, 2022
2ca3afc
Merge branch 'main' into security-solution/bulk-update-rulesClient
kibanamachine May 11, 2022
0a34da4
revisist of API keys invalidation
vitaliidm May 11, 2022
5045d45
fix test
vitaliidm May 11, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import { sortOrderSchema } from './common_schemas';
* - nested
* - reverse_nested
* - terms
* - multi_terms
*
* Not fully supported:
* - filter
Expand All @@ -37,7 +38,6 @@ import { sortOrderSchema } from './common_schemas';
* - global
* - ip_range
* - missing
* - multi_terms
* - parent
* - range
* - rare_terms
Expand All @@ -63,6 +63,36 @@ const boolSchema = s.object({
}),
});

const orderSchema = s.oneOf([
sortOrderSchema,
s.recordOf(s.string(), sortOrderSchema),
s.arrayOf(s.recordOf(s.string(), sortOrderSchema)),
]);

const termsSchema = s.object({
field: s.maybe(s.string()),
collect_mode: s.maybe(s.string()),
exclude: s.maybe(s.oneOf([s.string(), s.arrayOf(s.string())])),
include: s.maybe(s.oneOf([s.string(), s.arrayOf(s.string())])),
execution_hint: s.maybe(s.string()),
missing: s.maybe(s.number()),
min_doc_count: s.maybe(s.number({ min: 1 })),
size: s.maybe(s.number()),
show_term_doc_count_error: s.maybe(s.boolean()),
order: s.maybe(orderSchema),
});

const multiTermsSchema = s.object({
terms: s.arrayOf(termsSchema),
size: s.maybe(s.number()),
shard_size: s.maybe(s.number()),
show_term_doc_count_error: s.maybe(s.boolean()),
min_doc_count: s.maybe(s.number()),
shard_min_doc_count: s.maybe(s.number()),
collect_mode: s.maybe(s.oneOf([s.literal('depth_first'), s.literal('breadth_first')])),
order: s.maybe(s.recordOf(s.string(), orderSchema)),
});

export const bucketAggsSchemas: Record<string, ObjectType> = {
date_range: s.object({
field: s.string(),
Expand Down Expand Up @@ -104,22 +134,6 @@ export const bucketAggsSchemas: Record<string, ObjectType> = {
reverse_nested: s.object({
path: s.maybe(s.string()),
}),
terms: s.object({
field: s.maybe(s.string()),
collect_mode: s.maybe(s.string()),
exclude: s.maybe(s.oneOf([s.string(), s.arrayOf(s.string())])),
include: s.maybe(s.oneOf([s.string(), s.arrayOf(s.string())])),
execution_hint: s.maybe(s.string()),
missing: s.maybe(s.number()),
min_doc_count: s.maybe(s.number({ min: 1 })),
size: s.maybe(s.number()),
show_term_doc_count_error: s.maybe(s.boolean()),
order: s.maybe(
s.oneOf([
sortOrderSchema,
s.recordOf(s.string(), sortOrderSchema),
s.arrayOf(s.recordOf(s.string(), sortOrderSchema)),
])
),
}),
multi_terms: multiTermsSchema,
terms: termsSchema,
};
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,28 @@ describe('validateAndConvertAggregations', () => {
});
});

it('validates multi_terms aggregations', () => {
expect(
validateAndConvertAggregations(
['foo'],
{
aggName: {
multi_terms: {
terms: [{ field: 'foo.attributes.description' }, { field: 'foo.attributes.bytes' }],
},
},
},
mockMappings
)
).toEqual({
aggName: {
multi_terms: {
terms: [{ field: 'foo.description' }, { field: 'foo.bytes' }],
},
},
});
});

it('validates a nested field in simple aggregations', () => {
expect(
validateAndConvertAggregations(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@

import * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
import { ObjectType } from '@kbn/config-schema';
import { isPlainObject } from 'lodash';
import { isPlainObject, isArray } from 'lodash';

import { IndexMapping } from '../../../mappings';
import {
Expand Down Expand Up @@ -181,11 +181,17 @@ const recursiveRewrite = (

const nestedContext = childContext(context, key);
const newKey = rewriteKey ? validateAndRewriteAttributePath(key, nestedContext) : key;
const newValue = rewriteValue
? validateAndRewriteAttributePath(value, nestedContext)
: isPlainObject(value)
? recursiveRewrite(value, nestedContext, [...parents, key])
: value;

let newValue = value;
if (rewriteValue) {
vitaliidm marked this conversation as resolved.
Show resolved Hide resolved
newValue = validateAndRewriteAttributePath(value, nestedContext);
} else if (isArray(value)) {
newValue = value.map((v) =>
isPlainObject(v) ? recursiveRewrite(v, nestedContext, parents) : v
);
} else if (isPlainObject(value)) {
newValue = recursiveRewrite(value, nestedContext, [...parents, key]);
}

return {
...memo,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ export enum WriteOperations {
MuteAlert = 'muteAlert',
UnmuteAlert = 'unmuteAlert',
Snooze = 'snooze',
BulkEdit = 'bulkEdit',
Unsnooze = 'unsnooze',
}

Expand Down
9 changes: 8 additions & 1 deletion x-pack/plugins/alerting/server/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,14 @@ export type {
} from './types';
export { DEFAULT_MAX_EPHEMERAL_ACTIONS_PER_ALERT } from './config';
export type { PluginSetupContract, PluginStartContract } from './plugin';
export type { FindResult } from './rules_client';
export type {
FindResult,
BulkEditOperation,
BulkEditError,
BulkEditOptions,
BulkEditOptionsFilter,
BulkEditOptionsIds,
} from './rules_client';
export type { PublicAlert as Alert } from './alert';
export { parseDuration } from './lib';
export { getEsErrorMessage } from './lib/errors';
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { loggingSystemMock, savedObjectsClientMock } from '@kbn/core/server/mocks';
import { bulkMarkApiKeysForInvalidation } from './bulk_mark_api_keys_for_invalidation';

describe('bulkMarkApiKeysForInvalidation', () => {
test('should call savedObjectsClient bulkCreate with the proper params', async () => {
const unsecuredSavedObjectsClient = savedObjectsClientMock.create();
unsecuredSavedObjectsClient.bulkCreate.mockResolvedValueOnce({ saved_objects: [] });

await bulkMarkApiKeysForInvalidation(
{ apiKeys: [Buffer.from('123').toString('base64'), Buffer.from('456').toString('base64')] },
loggingSystemMock.create().get(),
unsecuredSavedObjectsClient
);

const bulkCreateCallMock = unsecuredSavedObjectsClient.bulkCreate.mock.calls[0];
const savedObjects = bulkCreateCallMock[0];

expect(unsecuredSavedObjectsClient.bulkCreate).toHaveBeenCalledTimes(1);
expect(bulkCreateCallMock).toHaveLength(1);

expect(savedObjects).toHaveLength(2);
expect(savedObjects[0]).toHaveProperty('type', 'api_key_pending_invalidation');
expect(savedObjects[0]).toHaveProperty('attributes.apiKeyId', '123');
expect(savedObjects[0]).toHaveProperty('attributes.createdAt', expect.any(String));
expect(savedObjects[1]).toHaveProperty('type', 'api_key_pending_invalidation');
expect(savedObjects[1]).toHaveProperty('attributes.apiKeyId', '456');
expect(savedObjects[1]).toHaveProperty('attributes.createdAt', expect.any(String));
});

test('should log the proper error when savedObjectsClient create failed', async () => {
const logger = loggingSystemMock.create().get();
const unsecuredSavedObjectsClient = savedObjectsClientMock.create();
unsecuredSavedObjectsClient.bulkCreate.mockRejectedValueOnce(new Error('Fail'));
await bulkMarkApiKeysForInvalidation(
{ apiKeys: [Buffer.from('123').toString('base64'), Buffer.from('456').toString('base64')] },
logger,
unsecuredSavedObjectsClient
);
expect(logger.error).toHaveBeenCalledWith(
'Failed to bulk mark list of API keys ["MTIz", "NDU2"] for invalidation: Fail'
);
});

test('should not call savedObjectsClient bulkCreate if list of apiKeys empty', async () => {
const unsecuredSavedObjectsClient = savedObjectsClientMock.create();
unsecuredSavedObjectsClient.bulkCreate.mockResolvedValueOnce({ saved_objects: [] });

await bulkMarkApiKeysForInvalidation(
{ apiKeys: [] },
loggingSystemMock.create().get(),
unsecuredSavedObjectsClient
);

expect(unsecuredSavedObjectsClient.bulkCreate).not.toHaveBeenCalled();
});
});
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
/*
mikecote marked this conversation as resolved.
Show resolved Hide resolved
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import { Logger, SavedObjectsClientContract } from '@kbn/core/server';

export const bulkMarkApiKeysForInvalidation = async (
{ apiKeys }: { apiKeys: string[] },
mikecote marked this conversation as resolved.
Show resolved Hide resolved
logger: Logger,
savedObjectsClient: SavedObjectsClientContract
): Promise<void> => {
if (apiKeys.length === 0) {
return;
}

try {
const apiKeyIds = apiKeys.map(
(apiKey) => Buffer.from(apiKey, 'base64').toString().split(':')[0]
);
await savedObjectsClient.bulkCreate(
apiKeyIds.map((apiKeyId) => ({
attributes: {
apiKeyId,
createdAt: new Date().toISOString(),
},
type: 'api_key_pending_invalidation',
}))
);
} catch (e) {
logger.error(
`Failed to bulk mark list of API keys [${apiKeys
.map((key) => `"${key}"`)
.join(', ')}] for invalidation: ${e.message}`
);
}
};

This file was deleted.

This file was deleted.

Loading