Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Search] Search Sessions Monitoring Task #85253

Merged
merged 97 commits into from
Jan 11, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
97 commits
Select commit Hold shift + click to select a range
1c82050
Monitor ids
Nov 24, 2020
52032fc
import fix
Nov 24, 2020
60abe6c
Merge remote-tracking branch 'upstream/master' into sessions/retry-logic
Nov 25, 2020
8a12991
solve circular dep
Nov 25, 2020
d692c1e
eslint
Nov 30, 2020
9a02d9c
Merge branch 'master' of github.com:elastic/kibana into sessions/retr…
Nov 30, 2020
1092c7a
mock circular dep
Nov 30, 2020
500f15d
max retries test
Nov 30, 2020
c6fea68
mock circular dep
Nov 30, 2020
e5325e9
test
Nov 30, 2020
c540fde
jest <(-:C
Dec 1, 2020
5f7db2d
jestttttt
Dec 1, 2020
80fdf82
Merge branch 'master' of github.com:elastic/kibana into sessions/retr…
Dec 1, 2020
4886819
[data.search] Move search method inside session service and add tests
lukasolson Dec 1, 2020
7387405
Merge branch 'master' of github.com:elastic/kibana into sessions/retr…
Dec 2, 2020
a91ea10
merge
Dec 2, 2020
0de7730
Move background session service to data_enhanced plugin
lukasolson Dec 2, 2020
5253b42
Merge branch 'master' of github.com:elastic/kibana into sessions/retr…
Dec 3, 2020
ec54b8b
Merge branch 'master' into search-session-search
lukasolson Dec 3, 2020
4ff6eaf
Merge branch 'search-session-search' into search-session-enhanced
lukasolson Dec 3, 2020
93128a1
Better logs
Dec 3, 2020
ea14d29
Fix types
lukasolson Dec 3, 2020
4ce9fef
Merge branch 'master' into search-session-enhanced
lukasolson Dec 3, 2020
7537e69
Merge branch 'master' into search-session-enhanced
kibanamachine Dec 3, 2020
53b8c32
Merge branch 'master' into search-session-enhanced
kibanamachine Dec 4, 2020
a0e2142
Space aware session service
Dec 6, 2020
54b440a
Merge branch 'master' of github.com:elastic/kibana into sessions/retr…
Dec 6, 2020
e8ec140
Merge remote-tracking branch 'lukasolson/search-session-enhanced' int…
Dec 6, 2020
149d94e
ts
Dec 6, 2020
f3975ce
Merge remote-tracking branch 'lukasolson/search-session-enhanced' int…
Dec 6, 2020
1fc434a
initial
Dec 6, 2020
7a1be88
initial
Dec 6, 2020
0b875ac
Fix session service saving
Dec 7, 2020
f40fcca
Merge branch 'master' of github.com:elastic/kibana into sessions/retr…
Dec 7, 2020
9011366
Merge branch 'master' of github.com:elastic/kibana into sessions/moni…
Dec 7, 2020
e039f32
Merge branch 'sessions/retry-logic' into sessions/monitor-task
Dec 7, 2020
c1ba50c
Merge remote-tracking branch 'upstream/master' into sessions/retry-logic
Dec 7, 2020
b5b0895
merge fix
Dec 7, 2020
f14f042
stable stringify
Dec 7, 2020
5b9782d
INMEM_MAX_SESSIONS
Dec 7, 2020
0858b98
INMEM_MAX_SESSIONS
Dec 7, 2020
1bdd7f7
Merge branch 'sessions/retry-logic' into sessions/monitor-task
Dec 7, 2020
cf0b6af
use the status API
Dec 7, 2020
a8d7233
Move task scheduling behind a feature flag
Dec 8, 2020
6c9e345
Merge branch 'master' of github.com:elastic/kibana into sessions/retr…
Dec 8, 2020
e85da48
Merge branch 'sessions/retry-logic' of github.com:lizozom/kibana into…
Dec 8, 2020
49c9535
Merge branch 'master' of github.com:elastic/kibana into sessions/moni…
Dec 8, 2020
4876f3d
Update x-pack/plugins/data_enhanced/server/search/session/session_ser…
lizozom Dec 8, 2020
685b7ae
Add unit tests
Dec 8, 2020
ffabaa9
Update x-pack/plugins/data_enhanced/server/search/session/session_ser…
lizozom Dec 8, 2020
63c7d6a
Use setTimeout to schedule monitoring steps
Dec 8, 2020
7fd8b5e
Update request_utils.ts
lizozom Dec 8, 2020
51143af
Merge branch 'master' of github.com:elastic/kibana into sessions/retr…
Dec 8, 2020
bf54579
Merge branch 'master' of github.com:elastic/kibana into sessions/retr…
Dec 9, 2020
07fe4bf
settimeout
Dec 9, 2020
62bd60a
Merge branch 'sessions/retry-logic' into sessions/monitor-task
Dec 9, 2020
686b4c5
Merge branch 'sessions/monitor-task' of github.com:lizozom/kibana int…
Dec 9, 2020
1347551
Merge branch 'master' of github.com:elastic/kibana into sessions/moni…
Dec 9, 2020
bdf62d5
Merge branch 'master' of github.com:elastic/kibana into sessions/moni…
Jan 4, 2021
cf511f0
tiny cleanup
Jan 4, 2021
8a792d1
Core review + use client.asyncSearch.status
Jan 4, 2021
8e81c3a
Merge branch 'master' of github.com:elastic/kibana into sessions/moni…
Jan 4, 2021
202a56b
update ts
Jan 4, 2021
e41daa6
fix unit test
Jan 5, 2021
2961eeb
Merge branch 'master' of github.com:elastic/kibana into sessions/moni…
Jan 5, 2021
f3d25d3
code review fixes
Jan 5, 2021
44fcbf7
Save individual search errors on SO
Jan 5, 2021
4be1a93
Don't re-fetch completed or errored searches
Jan 5, 2021
36ad90c
Rename Background Sessions to Search Sessions (with a send to backgro…
Jan 6, 2021
d0df853
doc
Jan 6, 2021
e9b690c
doc
Jan 6, 2021
5618252
jest fun
Jan 6, 2021
20a6d86
rename rfc
Jan 6, 2021
3c04478
Merge branch 'master' of github.com:elastic/kibana into sessions/rena…
Jan 6, 2021
ee0eabb
Merge branch 'master' into sessions/rename-feature
kibanamachine Jan 6, 2021
530f9be
translations
Jan 7, 2021
85cd3b6
Merge branch 'master' of github.com:elastic/kibana into sessions/rena…
Jan 7, 2021
92d98f5
Merge branch 'sessions/rename-feature' of github.com:lizozom/kibana i…
Jan 7, 2021
4db8971
Merge branch 'master' of github.com:elastic/kibana into sessions/moni…
Jan 7, 2021
8f670f0
Merge branch 'sessions/rename-feature' into sessions/monitor-task
Jan 7, 2021
ea8d1a5
merge fix
Jan 7, 2021
1cdd02c
merge fix
Jan 7, 2021
5e52418
code review
Jan 7, 2021
19dee24
update so name in features
Jan 7, 2021
70dd89a
Merge branch 'sessions/rename-feature' into sessions/monitor-task
Jan 7, 2021
c71139d
Move deleteTaskIfItExists to task manager
Jan 7, 2021
19e423f
task_manager to ts project
mshustov Jan 7, 2021
6ada7f0
Merge branch 'pr/87646' into sessions/monitor-task
Jan 7, 2021
b6c4ad8
Merge branch 'master' of github.com:elastic/kibana into sessions/moni…
Jan 7, 2021
47c3af2
Move deleteTaskIfItExists to public contract
Jan 7, 2021
056a17b
mock
Jan 7, 2021
762bcc7
use task store
Jan 7, 2021
740f7d9
ts
Jan 7, 2021
9b26ff5
code review
Jan 7, 2021
ce23e86
Merge branch 'master' of github.com:elastic/kibana into sessions/moni…
Jan 11, 2021
8415b05
code review + jest
Jan 11, 2021
52a4ba1
Alerting code review
Jan 11, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions x-pack/plugins/alerts/server/alerts_client/alerts_client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@ import {
import { EncryptedSavedObjectsClient } from '../../../encrypted_saved_objects/server';
import { TaskManagerStartContract } from '../../../task_manager/server';
import { taskInstanceToAlertTaskInstance } from '../task_runner/alert_task_instance';
import { deleteTaskIfItExists } from '../lib/delete_task_if_it_exists';
import { RegistryAlertType, UntypedNormalizedAlertType } from '../alert_type_registry';
import { AlertsAuthorization, WriteOperations, ReadOperations } from '../authorization';
import { IEventLogClient } from '../../../../plugins/event_log/server';
Expand Down Expand Up @@ -602,7 +601,7 @@ export class AlertsClient {
const removeResult = await this.unsecuredSavedObjectsClient.delete('alert', id);

await Promise.all([
taskIdToRemove ? deleteTaskIfItExists(this.taskManager, taskIdToRemove) : null,
taskIdToRemove ? this.taskManager.removeIfExists(taskIdToRemove) : null,
apiKeyToInvalidate
? markApiKeyForInvalidation(
{ apiKey: apiKeyToInvalidate },
Expand Down Expand Up @@ -1060,7 +1059,7 @@ export class AlertsClient {

await Promise.all([
attributes.scheduledTaskId
? deleteTaskIfItExists(this.taskManager, attributes.scheduledTaskId)
? this.taskManager.removeIfExists(attributes.scheduledTaskId)
: null,
apiKeyToInvalidate
? await markApiKeyForInvalidation(
Expand Down
10 changes: 5 additions & 5 deletions x-pack/plugins/alerts/server/alerts_client/tests/delete.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ describe('delete()', () => {
const result = await alertsClient.delete({ id: '1' });
expect(result).toEqual({ success: true });
expect(unsecuredSavedObjectsClient.delete).toHaveBeenCalledWith('alert', '1');
expect(taskManager.remove).toHaveBeenCalledWith('task-123');
expect(taskManager.removeIfExists).toHaveBeenCalledWith('task-123');
expect(unsecuredSavedObjectsClient.create.mock.calls[0][0]).toBe(
'api_key_pending_invalidation'
);
Expand All @@ -135,7 +135,7 @@ describe('delete()', () => {
const result = await alertsClient.delete({ id: '1' });
expect(result).toEqual({ success: true });
expect(unsecuredSavedObjectsClient.delete).toHaveBeenCalledWith('alert', '1');
expect(taskManager.remove).toHaveBeenCalledWith('task-123');
expect(taskManager.removeIfExists).toHaveBeenCalledWith('task-123');
expect(unsecuredSavedObjectsClient.create).not.toHaveBeenCalled();
expect(unsecuredSavedObjectsClient.get).toHaveBeenCalledWith('alert', '1');
expect(alertsClientParams.logger.error).toHaveBeenCalledWith(
Expand All @@ -153,7 +153,7 @@ describe('delete()', () => {
});

await alertsClient.delete({ id: '1' });
expect(taskManager.remove).not.toHaveBeenCalled();
expect(taskManager.removeIfExists).not.toHaveBeenCalled();
});

test(`doesn't invalidate API key when apiKey is null`, async () => {
Expand Down Expand Up @@ -217,8 +217,8 @@ describe('delete()', () => {
);
});

test('throws error when taskManager.remove throws an error', async () => {
taskManager.remove.mockRejectedValue(new Error('TM Fail'));
test('throws error when taskManager.removeIfExists throws an error', async () => {
taskManager.removeIfExists.mockRejectedValue(new Error('TM Fail'));

await expect(alertsClient.delete({ id: '1' })).rejects.toThrowErrorMatchingInlineSnapshot(
`"TM Fail"`
Expand Down
10 changes: 5 additions & 5 deletions x-pack/plugins/alerts/server/alerts_client/tests/disable.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -199,7 +199,7 @@ describe('disable()', () => {
version: '123',
}
);
expect(taskManager.remove).toHaveBeenCalledWith('task-123');
expect(taskManager.removeIfExists).toHaveBeenCalledWith('task-123');
expect(
(unsecuredSavedObjectsClient.create.mock.calls[0][1] as InvalidatePendingApiKey).apiKeyId
).toBe('123');
Expand Down Expand Up @@ -254,7 +254,7 @@ describe('disable()', () => {
version: '123',
}
);
expect(taskManager.remove).toHaveBeenCalledWith('task-123');
expect(taskManager.removeIfExists).toHaveBeenCalledWith('task-123');
expect(unsecuredSavedObjectsClient.create).not.toHaveBeenCalled();
});

Expand All @@ -280,7 +280,7 @@ describe('disable()', () => {

await alertsClient.disable({ id: '1' });
expect(unsecuredSavedObjectsClient.update).not.toHaveBeenCalled();
expect(taskManager.remove).not.toHaveBeenCalled();
expect(taskManager.removeIfExists).not.toHaveBeenCalled();
expect(unsecuredSavedObjectsClient.create).not.toHaveBeenCalled();
});

Expand Down Expand Up @@ -314,7 +314,7 @@ describe('disable()', () => {

await alertsClient.disable({ id: '1' });
expect(unsecuredSavedObjectsClient.update).toHaveBeenCalled();
expect(taskManager.remove).toHaveBeenCalled();
expect(taskManager.removeIfExists).toHaveBeenCalled();
expect(unsecuredSavedObjectsClient.create).not.toHaveBeenCalled();
expect(alertsClientParams.logger.error).toHaveBeenCalledWith(
'disable(): Failed to load API key to invalidate on alert 1: Fail'
Expand All @@ -338,7 +338,7 @@ describe('disable()', () => {
});

test('throws when failing to remove task from task manager', async () => {
taskManager.remove.mockRejectedValueOnce(new Error('Failed to remove task'));
taskManager.removeIfExists.mockRejectedValueOnce(new Error('Failed to remove task'));

await expect(alertsClient.disable({ id: '1' })).rejects.toThrowErrorMatchingInlineSnapshot(
`"Failed to remove task"`
Expand Down
1 change: 0 additions & 1 deletion x-pack/plugins/data_enhanced/common/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ export {
IAsyncSearchOptions,
pollSearch,
SearchSessionSavedObjectAttributes,
SearchSessionFindOptions,
SearchSessionStatus,
SearchSessionRequestInfo,
} from './search';
41 changes: 39 additions & 2 deletions x-pack/plugins/data_enhanced/common/search/session/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,18 +13,55 @@ export interface SearchSessionSavedObjectAttributes {
* App that created the session. e.g 'discover'
*/
appId: string;
/**
* Creation time of the session
*/
created: string;
/**
* Expiration time of the session. Expiration itself is managed by Elasticsearch.
*/
expires: string;
/**
* status
*/
status: string;
/**
* urlGeneratorId
*/
urlGeneratorId: string;
/**
* The application state that was used to create the session.
* Should be used, for example, to re-load an expired search session.
*/
initialState: Record<string, unknown>;
/**
* Application state that should be used to restore the session.
* For example, relative dates are conveted to absolute ones.
*/
restoreState: Record<string, unknown>;
/**
* Mapping of search request hashes to their corresponsing info (async search id, etc.)
*/
idMapping: Record<string, SearchSessionRequestInfo>;
}

export interface SearchSessionRequestInfo {
id: string; // ID of the async search request
strategy: string; // Search strategy used to submit the search request
/**
* ID of the async search request
*/
id: string;
/**
* Search strategy used to submit the search request
*/
strategy: string;
/**
* status
*/
status: string;
/**
* An optional error. Set if status is set to error.
*/
error?: string;
}

export interface SearchSessionFindOptions {
Expand Down
3 changes: 2 additions & 1 deletion x-pack/plugins/data_enhanced/kibana.json
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,8 @@
"requiredPlugins": [
"bfetch",
"data",
"features"
"features",
"taskManager"
],
"optionalPlugins": ["kibanaUtils", "usageCollection"],
"server": true,
Expand Down
20 changes: 17 additions & 3 deletions x-pack/plugins/data_enhanced/server/plugin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
*/

import { CoreSetup, CoreStart, Logger, Plugin, PluginInitializerContext } from 'kibana/server';
import { TaskManagerSetupContract, TaskManagerStartContract } from '../../task_manager/server';
import {
PluginSetup as DataPluginSetup,
PluginStart as DataPluginStart,
Expand All @@ -24,9 +25,15 @@ import { getUiSettings } from './ui_settings';
interface SetupDependencies {
data: DataPluginSetup;
usageCollection?: UsageCollectionSetup;
taskManager: TaskManagerSetupContract;
}
export interface StartDependencies {
data: DataPluginStart;
taskManager: TaskManagerStartContract;
}

export class EnhancedDataServerPlugin implements Plugin<void, void, SetupDependencies> {
export class EnhancedDataServerPlugin
implements Plugin<void, void, SetupDependencies, StartDependencies> {
private readonly logger: Logger;
private sessionService!: SearchSessionService;

Expand Down Expand Up @@ -65,10 +72,17 @@ export class EnhancedDataServerPlugin implements Plugin<void, void, SetupDepende

const router = core.http.createRouter();
registerSessionRoutes(router);

this.sessionService.setup(core, {
taskManager: deps.taskManager,
});
}

public start(core: CoreStart) {
this.sessionService.start(core, this.initializerContext.config.create());
public start(core: CoreStart, { taskManager }: StartDependencies) {
this.sessionService.start(core, {
taskManager,
config$: this.initializerContext.config.create(),
});
}

public stop() {
Expand Down
Loading