-
Notifications
You must be signed in to change notification settings - Fork 8.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Security Solutions] Adds bsearch service to FTR e2e tests to reduce …
…flake, boilerplate, and technique choices (#116211) ## Summary Fixes flake tests of: #115918 #103273 #108640 #109447 #100630 #94535 #104260 Security solution has been using `bsearch` and has encountered flake in various forms. Different developers have been fixing the flake in a few odd ways (myself included) which aren't 100%. This PR introduces a once-in-for-all REST API retry service called `bsearch` which will query `bsearch` and if `bsearch` is not completed because of async occurring due to slower CI runtimes it will continuously call into the `bsearch` with the correct API to ensure it gets a complete response before returning. ## Usage Anyone can use this service like so: ```ts const bsearch = getService('bsearch'); const response = await bsearch.send<MyType>({ supertest, options: { defaultIndex: ['large_volume_dns_data'], } strategy: 'securitySolutionSearchStrategy', }); ``` If you're using a custom auth then you can set that beforehand like so: ```ts const bsearch = getService('bsearch'); const supertestWithoutAuth = getService('supertestWithoutAuth'); const supertest supertestWithoutAuth.auth(username, password); const response = await bsearch.send<MyType>({ supertest, options: { defaultIndex: ['large_volume_dns_data'], } strategy: 'securitySolutionSearchStrategy', }); ``` ## Misconceptions in the tests leading to flake * Can you just call the bsearch REST API and it will always return data first time? Not always true, as when CI slows down or data increases `bsearch` will give you back an async reference and then your test will blow up. * Can we wrap the REST API in `retry` to fix the flake? Not always but mostly true, as when CI slows down or data increases `bsearch` could return the async version continuously which could then fail your test. It's also tedious to tell everyone in code reviews to wrap everything in `retry` instead of just fixing it with a service as well as inform new people why we are constantly wrapping these tests in `retry`. * Can we manually parse the `bsearch` if it has `async` for each test? This is true but is error prone and I did this for one test and it's ugly and I had issues as I have to wrap 2 things in `retry` and test several conditions. Also it's harder for people to read the tests rather than just reading there is a service call. Also people in code reviews missed where I had bugs with it. Also lots of boiler plate. * Can we just increase the timeout with `wait_for_completion_timeout` and the tests will pass for sure then? Not true today but maybe true later, as this hasn't been added as plumbing yet. See this [open ticket](#107241). Even if it is and we increase the timeout to a very large number bsearch might return with an `async` or you might want to test the `async` path. Either way, if/when we add the ability we can increase it within 1 spot which is this service for everyone rather than going to each individual test to add it. If/when it's added if people don't use the bsearch service we can remove it later if we find this is deterministic enough and no one wants to test bsearch features with their strategies down the road. ## Manual test of bsearch service If you want to manually watch the bsearch operate as if the CI system is running slow or to cause an `async` manually you manually modify this setting here: https://github.com/elastic/kibana/blob/master/src/plugins/data/server/search/strategies/ese_search/request_utils.ts#L61 To be of a lower number such as `1ms` and then you will see it enter the `async` code within `bsearch` consistently ## Reference PRs We cannot set the wait_for_complete just yet #107241 so we decided this was the best way to reduce flake for testing for now. ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- Loading branch information
1 parent
c3f207a
commit ae7b5a9
Showing
20 changed files
with
819 additions
and
718 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,122 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0 and the Server Side Public License, v 1; you may not use this file except | ||
| * in compliance with, at your election, the Elastic License 2.0 or the Server | ||
| * Side Public License, v 1. | ||
| */ | ||
|
|
||
| import expect from '@kbn/expect'; | ||
| import request from 'superagent'; | ||
| import type SuperTest from 'supertest'; | ||
| import { IEsSearchResponse } from 'src/plugins/data/common'; | ||
| import { FtrProviderContext } from '../ftr_provider_context'; | ||
| import { RetryService } from './retry/retry'; | ||
|
|
||
| /** | ||
| * Function copied from here: | ||
| * test/api_integration/apis/search/bsearch.ts without the compress | ||
| * | ||
| * Splits the JSON lines from bsearch | ||
| */ | ||
| const parseBfetchResponse = (resp: request.Response): Array<Record<string, any>> => { | ||
| return resp.text | ||
| .trim() | ||
| .split('\n') | ||
| .map((item) => JSON.parse(item)); | ||
| }; | ||
|
|
||
| /** | ||
| * Function copied from here: | ||
| * x-pack/test/rule_registry/common/lib/authentication/spaces.ts | ||
| * @param spaceId The space id we want to utilize | ||
| */ | ||
| const getSpaceUrlPrefix = (spaceId?: string): string => { | ||
| return spaceId && spaceId !== 'default' ? `/s/${spaceId}` : ``; | ||
| }; | ||
|
|
||
| /** | ||
| * Options for the send method | ||
| */ | ||
| interface SendOptions { | ||
| supertest: SuperTest.SuperTest<SuperTest.Test>; | ||
| options: object; | ||
| strategy: string; | ||
| space?: string; | ||
| } | ||
|
|
||
| /** | ||
| * Bsearch factory which will return a new bsearch capable service that can reduce flake | ||
| * on the CI systems when they are under pressure and bsearch returns an async search | ||
| * response or a sync response. | ||
| * | ||
| * @example | ||
| * const supertest = getService('supertest'); | ||
| * const bsearch = getService('bsearch'); | ||
| * const response = await bsearch.send<MyType>({ | ||
| * supertest, | ||
| * options: { | ||
| * defaultIndex: ['large_volume_dns_data'], | ||
| * } | ||
| * strategy: 'securitySolutionSearchStrategy', | ||
| * }); | ||
| * expect(response).eql({ ... your value ... }); | ||
| */ | ||
| export const BSearchFactory = (retry: RetryService) => ({ | ||
| /** Send method to send in your supertest, url, options, and strategy name */ | ||
| send: async <T extends IEsSearchResponse>({ | ||
| supertest, | ||
| options, | ||
| strategy, | ||
| space, | ||
| }: SendOptions): Promise<T> => { | ||
| const spaceUrl = getSpaceUrlPrefix(space); | ||
| const { body } = await retry.try(async () => { | ||
| return supertest | ||
| .post(`${spaceUrl}/internal/search/${strategy}`) | ||
| .set('kbn-xsrf', 'true') | ||
| .send(options) | ||
| .expect(200); | ||
| }); | ||
|
|
||
| if (body.isRunning) { | ||
| const result = await retry.try(async () => { | ||
| const resp = await supertest | ||
| .post(`${spaceUrl}/internal/bsearch`) | ||
| .set('kbn-xsrf', 'true') | ||
| .send({ | ||
| batch: [ | ||
| { | ||
| request: { | ||
| id: body.id, | ||
| ...options, | ||
| }, | ||
| options: { | ||
| strategy, | ||
| }, | ||
| }, | ||
| ], | ||
| }) | ||
| .expect(200); | ||
| const [parsedResponse] = parseBfetchResponse(resp); | ||
| expect(parsedResponse.result.isRunning).equal(false); | ||
| return parsedResponse.result; | ||
| }); | ||
| return result; | ||
| } else { | ||
| return body; | ||
| } | ||
| }, | ||
| }); | ||
|
|
||
| /** | ||
| * Bsearch provider which will return a new bsearch capable service that can reduce flake | ||
| * on the CI systems when they are under pressure and bsearch returns an async search response | ||
| * or a sync response. | ||
| */ | ||
| export function BSearchProvider({ | ||
| getService, | ||
| }: FtrProviderContext): ReturnType<typeof BSearchFactory> { | ||
| const retry = getService('retry'); | ||
| return BSearchFactory(retry); | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.