Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/basic/tests/find_statuses·ts - detection engine api security and spaces enabled find_statuses "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added" #69632

kibanamachine · 2020-06-21T02:49:19Z

A test failed on a tracked branch

{ ResponseError: Response Error
    at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25)
    at endReadableNT (_stream_readable.js:1145:12)
    at process._tickCallback (internal/process/next_tick.js:63:19)
  name: 'ResponseError',
  meta:
   { body:
      { took: 7,
        timed_out: false,
        total: 1,
        deleted: 0,
        batches: 1,
        version_conflicts: 1,
        noops: 0,
        retries: [Object],
        throttled_millis: 0,
        requests_per_second: -1,
        throttled_until_millis: 0,
        failures: [Array] },
     statusCode: 409,
     headers:
      { 'content-type': 'application/json; charset=UTF-8',
        'content-length': '663' },
     warnings: null,
     meta:
      { context: null,
        request: [Object],
        name: 'elasticsearch-js',
        connection: [Object],
        attempts: 0,
        aborted: false } } }

First failure: Jenkins Build

The text was updated successfully, but these errors were encountered:

elasticmachine · 2020-06-21T02:49:21Z

Pinging @elastic/kibana-test-triage (failed-test)

elasticmachine · 2020-06-22T11:48:27Z

Pinging @elastic/siem (Team:SIEM)

kibanamachine · 2020-06-25T15:45:23Z

New failure: Jenkins Build

kibanamachine · 2020-06-30T09:36:51Z

New failure: Jenkins Build

kibanamachine · 2020-07-01T00:05:21Z

New failure: Jenkins Build

FrankHassanabad · 2020-07-01T01:33:51Z

More of the errors from one of the builds. It looks almost like another test might be interfering?

03:19:25                 └-> "before each" hook: globalbefore each
03:19:25               └-> should request documents when zoomed to smaller regions showing less data
03:19:25                 └-> "before each" hook: globalbefore each
03:19:26                 └- ✓ pass  (1.0s) "saved objects security and spaces enabled _create user with all at the space within the default space with overwrite enabled should return 200 success [sharedtype/new-sharedtype-id]"
03:19:26               └-> should return 200 success [globaltype/new-globaltype-id]
03:19:26                 └-> "before each" hook: global before each
03:19:27                 └- ✓ pass  (1.0s) "saved objects security and spaces enabled _create user with all at the space within the default space with overwrite enabled should return 200 success [globaltype/new-globaltype-id]"
03:19:27               └-> should return 403 forbidden [hiddentype/any]
03:19:27                 └-> "before each" hook: global before each
03:19:27                 └- ✓ pass  (50ms) "saved objects security and spaces enabled _create user with all at the space within the default space with overwrite enabled should return 403 forbidden [hiddentype/any]"
03:19:27               └-> "after all" hook
03:19:27               └-> "after all" hook
03:19:27             └-: superuser within the default space with overwrite enabled
03:19:27               └-> "before all" hook
03:19:27               └-> "before all" hook
03:19:27                 │ proc [kibana]   log   [09:19:27.513] [info][savedobjects-service] Creating index .kibana_2.
03:19:27                 │ proc [kibana]   log   [09:19:27.600] [info][savedobjects-service] Reindexing .kibana o .kibana_1
03:19:27               │ERROR browser[SEVERE] http://localhost:61111/34162/bundles/core/core.entry.js 75:261772 TypeError: Faied to fetch
03:19:27                 │ proc [kibana]   log   [09:19:27.971] [info][savedobjects-service] Migrating .kibana_1 saved objects to .kibana_2
03:19:28                 │ proc [kibana]   log   [09:19:28.162] [info][savedobjects-service] Pointing alias .kibana to .kibana_2.
03:19:28                 │ proc [kibana]   log   [09:19:28.239] [info][savedobjects-service] Finished in 727ms.
03:19:28               └-> should return 200 success [isolatedtype/defaultspace-isolatedtype-id]
03:19:28                 └-> "before each" hook: global before each
03:19:28                 └- ✓ pass  (331ms) "saved objects security and spaces enabled _create superuser within the default space with overwrite enabled should return 200 success [isolatedtype/defaultspace-isolatedtype-id]"
03:19:28               └-> should return 200 success [isolatedtype/space1-isolatedtype-id]
03:19:28                 └-> "before each" hook: globalbefore each
03:19:29               └- ✓ pass  (4.5s) "detection engine api security and spaces enabled find_statuses should return a single rule status when a single rule is loaded from a find status with defaults added"
03:19:29             └-> "after each" hook
03:19:29  
03:19:29    1) detection engine api security and spaces enabled
03:19:29         find_statuses
03:19:29           "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added":
03:19:29       ResponseError: Response Error
03:19:29        at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25)
03:19:29        at endReadableNT (_stream_readable.js:1145:12)
03:19:29        at process._tickCallback (internal/process/next_tick.js:63:19)
03:19:29  
03:19:29               └- ✖ fail: "detection engine api security and spaces enabled find_statuses "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added""
03:19:29               │
03:19:29               └-> "after all" hook
03:19:29             └-> "after all" hook
03:19:29

kibanamachine · 2020-07-02T22:53:22Z

New failure: Jenkins Build

kibanamachine · 2020-07-03T12:12:22Z

New failure: Jenkins Build

kibanamachine · 2020-07-04T04:11:07Z

New failure: Jenkins Build

kibanamachine · 2020-07-04T09:43:22Z

New failure: Jenkins Build

kibanamachine · 2020-07-05T03:45:00Z

New failure: Jenkins Build

kibanamachine · 2020-07-06T02:59:35Z

New failure: Jenkins Build

kibanamachine · 2020-07-06T06:03:10Z

New failure: Jenkins Build

kibanamachine · 2020-07-06T08:48:29Z

New failure: Jenkins Build

(cherry picked from commit 43c6fbf)

spalger · 2020-07-08T21:52:08Z

This test is clearly very flaky, and just failed in a snapshot verification, so I'm skipping for now

master: 43c6fbf
7.x/7.9: 9a75c48

* master: (39 commits) [APM] Add warning to notify user about legacy ML jobs (elastic#71030) updates consumer to siem (elastic#71117) Index pattern creation flow - fix spelling (elastic#71192) [Security Solution][Endpoint] User Manifest Cleanup + Artifact Compression (elastic#70759) [SECURITY] Rearrange rule name's column in Alert Table (elastic#71020) [SECURITY] Alerts back to Detections (elastic#71142) [Security Solution][Exceptions Builder] - Fixes operator selection bug (elastic#71178) [SIEM][Detection Engine] Speeds up value list imports by enabling streaming of files. [APM] Update ML job ID in data telemetry tasks (elastic#71044) [Resolver] Remove `currentPanelView` selector (elastic#71154) add meta.managed to index templates (elastic#71135) Clarify trial subscription levels (elastic#70900) [Security Solution] fix panel links (elastic#71148) skip flaky suite (elastic#69632) skip suite failing ES Promotion (elastic#71018) [ML] DF Analytics: add results field to wizard and show regression stats (elastic#70893) [SIEM] update wordings (elastic#71119) [SECURITY SOLUTION] Rename to hosts and administration (elastic#70913) [ML] Improvements for urlState hook. (elastic#70576) Removing uptime guide (elastic#71124) ...

## Summary * #69632 * Adds a retry loop in case of a network outage/issue which should increase the chances of success * If there is still an issue after the 20th try, then it moves on and there is a high likelihood the tests will continue without issues. * Adds console logging statements so we know if this flakiness happens again a bit more insight into why the network is behaving the way it is. * Helps prevent the other tests from being skipped in the future due to bad networking issues. The errors that were coming back from the failed tests are in the `afterEach` and look to be network related or another test interfering: ```ts 1) detection engine api security and spaces enabled 01:59:54 find_statuses 01:59:54 "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added": 01:59:54 ResponseError: Response Error 01:59:54 at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25) 01:59:54 at endReadableNT (_stream_readable.js:1145:12) 01:59:54 at process._tickCallback (internal/process/next_tick.js:63:19) 01:59:54 01:59:54 └- ✖ fail: "detection engine api security and spaces enabled find_statuses "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added"" 01:59:54 │ 01:59:54 └-> "after all" hook 01:59:54 └-> "after all" hook 01:59:54 │ 01:59:54 │42 passing (2.0m) 01:59:54 │1 failing ``` So this should fix it to where the afterEach calls try up to 20 times before giving up and then on giving up they move on with the hope a different test doesn't fail. ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

## Summary * elastic#69632 * Adds a retry loop in case of a network outage/issue which should increase the chances of success * If there is still an issue after the 20th try, then it moves on and there is a high likelihood the tests will continue without issues. * Adds console logging statements so we know if this flakiness happens again a bit more insight into why the network is behaving the way it is. * Helps prevent the other tests from being skipped in the future due to bad networking issues. The errors that were coming back from the failed tests are in the `afterEach` and look to be network related or another test interfering: ```ts 1) detection engine api security and spaces enabled 01:59:54 find_statuses 01:59:54 "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added": 01:59:54 ResponseError: Response Error 01:59:54 at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25) 01:59:54 at endReadableNT (_stream_readable.js:1145:12) 01:59:54 at process._tickCallback (internal/process/next_tick.js:63:19) 01:59:54 01:59:54 └- ✖ fail: "detection engine api security and spaces enabled find_statuses "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added"" 01:59:54 │ 01:59:54 └-> "after all" hook 01:59:54 └-> "after all" hook 01:59:54 │ 01:59:54 │42 passing (2.0m) 01:59:54 │1 failing ``` So this should fix it to where the afterEach calls try up to 20 times before giving up and then on giving up they move on with the hope a different test doesn't fail. ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

## Summary * #69632 * Adds a retry loop in case of a network outage/issue which should increase the chances of success * If there is still an issue after the 20th try, then it moves on and there is a high likelihood the tests will continue without issues. * Adds console logging statements so we know if this flakiness happens again a bit more insight into why the network is behaving the way it is. * Helps prevent the other tests from being skipped in the future due to bad networking issues. The errors that were coming back from the failed tests are in the `afterEach` and look to be network related or another test interfering: ```ts 1) detection engine api security and spaces enabled 01:59:54 find_statuses 01:59:54 "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added": 01:59:54 ResponseError: Response Error 01:59:54 at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25) 01:59:54 at endReadableNT (_stream_readable.js:1145:12) 01:59:54 at process._tickCallback (internal/process/next_tick.js:63:19) 01:59:54 01:59:54 └- ✖ fail: "detection engine api security and spaces enabled find_statuses "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added"" 01:59:54 │ 01:59:54 └-> "after all" hook 01:59:54 └-> "after all" hook 01:59:54 │ 01:59:54 │42 passing (2.0m) 01:59:54 │1 failing ``` So this should fix it to where the afterEach calls try up to 20 times before giving up and then on giving up they move on with the hope a different test doesn't fail. ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

FrankHassanabad · 2020-07-13T14:29:47Z

This should be fixed by: #71347

As it looked like all the issues were in afterEach with a few different networking issues, one of which was an Elastic Conflict, another which looked like it was networking connection problems. So 🤞 I am really hoping this does not cause other problems with builds failing at this point.

Going to close as fixed.

kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Jun 21, 2020

timroes added Team:SIEM triage_needed labels Jun 22, 2020

spalger added a commit that referenced this issue Jul 8, 2020

skip flaky suite (#69632)

43c6fbf

spalger added a commit that referenced this issue Jul 8, 2020

skip flaky suite (#69632)

9a75c48

(cherry picked from commit 43c6fbf)

spalger added blocker skipped-test v7.9.0 labels Jul 8, 2020

FrankHassanabad self-assigned this Jul 9, 2020

FrankHassanabad mentioned this issue Jul 9, 2020

[SIEM][Detection Engine] Fixes skipped tests #71347

Merged

1 task

FrankHassanabad closed this as completed Jul 13, 2020

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Oct 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kibanamachine commented Jun 21, 2020 •

edited

Loading

elasticmachine commented Jun 21, 2020

elasticmachine commented Jun 22, 2020

kibanamachine commented Jun 25, 2020

kibanamachine commented Jun 30, 2020

kibanamachine commented Jul 1, 2020

FrankHassanabad commented Jul 1, 2020

kibanamachine commented Jul 2, 2020

kibanamachine commented Jul 3, 2020

kibanamachine commented Jul 4, 2020

kibanamachine commented Jul 4, 2020

kibanamachine commented Jul 5, 2020

kibanamachine commented Jul 6, 2020

kibanamachine commented Jul 6, 2020

kibanamachine commented Jul 6, 2020

spalger commented Jul 8, 2020

FrankHassanabad commented Jul 13, 2020

Comments

kibanamachine commented Jun 21, 2020 • edited Loading

elasticmachine commented Jun 21, 2020

elasticmachine commented Jun 22, 2020

kibanamachine commented Jun 25, 2020

kibanamachine commented Jun 30, 2020

kibanamachine commented Jul 1, 2020

FrankHassanabad commented Jul 1, 2020

kibanamachine commented Jul 2, 2020

kibanamachine commented Jul 3, 2020

kibanamachine commented Jul 4, 2020

kibanamachine commented Jul 4, 2020

kibanamachine commented Jul 5, 2020

kibanamachine commented Jul 6, 2020

kibanamachine commented Jul 6, 2020

kibanamachine commented Jul 6, 2020

spalger commented Jul 8, 2020

FrankHassanabad commented Jul 13, 2020

kibanamachine commented Jun 21, 2020 •

edited

Loading