Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/basic/tests/add_prepackaged_rules·ts - detection engine api security and spaces enabled add_prepackaged_rules creating prepackaged rules should be possible to call the API twice and the second time the number of rules installed should be zero #71867

Closed
kibanamachine opened this issue Jul 15, 2020 · 5 comments
Closed

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/basic/tests/add_prepackaged_rules·ts - detection engine api security and spaces enabled add_prepackaged_rules creating prepackaged rules should be possible to call the API twice and the second time the number of rules installed should be zero #71867

kibanamachine opened this issue Jul 15, 2020 · 5 comments
Assignees
@FrankHassanabad
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:SIEM v7.9.0

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Jul 15, 2020 •
edited
Loading

A test failed on a tracked branch

{ Error: expected 1 to sort of equal 0
    at Assertion.assert (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/kibana/packages/kbn-expect/expect.js:244:8)
    at Context.it (test/detection_engine_api_integration/basic/tests/add_prepackaged_rules.ts:99:41) actual: '1', expected: '0', showDiff: true }

First failure: Jenkins Build
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Jul 15, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-test-triage (failed-test)

@spalger spalger mentioned this issue Jul 15, 2020
Closed
@spalger
Copy link
Contributor

spalger commented Jul 15, 2020 •
edited
Loading

There is no automated issue yet, as this has not failed on a tracked branch but we're seeing widespread failures for it on PR's.

image

There is also a related failure in alerts_detection_rules_prebuilt.spec.ts.

@andrew-goldstein encountered both these failures on his PR: https://kibana-ci.elastic.co/job/elastic+kibana+pipeline-pull-request/61797/flowGraphTable/

skipped by @tylersmalley in #71814:

master/8.0: ddbfe53 + d74cd9e
7.x/7.10: 65c0dab + 670e730
7.9: 65c0dab + d77dbe4

@elasticmachine
Copy link
Contributor

Pinging @elastic/siem (Team:SIEM)

@kibanamachine
Copy link
Contributor Author

New failure: Jenkins Build

spalger added a commit that referenced this issue Jul 15, 2020
@spalger
skip flaky suite (#71867)
d74cd9e
spalger added a commit that referenced this issue Jul 15, 2020
@spalger
skip flaky suite (#71867)
670e730 
(cherry picked from commit d74cd9e)
spalger added a commit that referenced this issue Jul 15, 2020
@spalger
skip flaky suite (#71867)
d77dbe4 
(cherry picked from commit d74cd9e)
gmmorris added a commit to gmmorris/kibana that referenced this issue Jul 16, 2020
@gmmorris
Merge branch 'master' into alerting/consumer-based-rbac
a6989b5 
* master: (37 commits)
  [Lens] Handle failing existence check (elastic#70718)
  [Security Solution]Fix in-app links and popup window text (elastic#71403)
  [esArchiver] automatically retry if alias creation fails (elastic#71910)
  Move data stream index pattern creation test to xpack (elastic#71511)
  [Maps] Improve language for mvt card (elastic#71947)
  [Security][Detections] Unskip failing modal tests (elastic#71969)
  skip flaky suite (elastic#71987)
  skip flaky suite (elastic#71979)
  [Security Solution] [Detections] Revert "[Security Solution] [Detections] Fixes bug for determining when we hit max signals after filtering with lists (elastic#71768)" (elastic#71956)
  rename ilm policy to remove -default (elastic#71952)
  Adjust ordering of Management category apps to make Ingest Manager higher (elastic#71948)
  skip flaky suite (elastic#71971)
  skip flaky suite (elastic#71951)
  [kbn/optimizer] ignore compressed files when reporting stats (elastic#71940)
  skip flaky suite (elastic#71867)
  [ML] Fix new job with must_not saved search (elastic#71831)
  [Resolver] Fix bug where process detail panel doesn't show up (elastic#71754)
  Cleanup (elastic#71849)
  [Resolver] aria-level and aria-flowto support enhancements (elastic#71887)
  skip flaky suite (elastic#71304)
  ...
gmmorris added a commit to gmmorris/kibana that referenced this issue Jul 16, 2020
@gmmorris
Merge branch 'alerting/consumer-based-rbac' into alerting/management-…
47a4b93 
…feature-privileges

* alerting/consumer-based-rbac: (491 commits)
  [Lens] Handle failing existence check (elastic#70718)
  [Security Solution]Fix in-app links and popup window text (elastic#71403)
  [esArchiver] automatically retry if alias creation fails (elastic#71910)
  Move data stream index pattern creation test to xpack (elastic#71511)
  [Maps] Improve language for mvt card (elastic#71947)
  [Security][Detections] Unskip failing modal tests (elastic#71969)
  skip flaky suite (elastic#71987)
  skip flaky suite (elastic#71979)
  [Security Solution] [Detections] Revert "[Security Solution] [Detections] Fixes bug for determining when we hit max signals after filtering with lists (elastic#71768)" (elastic#71956)
  rename ilm policy to remove -default (elastic#71952)
  Adjust ordering of Management category apps to make Ingest Manager higher (elastic#71948)
  skip flaky suite (elastic#71971)
  skip flaky suite (elastic#71951)
  [kbn/optimizer] ignore compressed files when reporting stats (elastic#71940)
  skip flaky suite (elastic#71867)
  [ML] Fix new job with must_not saved search (elastic#71831)
  [Resolver] Fix bug where process detail panel doesn't show up (elastic#71754)
  Cleanup (elastic#71849)
  [Resolver] aria-level and aria-flowto support enhancements (elastic#71887)
  skip flaky suite (elastic#71304)
  ...
@FrankHassanabad FrankHassanabad mentioned this issue Jul 20, 2020
Merged
1 task
FrankHassanabad added a commit that referenced this issue Jul 20, 2020
@FrankHassanabad
[SIEM][Detection Engine] Reduces flakiness within the tests using wai…
3ccdd79 
…tFor() and un-skips the tests (#72479)

## Summary

* Utilizes the `waitFor` so that the tests are less flaky and more resilient
* Unskips a test that was due to a regression within Elastic Search that should be fixed now
* #71867
* #71814
* #71612

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
FrankHassanabad added a commit to FrankHassanabad/kibana that referenced this issue Jul 20, 2020
@FrankHassanabad
[SIEM][Detection Engine] Reduces flakiness within the tests using wai…
55e38f1 
…tFor() and un-skips the tests (elastic#72479)

## Summary

* Utilizes the `waitFor` so that the tests are less flaky and more resilient
* Unskips a test that was due to a regression within Elastic Search that should be fixed now
* elastic#71867
* elastic#71814
* elastic#71612

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
FrankHassanabad added a commit to FrankHassanabad/kibana that referenced this issue Jul 20, 2020
@FrankHassanabad
[SIEM][Detection Engine] Reduces flakiness within the tests using wai…
6108e80 
…tFor() and un-skips the tests (elastic#72479)

## Summary

* Utilizes the `waitFor` so that the tests are less flaky and more resilient
* Unskips a test that was due to a regression within Elastic Search that should be fixed now
* elastic#71867
* elastic#71814
* elastic#71612

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
FrankHassanabad added a commit that referenced this issue Jul 20, 2020
@FrankHassanabad
[SIEM][Detection Engine] Reduces flakiness within the tests using wai…
6130e8b 
…tFor() and un-skips the tests (#72479) (#72508)

## Summary

* Utilizes the `waitFor` so that the tests are less flaky and more resilient
* Unskips a test that was due to a regression within Elastic Search that should be fixed now
* #71867
* #71814
* #71612

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
FrankHassanabad added a commit that referenced this issue Jul 20, 2020
@FrankHassanabad
[SIEM][Detection Engine] Reduces flakiness within the tests using wai…
e1a7db0 
…tFor() and un-skips the tests (#72479) (#72507)

## Summary

* Utilizes the `waitFor` so that the tests are less flaky and more resilient
* Unskips a test that was due to a regression within Elastic Search that should be fixed now
* #71867
* #71814
* #71612

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
@FrankHassanabad FrankHassanabad self-assigned this Jul 21, 2020
@FrankHassanabad
Copy link
Contributor

Should be fixed with changes to utilizing wait/waitFor techniques which do polling.

@MindyRS MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Oct 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@FrankHassanabad FrankHassanabad

Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:SIEM v7.9.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@FrankHassanabad @spalger @MindyRS @elasticmachine @kibanamachine