[EDR Workflows][tests] Enable endpoint exceptions integration test #183471
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gergoabraham
added
release_note:skip
|
/ci |
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#5977[✅] x-pack/test/security_solution_endpoint/integrations.config.ts: 25/25 tests passed. |
gergoabraham
force-pushed
the
enable-endpoint-exceptions-integration-test
branch
from
453d614
to
43b5845
Compare
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#5979[✅] x-pack/test/security_solution_endpoint/integrations.config.ts: 25/25 tests passed. |
This reverts commit 8356bcf.
|
Pinging @elastic/security-defend-workflows (Team:Defend Workflows) |
gergoabraham
requested review from
dasansol92
and removed request for
szwarckonrad
|
/ci |
|
@elasticmachine merge upstream |
|
@elasticmachine merge upstream |
|
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: |
dasansol92
approved these changes
May 23, 2024
1 task
gergoabraham
added a commit
that referenced
this pull request
Oct 24, 2024
## Summary closes #173184 closes #173441 closes #196003 This PR tries to improve on the `StaleElementReferenceError` happening in Endpoint Exception tests. This error is thrown if an element has already been removed from the DOM when trying to perform an action on it. For some reference, see #140427 Improvements: - the part that was failing is wrapped inside the `retryOnStale` helper: 602f229 **note:** actually the test fails have started in December, 2023, but the line where the fail was in the last test runs were added in May, 2024 (#183471). unfortunately, the log artifacts from 2023 are already removed from Buildkite, so no certainty on what happened back then - another suspicious part was wrapped as well: ec8c5cf and e5245ad - and as an extra, wait for page load: 7cd867f flaky 50/50 ✅ but this doesn't mean much, as this issue happens quite rarely ¯\\(◉‿◉)/¯ ### Checklist Delete any items that are not applicable to this PR. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 24, 2024
…#197457) ## Summary closes elastic#173184 closes elastic#173441 closes elastic#196003 This PR tries to improve on the `StaleElementReferenceError` happening in Endpoint Exception tests. This error is thrown if an element has already been removed from the DOM when trying to perform an action on it. For some reference, see elastic#140427 Improvements: - the part that was failing is wrapped inside the `retryOnStale` helper: 602f229 **note:** actually the test fails have started in December, 2023, but the line where the fail was in the last test runs were added in May, 2024 (elastic#183471). unfortunately, the log artifacts from 2023 are already removed from Buildkite, so no certainty on what happened back then - another suspicious part was wrapped as well: ec8c5cf and e5245ad - and as an extra, wait for page load: 7cd867f flaky 50/50 ✅ but this doesn't mean much, as this issue happens quite rarely ¯\\(◉‿◉)/¯ ### Checklist Delete any items that are not applicable to this PR. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit f151e2c)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 24, 2024
…#197457) ## Summary closes elastic#173184 closes elastic#173441 closes elastic#196003 This PR tries to improve on the `StaleElementReferenceError` happening in Endpoint Exception tests. This error is thrown if an element has already been removed from the DOM when trying to perform an action on it. For some reference, see elastic#140427 Improvements: - the part that was failing is wrapped inside the `retryOnStale` helper: 602f229 **note:** actually the test fails have started in December, 2023, but the line where the fail was in the last test runs were added in May, 2024 (elastic#183471). unfortunately, the log artifacts from 2023 are already removed from Buildkite, so no certainty on what happened back then - another suspicious part was wrapped as well: ec8c5cf and e5245ad - and as an extra, wait for page load: 7cd867f flaky 50/50 ✅ but this doesn't mean much, as this issue happens quite rarely ¯\\(◉‿◉)/¯ ### Checklist Delete any items that are not applicable to this PR. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> (cherry picked from commit f151e2c)
kibanamachine
added a commit
that referenced
this pull request
Oct 24, 2024
…197457) (#197662) # Backport This will backport the following commits from `main` to `8.x`: - [[EDR Workflows] Unskip and fix flaky endpoint exceptions FTR (#197457)](#197457) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gergő Ábrahám","email":"gergo.abraham@elastic.co"},"sourceCommit":{"committedDate":"2024-10-24T14:27:08Z","message":"[EDR Workflows] Unskip and fix flaky endpoint exceptions FTR (#197457)\n\n## Summary\r\n\r\ncloses #173184 \r\ncloses #173441\r\ncloses #196003\r\n\r\nThis PR tries to improve on the `StaleElementReferenceError` happening\r\nin Endpoint Exception tests.\r\n\r\nThis error is thrown if an element has already been removed from the DOM\r\nwhen trying to perform an action on it. For some reference, see\r\nhttps://github.com//pull/140427\r\n\r\nImprovements:\r\n- the part that was failing is wrapped inside the `retryOnStale` helper:\r\n602f2294fddb9bee8b69ebf2fd8382e9f025d59d\r\n**note:** actually the test fails have started in December, 2023, but\r\nthe line where the fail was in the last test runs were added in May,\r\n2024 (#183471). unfortunately, the\r\nlog artifacts from 2023 are already removed from Buildkite, so no\r\ncertainty on what happened back then\r\n- another suspicious part was wrapped as well:\r\nec8c5cfd94812c8e5b357e00aac8bfae93ceecf4 and\r\ne5245ad010a02527105a56973465a25feb52ec85\r\n- and as an extra, wait for page load:\r\n7cd867fcb9489b24e79066dce750a2381af93d7d\r\n\r\nflaky 50/50 ✅ but this doesn't mean much, as this issue happens quite\r\nrarely ¯\\\\(◉‿◉)/¯\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"f151e2ccaa55cc5e13740f49e88c323c0e1d8f6d","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Defend Workflows","backport:prev-major"],"title":"[EDR Workflows] Unskip and fix flaky endpoint exceptions FTR","number":197457,"url":"https://github.com/elastic/kibana/pull/197457","mergeCommit":{"message":"[EDR Workflows] Unskip and fix flaky endpoint exceptions FTR (#197457)\n\n## Summary\r\n\r\ncloses #173184 \r\ncloses #173441\r\ncloses #196003\r\n\r\nThis PR tries to improve on the `StaleElementReferenceError` happening\r\nin Endpoint Exception tests.\r\n\r\nThis error is thrown if an element has already been removed from the DOM\r\nwhen trying to perform an action on it. For some reference, see\r\nhttps://github.com//pull/140427\r\n\r\nImprovements:\r\n- the part that was failing is wrapped inside the `retryOnStale` helper:\r\n602f2294fddb9bee8b69ebf2fd8382e9f025d59d\r\n**note:** actually the test fails have started in December, 2023, but\r\nthe line where the fail was in the last test runs were added in May,\r\n2024 (#183471). unfortunately, the\r\nlog artifacts from 2023 are already removed from Buildkite, so no\r\ncertainty on what happened back then\r\n- another suspicious part was wrapped as well:\r\nec8c5cfd94812c8e5b357e00aac8bfae93ceecf4 and\r\ne5245ad010a02527105a56973465a25feb52ec85\r\n- and as an extra, wait for page load:\r\n7cd867fcb9489b24e79066dce750a2381af93d7d\r\n\r\nflaky 50/50 ✅ but this doesn't mean much, as this issue happens quite\r\nrarely ¯\\\\(◉‿◉)/¯\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"f151e2ccaa55cc5e13740f49e88c323c0e1d8f6d"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197457","number":197457,"mergeCommit":{"message":"[EDR Workflows] Unskip and fix flaky endpoint exceptions FTR (#197457)\n\n## Summary\r\n\r\ncloses #173184 \r\ncloses #173441\r\ncloses #196003\r\n\r\nThis PR tries to improve on the `StaleElementReferenceError` happening\r\nin Endpoint Exception tests.\r\n\r\nThis error is thrown if an element has already been removed from the DOM\r\nwhen trying to perform an action on it. For some reference, see\r\nhttps://github.com//pull/140427\r\n\r\nImprovements:\r\n- the part that was failing is wrapped inside the `retryOnStale` helper:\r\n602f2294fddb9bee8b69ebf2fd8382e9f025d59d\r\n**note:** actually the test fails have started in December, 2023, but\r\nthe line where the fail was in the last test runs were added in May,\r\n2024 (#183471). unfortunately, the\r\nlog artifacts from 2023 are already removed from Buildkite, so no\r\ncertainty on what happened back then\r\n- another suspicious part was wrapped as well:\r\nec8c5cfd94812c8e5b357e00aac8bfae93ceecf4 and\r\ne5245ad010a02527105a56973465a25feb52ec85\r\n- and as an extra, wait for page load:\r\n7cd867fcb9489b24e79066dce750a2381af93d7d\r\n\r\nflaky 50/50 ✅ but this doesn't mean much, as this issue happens quite\r\nrarely ¯\\\\(◉‿◉)/¯\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"f151e2ccaa55cc5e13740f49e88c323c0e1d8f6d"}}]}] BACKPORT--> Co-authored-by: Gergő Ábrahám <gergo.abraham@elastic.co>
kibanamachine
added a commit
that referenced
this pull request
Oct 24, 2024
…197457) (#197661) # Backport This will backport the following commits from `main` to `8.16`: - [[EDR Workflows] Unskip and fix flaky endpoint exceptions FTR (#197457)](#197457) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Gergő Ábrahám","email":"gergo.abraham@elastic.co"},"sourceCommit":{"committedDate":"2024-10-24T14:27:08Z","message":"[EDR Workflows] Unskip and fix flaky endpoint exceptions FTR (#197457)\n\n## Summary\r\n\r\ncloses #173184 \r\ncloses #173441\r\ncloses #196003\r\n\r\nThis PR tries to improve on the `StaleElementReferenceError` happening\r\nin Endpoint Exception tests.\r\n\r\nThis error is thrown if an element has already been removed from the DOM\r\nwhen trying to perform an action on it. For some reference, see\r\nhttps://github.com//pull/140427\r\n\r\nImprovements:\r\n- the part that was failing is wrapped inside the `retryOnStale` helper:\r\n602f2294fddb9bee8b69ebf2fd8382e9f025d59d\r\n**note:** actually the test fails have started in December, 2023, but\r\nthe line where the fail was in the last test runs were added in May,\r\n2024 (#183471). unfortunately, the\r\nlog artifacts from 2023 are already removed from Buildkite, so no\r\ncertainty on what happened back then\r\n- another suspicious part was wrapped as well:\r\nec8c5cfd94812c8e5b357e00aac8bfae93ceecf4 and\r\ne5245ad010a02527105a56973465a25feb52ec85\r\n- and as an extra, wait for page load:\r\n7cd867fcb9489b24e79066dce750a2381af93d7d\r\n\r\nflaky 50/50 ✅ but this doesn't mean much, as this issue happens quite\r\nrarely ¯\\\\(◉‿◉)/¯\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"f151e2ccaa55cc5e13740f49e88c323c0e1d8f6d","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Defend Workflows","backport:prev-major"],"title":"[EDR Workflows] Unskip and fix flaky endpoint exceptions FTR","number":197457,"url":"https://github.com/elastic/kibana/pull/197457","mergeCommit":{"message":"[EDR Workflows] Unskip and fix flaky endpoint exceptions FTR (#197457)\n\n## Summary\r\n\r\ncloses #173184 \r\ncloses #173441\r\ncloses #196003\r\n\r\nThis PR tries to improve on the `StaleElementReferenceError` happening\r\nin Endpoint Exception tests.\r\n\r\nThis error is thrown if an element has already been removed from the DOM\r\nwhen trying to perform an action on it. For some reference, see\r\nhttps://github.com//pull/140427\r\n\r\nImprovements:\r\n- the part that was failing is wrapped inside the `retryOnStale` helper:\r\n602f2294fddb9bee8b69ebf2fd8382e9f025d59d\r\n**note:** actually the test fails have started in December, 2023, but\r\nthe line where the fail was in the last test runs were added in May,\r\n2024 (#183471). unfortunately, the\r\nlog artifacts from 2023 are already removed from Buildkite, so no\r\ncertainty on what happened back then\r\n- another suspicious part was wrapped as well:\r\nec8c5cfd94812c8e5b357e00aac8bfae93ceecf4 and\r\ne5245ad010a02527105a56973465a25feb52ec85\r\n- and as an extra, wait for page load:\r\n7cd867fcb9489b24e79066dce750a2381af93d7d\r\n\r\nflaky 50/50 ✅ but this doesn't mean much, as this issue happens quite\r\nrarely ¯\\\\(◉‿◉)/¯\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"f151e2ccaa55cc5e13740f49e88c323c0e1d8f6d"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/197457","number":197457,"mergeCommit":{"message":"[EDR Workflows] Unskip and fix flaky endpoint exceptions FTR (#197457)\n\n## Summary\r\n\r\ncloses #173184 \r\ncloses #173441\r\ncloses #196003\r\n\r\nThis PR tries to improve on the `StaleElementReferenceError` happening\r\nin Endpoint Exception tests.\r\n\r\nThis error is thrown if an element has already been removed from the DOM\r\nwhen trying to perform an action on it. For some reference, see\r\nhttps://github.com//pull/140427\r\n\r\nImprovements:\r\n- the part that was failing is wrapped inside the `retryOnStale` helper:\r\n602f2294fddb9bee8b69ebf2fd8382e9f025d59d\r\n**note:** actually the test fails have started in December, 2023, but\r\nthe line where the fail was in the last test runs were added in May,\r\n2024 (#183471). unfortunately, the\r\nlog artifacts from 2023 are already removed from Buildkite, so no\r\ncertainty on what happened back then\r\n- another suspicious part was wrapped as well:\r\nec8c5cfd94812c8e5b357e00aac8bfae93ceecf4 and\r\ne5245ad010a02527105a56973465a25feb52ec85\r\n- and as an extra, wait for page load:\r\n7cd867fcb9489b24e79066dce750a2381af93d7d\r\n\r\nflaky 50/50 ✅ but this doesn't mean much, as this issue happens quite\r\nrarely ¯\\\\(◉‿◉)/¯\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [x] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>","sha":"f151e2ccaa55cc5e13740f49e88c323c0e1d8f6d"}}]}] BACKPORT--> Co-authored-by: Gergő Ábrahám <gergo.abraham@elastic.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
fixes #176009
Improving and enabling Endpoint Exceptions integration tests.
The test failed on CI because it looked at alerts in the default time window (
today), and midnight has passed during the test run. ^^So with this change the test looks at alerts from the last 24 hours instead.
Flaky runner:
note: I had some transient issues locally with alerts not showing up at all, but it happened twice in a row, and never again, and only locally, so let's forget about it for now.