[SECURITY SOLUTION] [Detections] Add / Update e2e tests to ensure initial rule runs are successful #68441
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dhurley14
added
Feature:Detection Rules
|
Pinging @elastic/siem (Team:SIEM) |
dhurley14
added
the
release_note:skip
rylnd
approved these changes
Jun 5, 2020
There was a problem hiding this comment.
Checked these out, they look great! My one gripe is that a failure here wouldn't be immediately actionable: it would be unclear whether the test failed:
- due to a regression (of what? the test makes no mention of what could regress)
- due to task manager not completing (this would also result in 'going to run', right?)
I'ma approve because this will do what it says: fail on a regression, but I'm wondering if we can make that more clear/actionable somehow.
… rules are created, instead of just 'going to run'
…pecific regression to be on the lookout for if these start failing
dhurley14
force-pushed
the
narrow-e2e-status-tests
branch
from
a3a29c2
to
8d3640f
Compare
rylnd
reviewed
Jun 8, 2020
| @@ -65,6 +68,46 @@ export default ({ getService }: FtrProviderContext) => { | |||
| expect(bodyToCompare).to.eql(getSimpleRuleOutput()); | |||
| }); | |||
|
|
|||
| /* | |||
| This test is to ensure no future regressions introduced by the following scenario | |||
|
jenkins test this |
💛 Build succeeded, but was flaky
Test FailuresKibana Pipeline / kibana-oss-agent / Chrome UI Functional Tests.test/functional/apps/dashboard/dashboard_snapshots·js.dashboard app using current data dashboard snapshots compare area chart snapshotStandard OutStack TraceHistory
To update your PR or re-run it, just comment with: |
dhurley14
added a commit
to dhurley14/kibana
that referenced
this pull request
Jun 9, 2020
…tial rule runs are successful (elastic#68441) * adds/modifies e2e tests to ensure find_status returns succeeded after rules are created, instead of just 'going to run' * add documentation around newly created e2e tests explaining bug and specific regression to be on the lookout for if these start failing
gmmorris
added a commit
to gmmorris/kibana
that referenced
this pull request
Jun 9, 2020
* master: (37 commits) [DOCS] Adds documentation for drilldowns (elastic#68122) [Telemetry] telemetry.sendUsageFrom: "server" by default (elastic#68071) [ML] Transforms: Support sub-aggregations (elastic#68306) Fixed pre-configured docs link points to the wrong page and functional tests configs (elastic#68606) [Ingest Manager] Update queries from `stream.*` to `dataset.*` (elastic#68322) Enable Watcher by default to fix bug in which Watcher doesn't render in the side nav (elastic#68602) Convert Index Templates API routes to snakecase. (elastic#68463) [SECURITY SOLUTION] [Detections] Add / Update e2e tests to ensure initial rule runs are successful (elastic#68441) [Ingest] OpenAPI spec file (elastic#68323) chore(NA): skip apis Endpoint plugin Endpoint policy api (elastic#68638) bumping makelogs version to v6.0.0 (elastic#66163) [SIEM] Add create template button (elastic#66613) Bump websocket-extensions from 0.1.3 to 0.1.4 (elastic#68414) [ML] Sample data modules - use event.dataset instead of index name (elastic#68538) [ML] Functional tests - fix job validation API test with maxModelMemoryLimit (elastic#68501) [ML] Functional tests - stabilize DFA job creation (elastic#68495) [TSVB] Allows the user to change the tooltip mode (elastic#67775) chore(NA): skip apis Endpoint plugin Endpoint alert API when data is in elasticsearch (elastic#68613) chore(NA): skip endpoint Endpoint Alert Page: when es has data and user has navigated to the page (elastic#68596) [SIEM][Detection Engine] Converts from joi to use io-ts and moves the types to common (elastic#68127) ...
dhurley14
added a commit
that referenced
this pull request
Jun 9, 2020
…re initial rule runs are successful (#68441) (#68643) * adds/modifies e2e tests to ensure find_status returns succeeded after rules are created, instead of just 'going to run' * add documentation around newly created e2e tests explaining bug and specific regression to be on the lookout for if these start failing
MindyRS
added
the
Team: SecuritySolution
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
adds/modifies e2e tests to ensure find_status returns 'succeeded' after rules are created, instead of just 'going to run'.
Checklist
Delete any items that are not applicable to this PR.
For maintainers