[SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting #42804

andrew-goldstein · 2019-08-06T23:41:27Z

[SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting

This PR contains enhancements to the Cypress end-to-end (e2e) testing support in the SIEM app, as discussed with @spalger

Note: this PR depends on test refactorings in #42766

New: Run `Cypress` Tests Against Elastic Cloud

Cypress tests may now be run against a remote Elastic Cloud instance
(override baseUrl), interactively or via the command line.

Credentials are specified via kibna.dev.yml or environment variables.

New: Run `Cypress` Tests On the Command Line

Run tests on the command line (override baseUrl and pass credentials via
the CYPRESS_ELASTICSEARCH_USERNAME and CYPRESS_ELASTICSEARCH_PASSWORD
environment variables), via command line.

New: Reporting

Reports are configured to include:

An HTML report, suitable for e-mail notifications
Screenshots
Junit reports (for integration with Kibana CI)
Videos (optional)

Reports generated when Cypress tests are run on the command line are output
to the target directory, which follows specific conventions used by the
Kibana CI process:

Any directory under target that begins with kibana-, e.g. kibana-siem will be uploaded as an artifact to a bucket
Junit reports are picked up from the target/junit directory

The artifacts generated by running Cypress tests on the command line conform to the above conventions.

HTML Reports

An HTML report (e.g. for email notifications) is output to:

target/kibana-siem/cypress/results/output.html

Screenshots

Screenshots of failed tests are output to:

target/kibana-siem/cypress/screenshots

`junit` Reports

The Kibana CI process reports junit test results from the target/junit directory.

Cypress junit reports are generated in target/kibana-siem/cypress/results
and copied to the target/junit directory.

Videos (optional)

Videos are disabled by default, but can optionally be enabled by setting the
CYPRESS_video=true environment variable:

CYPRESS_video=true yarn cypress:run

Videos are (optionally) output to:

target/kibana-siem/cypress/videos

Updated `README`

The full details and instructions for running Cypress tests are in the
siem/cypress/README.md, which was updated to reflect the changes
in this in this PR, but in a nutshell, you may run tests interactively with
the following (new) environment variables:

cd x-pack/legacy/plugins/siem
CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:open

Running the command line version of the tests, which will output
the reports described above, is a similar experence:

cd x-pack/legacy/plugins/siem
CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:run

https://github.com/elastic/siem-team/issues/435
https://github.com/elastic/siem-team/issues/437

elasticmachine · 2019-08-06T23:41:30Z

Pinging @elastic/siem

x-pack/legacy/plugins/siem/package.json

andrew-goldstein · 2019-08-06T23:53:53Z

x-pack/legacy/plugins/siem/cypress/README.md

@@ -1,61 +1,237 @@
 # Cypress Tests


Note to PR reviewers: you may view a markdown-rendered version of the changes in this PR by clicking on the ... in the GitHub diff and selecting View file

elasticmachine · 2019-08-07T00:46:40Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

x-pack/legacy/plugins/siem/package.json

elasticmachine · 2019-08-09T00:09:05Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

spalger

LGTM

elasticmachine · 2019-08-13T18:35:15Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

stephmilovic · 2019-08-14T16:24:30Z

This is awesome, great work @andrew-goldstein ! Two improvements I'd like to see (whether in this PR or a new issue):

only run .spec.ts files
include screenshot or link to screenshot of failed test in the output.html report

🚀

andrew-goldstein · 2019-08-14T18:00:41Z

Thanks for the review @stephmilovic !

Two improvements I'd like to see (whether in this PR or a new issue):

only run .spec.ts files

I'm pushing the branch one more time with a change that adds --spec ./cypress/integration/**/*.spec.ts to the cypress:run target in siem/package.json, so it doesn't report on the helpers in integration/lib.

include screenshot or link to screenshot of failed test in the output.html report

That's an unfortunate limitation of the mochaawsome html reporter. I discussed this with @spalger , and we decided to use the current workaround of prefixing the reporting output under target with kibana- (i.e. target/kibana-siem), because the build process uses this naming convention to scoop up the artificats below it, including the screenshots, into a bucket.

@spalger

…ne / Reporting This PR contains enhancements to the `Cypress` end-to-end (e2e) testing support in the SIEM app, as discussed with @spalger Note: this PR depends on test refactorings in elastic#42766 ## New: Run `Cypress` Tests Against Elastic Cloud Cypress tests may now be run against a remote Elastic Cloud instance (override `baseUrl`), interactively or via the command line. Credentials are specified via `kibna.dev.yml` or environment variables. ## New: Run `Cypress` Tests On the Command Line Run tests on the command line (override `baseUrl` and pass credentials via the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD` environment variables), via command line. ## New: Reporting Reports are configured to include: - An HTML report, suitable for e-mail notifications - Screenshots - Junit reports (for integration with Kibana CI) - Videos (optional) Reports generated when `Cypress` tests are run on the command line are output to the `target` directory, which follows specific conventions used by the Kibana CI process: * Any directory under `target` that begins with `kibana-`, e.g. `kibana-siem` will be uploaded as an artifact to a bucket * Junit reports are picked up from the `target/junit` directory The artifacts generated by running `Cypress` tests on the command line conform to the above conventions. ### HTML Reports An HTML report (e.g. for email notifications) is output to: ``` target/kibana-siem/cypress/results/output.html ``` ### Screenshots Screenshots of failed tests are output to: ``` target/kibana-siem/cypress/screenshots ``` ### `junit` Reports The Kibana CI process reports `junit` test results from the `target/junit` directory. Cypress `junit` reports are generated in `target/kibana-siem/cypress/results` and copied to the `target/junit` directory. ### Videos (optional) Videos are disabled by default, but can optionally be enabled by setting the `CYPRESS_video=true` environment variable: ``` CYPRESS_video=true yarn cypress:run ``` Videos are (optionally) output to: ``` target/kibana-siem/cypress/videos ``` # Updated `README` The full details and instructions for running Cypress tests are in the `siem/cypress/README.md`, which was updated to reflect the changes in this in this PR, but in a nutshell, you may run tests interactively with the following (new) environment variables: ```sh cd x-pack/legacy/plugins/siem CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:open ``` Running the command line version of the tests, which will output the reports described above, is a similar experence: ```sh cd x-pack/legacy/plugins/siem CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:run ``` elastic/siem-team#435 elastic/siem-team#437

elasticmachine · 2019-08-14T19:28:21Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

@spalger

… / Reporting (elastic#42804) This PR contains enhancements to the `Cypress` end-to-end (e2e) testing support in the SIEM app, as discussed with @spalger Note: this PR depends on test refactorings in elastic#42766 Cypress tests may now be run against a remote Elastic Cloud instance (override `baseUrl`), interactively or via the command line. Credentials are specified via `kibna.dev.yml` or environment variables. Run tests on the command line (override `baseUrl` and pass credentials via the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD` environment variables), via command line. Reports are configured to include: - An HTML report, suitable for e-mail notifications - Screenshots - Junit reports (for integration with Kibana CI) - Videos (optional) Reports generated when `Cypress` tests are run on the command line are output to the `target` directory, which follows specific conventions used by the Kibana CI process: * Any directory under `target` that begins with `kibana-`, e.g. `kibana-siem` will be uploaded as an artifact to a bucket * Junit reports are picked up from the `target/junit` directory The artifacts generated by running `Cypress` tests on the command line conform to the above conventions. An HTML report (e.g. for email notifications) is output to: ``` target/kibana-siem/cypress/results/output.html ``` Screenshots of failed tests are output to: ``` target/kibana-siem/cypress/screenshots ``` The Kibana CI process reports `junit` test results from the `target/junit` directory. Cypress `junit` reports are generated in `target/kibana-siem/cypress/results` and copied to the `target/junit` directory. Videos are disabled by default, but can optionally be enabled by setting the `CYPRESS_video=true` environment variable: ``` CYPRESS_video=true yarn cypress:run ``` Videos are (optionally) output to: ``` target/kibana-siem/cypress/videos ``` The full details and instructions for running Cypress tests are in the `siem/cypress/README.md`, which was updated to reflect the changes in this in this PR, but in a nutshell, you may run tests interactively with the following (new) environment variables: ```sh cd x-pack/legacy/plugins/siem CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:open ``` Running the command line version of the tests, which will output the reports described above, is a similar experence: ```sh cd x-pack/legacy/plugins/siem CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:run ``` elastic/siem-team#435 elastic/siem-team#437 - regen yarn.lock

@spalger

… / Reporting (#42804) (#43315) This PR contains enhancements to the `Cypress` end-to-end (e2e) testing support in the SIEM app, as discussed with @spalger Note: this PR depends on test refactorings in #42766 Cypress tests may now be run against a remote Elastic Cloud instance (override `baseUrl`), interactively or via the command line. Credentials are specified via `kibna.dev.yml` or environment variables. Run tests on the command line (override `baseUrl` and pass credentials via the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD` environment variables), via command line. Reports are configured to include: - An HTML report, suitable for e-mail notifications - Screenshots - Junit reports (for integration with Kibana CI) - Videos (optional) Reports generated when `Cypress` tests are run on the command line are output to the `target` directory, which follows specific conventions used by the Kibana CI process: * Any directory under `target` that begins with `kibana-`, e.g. `kibana-siem` will be uploaded as an artifact to a bucket * Junit reports are picked up from the `target/junit` directory The artifacts generated by running `Cypress` tests on the command line conform to the above conventions. An HTML report (e.g. for email notifications) is output to: ``` target/kibana-siem/cypress/results/output.html ``` Screenshots of failed tests are output to: ``` target/kibana-siem/cypress/screenshots ``` The Kibana CI process reports `junit` test results from the `target/junit` directory. Cypress `junit` reports are generated in `target/kibana-siem/cypress/results` and copied to the `target/junit` directory. Videos are disabled by default, but can optionally be enabled by setting the `CYPRESS_video=true` environment variable: ``` CYPRESS_video=true yarn cypress:run ``` Videos are (optionally) output to: ``` target/kibana-siem/cypress/videos ``` The full details and instructions for running Cypress tests are in the `siem/cypress/README.md`, which was updated to reflect the changes in this in this PR, but in a nutshell, you may run tests interactively with the following (new) environment variables: ```sh cd x-pack/legacy/plugins/siem CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:open ``` Running the command line version of the tests, which will output the reports described above, is a similar experence: ```sh cd x-pack/legacy/plugins/siem CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:run ``` elastic/siem-team#435 elastic/siem-team#437 - regen yarn.lock

…p-metrics-selectall * 'master' of github.com:elastic/kibana: (50 commits) [Uptime] update monitor list configs for mobile view (elastic#43218) [APM] Local UI filters (elastic#41588) [Code] Upgrade ctags langserver (elastic#43252) [Code] show multiple definition results in panel (elastic#43249) Adds Metric Type to full screen launch tracking (elastic#42692) [Canvas] Convert Autocomplete to Typescript (elastic#42502) [telemetry] add spacesEnabled config back to xpack_main (elastic#43312) [ML] Adds DF Transform Analytics list to Kibana management (elastic#43151) Add TLS client authentication support. (elastic#43090) [csp] Telemetry for csp configuration (elastic#43223) [SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting (elastic#42804) docs: add tip on agent config in a dt (elastic#43301) [ML] Adding bucket span estimator to new wizards (elastic#43288) disable flaky tests (elastic#43017) Fix percy target branch for PRs (elastic#43160) [ML] Adding post create job options (elastic#43205) Restore discover histogram selection triggering fetch (elastic#43097) Per panel time range (elastic#43153) [Infra UI] Add APM to Metadata Endpoint (elastic#42197) Sentence case copy changes (elastic#43215) ...

@spalger

… / Reporting (elastic#42804) # [SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting (elastic#42804) This PR contains enhancements to the `Cypress` end-to-end (e2e) testing support in the SIEM app, as discussed with @spalger Note: this PR depends on test refactorings in elastic#42766 ## New: Run `Cypress` Tests Against Elastic Cloud Cypress tests may now be run against a remote Elastic Cloud instance (override `baseUrl`), interactively or via the command line. Credentials are specified via `kibna.dev.yml` or environment variables. ## New: Run `Cypress` Tests On the Command Line Run tests on the command line (override `baseUrl` and pass credentials via the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD` environment variables), via command line. ## New: Reporting Reports are configured to include: - An HTML report, suitable for e-mail notifications - Screenshots - Junit reports (for integration with Kibana CI) - Videos (optional) Reports generated when `Cypress` tests are run on the command line are output to the `target` directory, which follows specific conventions used by the Kibana CI process: * Any directory under `target` that begins with `kibana-`, e.g. `kibana-siem` will be uploaded as an artifact to a bucket * Junit reports are picked up from the `target/junit` directory The artifacts generated by running `Cypress` tests on the command line conform to the above conventions. ### HTML Reports An HTML report (e.g. for email notifications) is output to: ``` target/kibana-siem/cypress/results/output.html ``` ### Screenshots Screenshots of failed tests are output to: ``` target/kibana-siem/cypress/screenshots ``` ### `junit` Reports The Kibana CI process reports `junit` test results from the `target/junit` directory. Cypress `junit` reports are generated in `target/kibana-siem/cypress/results` and copied to the `target/junit` directory. ### Videos (optional) Videos are disabled by default, but can optionally be enabled by setting the `CYPRESS_video=true` environment variable: ``` CYPRESS_video=true yarn cypress:run ``` Videos are (optionally) output to: ``` target/kibana-siem/cypress/videos ``` # Updated `README` The full details and instructions for running Cypress tests are in the `siem/cypress/README.md`, which was updated to reflect the changes in this in this PR, but in a nutshell, you may run tests interactively with the following (new) environment variables: ```sh cd x-pack/legacy/plugins/siem CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:open ``` Running the command line version of the tests, which will output the reports described above, is a similar experence: ```sh cd x-pack/legacy/plugins/siem CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:run ``` https://github.com/elastic/siem-team/issues/435 https://github.com/elastic/siem-team/issues/437

tylersmalley · 2019-08-15T16:28:43Z

js-yaml was already a dependency, so now we are receiving the error during install ✖ x-pack: warning package.json: "dependencies" has dependency "js-yaml" with range "3.13.1" that collides with a dependency in "devDependencies" of the same name with version "^3.13.1".

Would you mind creating a PR to remove the duplicate devDependency?

## Summary Removes an extra entry for `js-yaml` in the `devDependencies` section of `x-pack/package.json`, which was introduced in #42804 The change in this PR did _not_ generate a new `yarn.lock` ### Checklist - [ ] ~~This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ - [ ] ~~Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ - [ ] ~~[Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [ ] ~~[Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~~ - [ ] ~~This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers - [ ] ~~This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ - [ ] ~~This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~

andrew-goldstein added loe:medium Medium Level of Effort Team:SIEM v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.4.0 labels Aug 6, 2019

andrew-goldstein requested review from spalger and stephmilovic August 6, 2019 23:41

andrew-goldstein commented Aug 6, 2019

View reviewed changes

x-pack/legacy/plugins/siem/package.json Show resolved Hide resolved

andrew-goldstein commented Aug 6, 2019

View reviewed changes

spalger reviewed Aug 7, 2019

View reviewed changes

x-pack/legacy/plugins/siem/package.json Show resolved Hide resolved

spalger reviewed Aug 7, 2019

View reviewed changes

x-pack/legacy/plugins/siem/package.json Show resolved Hide resolved

andrew-goldstein force-pushed the cypress-command-line-and-cloud branch from cd0ce0f to 123d5c4 Compare August 8, 2019 23:07

spalger approved these changes Aug 9, 2019

View reviewed changes

andrew-goldstein force-pushed the cypress-command-line-and-cloud branch from 123d5c4 to 031c6aa Compare August 13, 2019 17:30

stephmilovic approved these changes Aug 14, 2019

View reviewed changes

andrew-goldstein force-pushed the cypress-command-line-and-cloud branch from 031c6aa to 79c5b97 Compare August 14, 2019 18:18

andrew-goldstein merged commit 017c860 into elastic:master Aug 14, 2019

andrew-goldstein deleted the cypress-command-line-and-cloud branch August 14, 2019 19:38

andrew-goldstein mentioned this pull request Aug 14, 2019

[7.x] [SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting (#42804) #43315

Merged

andrew-goldstein mentioned this pull request Aug 15, 2019

[SIEM] Removes an extra js-yaml entry from devDependencies #43377

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting #42804

[SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting #42804

andrew-goldstein commented Aug 6, 2019 •

edited

Loading

elasticmachine commented Aug 6, 2019

andrew-goldstein Aug 6, 2019 •

edited

Loading

elasticmachine commented Aug 7, 2019

elasticmachine commented Aug 9, 2019

spalger left a comment

elasticmachine commented Aug 13, 2019

stephmilovic commented Aug 14, 2019

andrew-goldstein commented Aug 14, 2019

elasticmachine commented Aug 14, 2019

tylersmalley commented Aug 15, 2019

[SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting #42804

[SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting #42804

Conversation

andrew-goldstein commented Aug 6, 2019 • edited Loading

[SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting

New: Run Cypress Tests Against Elastic Cloud

New: Run Cypress Tests On the Command Line

New: Reporting

HTML Reports

Screenshots

junit Reports

Videos (optional)

Updated README

elasticmachine commented Aug 6, 2019

andrew-goldstein Aug 6, 2019 • edited Loading

Choose a reason for hiding this comment

elasticmachine commented Aug 7, 2019

💚 Build Succeeded

elasticmachine commented Aug 9, 2019

💚 Build Succeeded

spalger left a comment

Choose a reason for hiding this comment

elasticmachine commented Aug 13, 2019

💚 Build Succeeded

stephmilovic commented Aug 14, 2019

andrew-goldstein commented Aug 14, 2019

elasticmachine commented Aug 14, 2019

💚 Build Succeeded

tylersmalley commented Aug 15, 2019

andrew-goldstein commented Aug 6, 2019 •

edited

Loading

New: Run `Cypress` Tests Against Elastic Cloud

New: Run `Cypress` Tests On the Command Line

`junit` Reports

Updated `README`

andrew-goldstein Aug 6, 2019 •

edited

Loading