# [SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Li…

…ne / Reporting

This PR contains enhancements to the `Cypress` end-to-end (e2e) testing support in the SIEM app, as discussed with @spalger

Note: this PR depends on test refactorings in #42766

## New: Run `Cypress` Tests Against Elastic Cloud

Cypress tests may now be run against a remote Elastic Cloud instance
(override `baseUrl`), interactively or via the command line.

Credentials are specified via `kibna.dev.yml` or environment variables.

## New: Run `Cypress` Tests On the Command Line

Run tests on the command line (override `baseUrl` and pass credentials via
the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
environment variables), via command line.

## New: Reporting

Reports are configured to include:

- An HTML report, suitable for e-mail notifications
- Screenshots
- Junit reports (for integration with Kibana CI)
- Videos (optional)

Reports generated when `Cypress` tests are run on the command line are output
to the `target` directory, which follows specific conventions used by the
Kibana CI process:

* Any directory under `target` that begins with `kibana-`, e.g. `kibana-siem` will be uploaded as an artifact to a bucket
* Junit reports are picked up from the `target/junit` directory

The artifacts generated by running `Cypress` tests on the command line conform to the above conventions.

### HTML Reports

An HTML report (e.g. for email notifications) is output to:

```
target/kibana-siem/cypress/results/output.html
```

### Screenshots

Screenshots of failed tests are output to:

```
target/kibana-siem/cypress/screenshots
```

### `junit` Reports

The Kibana CI process reports `junit` test results from the `target/junit` directory.

Cypress `junit` reports are generated in `target/kibana-siem/cypress/results`
and copied to the `target/junit` directory.

### Videos (optional)

Videos are disabled by default, but can optionally be enabled by setting the
`CYPRESS_video=true` environment variable:

```
CYPRESS_video=true yarn cypress:run
```

Videos are (optionally) output to:

```
target/kibana-siem/cypress/videos
```

# Updated `README`

The full details and instructions for running Cypress tests are in the
`siem/cypress/README.md`, which was updated to reflect the changes
in this in this PR, but in a nutshell, you may run tests interactively with
the following (new) environment variables:

```sh
cd x-pack/legacy/plugins/siem
CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:open
```

Running the command line version of the tests, which will output
the reports described above, is a similar experence:

```sh
cd x-pack/legacy/plugins/siem
CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:run
```

elastic/siem-team#435
elastic/siem-team#437

x-pack/legacy/plugins/siem/cypress/README.md

@@ -1,61 +1,237 @@
 # Cypress Tests
 
-The `siem/cypress` directory contains end to end tests (specific to the `SIEM` app) that execute via [Cypress](https://www.cypress.io/).
+The `siem/cypress` directory contains end to end tests, (plus a few tests
+that rely on mocked API calls), that execute via [Cypress](https://www.cypress.io/).
 
-At present, these tests are only executed in a local development environment; they are **not** integrated in the Kibana CI infrastructure, and therefore do **not** run automatically when you submit a PR.
+Cypress tests may be run against:
 
-See the `Server and Authentication Requirements` section below for additional details.
+- A local Kibana instance, interactively or via the command line. Credentials
+are specified via `kibna.dev.yml` or environment variables.
+- A remote Elastic Cloud instance (override `baseUrl`), interactively or via
+the command line. Again, credentials are specified via `kibna.dev.yml` or
+environment variables.
+- As part of CI (override `baseUrl` and pass credentials via the
+`CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
+environment variables), via command line.
 
-## Organizing Tests and (Mock) Data
+At present, Cypress tests are only executed manually. They are **not** yet
+integrated in the Kibana CI infrastructure, and therefore do **not** run
+automatically when you submit a PR.
 
-- Code and CSS selectors that may be re-used across tests should be added to `siem/cypress/integration/lib`, as described below
-- Smoke Tests are located in `siem/cypress/integration/smoke_tests`
-- Mocked responses from the server are located in `siem/cypress/fixtures`
+## Smoke Tests
 
-### `cypress/integration/lib`
+Smoke Tests are located in `siem/cypress/integration/smoke_tests`
 
-The `cypress/integration/lib` folder contains code intended to be re-used across many different tests.
+## Test Helpers
 
-- Files named `helpers.ts` (e.g. `siem/cypress/integration/lib/login/helpers.ts`) contain functions (e.g. `login`) that may be imported and invoked from multiple tests.
+_Test helpers_ are functions that may be re-used across tests.
 
-- Files named `selectors.ts` export CSS selectors for re-use. For example, `siem/cypress/integration/lib/login/selectors.ts` exports the following selector that matches the Username text area in the Kibana login page:
+- Reusable code and CSS selectors should be added to
+`siem/cypress/integration/lib`, as described below.
 
-```
+### Reusable Test Helper Functions and CSS Selectors
+
+The `cypress/integration/lib` directory contains code intended to be re-used
+across many different tests. Add reusable test helper functions and CSS
+selectors to directories under `cypress/integration/lib`.
+
+- Files named `helpers.ts` (e.g. `siem/cypress/integration/lib/login/helpers.ts`)
+contain functions (e.g. `login`) that may be imported and invoked from multiple tests.
+
+- Files named `selectors.ts` export CSS selectors for re-use. For example,
+`siem/cypress/integration/lib/login/selectors.ts` exports the following selector
+that matches the Username text area in the Kibana login page:
+
+```sh
 export const USERNAME = '[data-test-subj="loginUsername"]';
 ```
 
-## Server and Authentication Requirements
+## Mock Data
+
+We prefer not to mock API responses in most of our smoke tests, but sometimes
+it's necessary because a test must assert that a specific value is rendered,
+and it's not possible to derive that value based on the data in the
+envrionment where tests are running.
+
+Mocked responses API from the server are located in `siem/cypress/fixtures`.
+
+## Authentication
+
+When running tests, there are two ways to specify the credentials used to
+authenticate with Kibana:
+
+- Via `kibana.dev.yml` (recommended for developers)
+- Via the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
+environment variables (recommended for CI), or when testing a remote Kibana
+instance, e.g. in Elastic Cloud.
 
-The current version of the Smoke Tests require running a local Kibana server that connects to an instance of `elasticsearch`. A file named `config/kibana.dev.yml` like the example below is required to run the tests:
+Note: Tests that use the `login()` test helper function for authentication will
+automatically use the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
+environment variables when they are defined, and fall back to the values in
+`config/kibana.dev.yml` when they are unset.
+
+### Content Security Policy (CSP) Settings
+
+Your local or cloud Kibana server must have the `csp.strict: false` setting
+configured in `kibana.dev.yml`, or `kibana.yml`, as shown in the example below:
+
+```yaml
+csp.strict: false
+```
+
+The above setting is required to prevent the _Please upgrade
+your browser_ / _This Kibana installation has strict security requirements
+enabled that your current browser does not meet._ warning that's displayed for
+unsupported user agents, like the one reported by Cypress when running tests.
+
+### Example `kibana.dev.yml`
+
+If you're a developer running tests interactively or on the command line, the
+easiset way to specify the credentials used for authentication is to update
+ `kibana.dev.yml` per the following example:
 
 ```yaml
+csp.strict: false
 elasticsearch:
  username: 'elastic'
  password: '<password>'
  hosts: ['https://<server>:9200']
 ```
 
-The `username` and `password` from `config/kibana.dev.yml` will be read by the `login` test helper function when tests authenticate with Kibana.
+## Running Tests Interactively
 
-See the `Running Tests Interactively` section for details.
+To run and debug tests in interactively via the Cypress test runner:
 
-## Running Tests Interactively
+1. Disable CSP on the local or remote Kibana instance, as described in the
+_Content Security Policy (CSP) Settings_ section above.
 
-To run tests in interactively via the Cypress test runner:
+2. To specify the credentials required for authentication, configure
+`config/kibana.dev.yml`, as described in the _Server and Authentication
+Requirements_ section above, or specify them via environment variables
+as described later in this section.
 
-1. Create and configure a `config/kibana.dev.yml`, as described in the `Server and Authentication Requirements` section above.
+3. Start a local instance of the Kibana development server (only if testing against a
+local host):
 
-2. Start a local instance of the Kibana development server:
+```sh
+yarn start --no-base-path
+```
+
+4. Launch the Cypress interactive test runner via one of the following options:
+
+- To run tests interactively against the default (local) host specified by
+`baseUrl`, as configured in `plugins/siem/cypress.json`:
+
+```sh
+cd x-pack/legacy/plugins/siem
+yarn cypress:open
+```
 
+- To (optionally) run tests interactively against a different host, pass the
+`CYPRESS_baseUrl` environment variable on the command line when launching the
+test runner, as shown in the following example:
+
+```sh
+cd x-pack/legacy/plugins/siem
+CYPRESS_baseUrl=http://localhost:5601 yarn cypress:open
+```
+
+- To (optionally) override username and password via environment variables when
+running tests interactively:
+
+```sh
+cd x-pack/legacy/plugins/siem
+CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:open
 ```
+
+5. Click the `Run all specs` button in the Cypress test runner
+
+## Running Tests on the Command Line
+
+To run tests on the command line:
+
+1. Disable CSP on the local or remote Kibana instance, as described in the
+_Content Security Policy (CSP) Settings_ section above.
+
+2. To specify the credentials required for authentication, configure
+`config/kibana.dev.yml`, as described in the _Server and Authentication
+Requirements_ section above, or specify them via environment variables
+as described later in this section.
+
+3. Start a local instance of the Kibana development server (only if testing against a
+local host):
+
+```sh
 yarn start --no-base-path
 ```
 
-3. Launch the Cypress interactive test runner:
+4. Launch the Cypress command line test runner via one of the following options:
+
+- To run tests on the command line against the default (local) host specified by
+`baseUrl`, as configured in `plugins/siem/cypress.json`:
 
 ```sh
 cd x-pack/legacy/plugins/siem
-yarn cypress:open
+yarn cypress:run
 ```
 
-4. Click the `Run all specs` button in the Cypress test runner
+- To (optionally) run tests on the command line against a different host, pass
+`CYPRESS_baseUrl` as an environment variable on the command line, as shown in
+the following example:
+
+```sh
+cd x-pack/legacy/plugins/siem
+CYPRESS_baseUrl=http://localhost:5601 yarn cypress:run
+```
+
+- To (optionally) override username and password via environment variables when
+running via the command line:
+
+```sh
+cd x-pack/legacy/plugins/siem
+CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:run
+```
+
+## Reporting
+
+When Cypress tests are run on the command line via `yarn cypress:run`,
+reporting artifacts are generated under the `target` directory in the root
+of the Kibana, as detailed for each artifact type in the sections bleow.
+
+### HTML Reports
+
+An HTML report (e.g. for email notifications) is output to:
+
+```
+target/kibana-siem/cypress/results/output.html
+```
+
+### Screenshots
+
+Screenshots of failed tests are output to:
+
+```
+target/kibana-siem/cypress/screenshots
+```
+
+### `junit` Reports
+
+The Kibana CI process reports `junit` test results from the `target/junit` directory.
+
+Cypress `junit` reports are generated in `target/kibana-siem/cypress/results`
+and copied to the `target/junit` directory.
+
+### Videos (optional)
+
+Videos are disabled by default, but can optionally be enabled by setting the
+`CYPRESS_video=true` environment variable:
+
+```
+CYPRESS_video=true yarn cypress:run
+```
+
+Videos are (optionally) output to:
+
+```
+target/kibana-siem/cypress/videos
+```

x-pack/legacy/plugins/siem/cypress/cypress.json

@@ -0,0 +1,7 @@
+{
+ "baseUrl": "http://localhost:5601",
+ "screenshotsFolder": "../../../../target/kibana-siem/cypress/screenshots",
+ "trashAssetsBeforeRuns": false,
+ "video": false,
+ "videosFolder": "../../../../target/kibana-siem/cypress/videos"
+}

x-pack/legacy/plugins/siem/package.json

@@ -6,20 +6,18 @@
  "license": "Elastic-License",
  "scripts": {
  "build-graphql-types": "node scripts/generate_types_from_graphql.js",
- "cypress:open": "cypress open"
+ "cypress:open": "cypress open",
+ "cypress:run": "cypress run --reporter mocha-multi-reporters --reporter-options configFile=./reporter_config.json; mochawesome-merge --reportDir ../../../../target/kibana-siem/cypress/results > ../../../../target/kibana-siem/cypress/results/output.json; marge ../../../../target/kibana-siem/cypress/results/output.json --reportDir ../../../../target/kibana-siem/cypress/results; mkdir -p ../../../../target/junit && cp ../../../../target/kibana-siem/cypress/results/*.xml ../../../../target/junit/"
  },
  "devDependencies": {
  "@cypress/webpack-preprocessor": "^4.1.0",
  "@types/js-yaml": "^3.12.1",
  "@types/lodash": "^4.14.110",
- "@types/react-beautiful-dnd": "^10.0.1",
- "cypress": "^3.3.1",
- "js-yaml": "^3.13.1",
- "ts-loader": "^6.0.4"
+ "@types/react-beautiful-dnd": "^10.0.1"
  },
  "dependencies": {
  "lodash": "^4.17.13",
  "react-beautiful-dnd": "^10.0.1",
  "react-markdown": "^4.0.6"
  }
-}
+}

x-pack/legacy/plugins/siem/reporter_config.json

@@ -0,0 +1,10 @@
+{
+ "reporterEnabled": "mochawesome, mocha-junit-reporter",
+ "reporterOptions": {
+ "html": false,
+ "json": true,
+ "mochaFile": "../../../../target/kibana-siem/cypress/results/results-[hash].xml",
+ "overwrite": false,
+ "reportDir": "../../../../target/kibana-siem/cypress/results"
+ }
+}

x-pack/package.json

@@ -118,6 +118,7 @@
  "cheerio": "0.22.0",
  "commander": "2.20.0",
  "copy-webpack-plugin": "^5.0.0",
+ "cypress": "^3.4.1",
  "del": "^4.0.0",
  "dotenv": "2.0.0",
  "enzyme": "^3.10.0",
@@ -139,8 +140,14 @@
  "jest-cli": "^24.8.0",
  "jest-styled-components": "^6.2.2",
  "jsdom": "^12.0.0",
+ "js-yaml": "^3.13.1",
  "madge": "3.4.4",
  "mocha": "3.5.3",
+ "mocha-junit-reporter": "^1.23.1",
+ "mocha-multi-reporters": "^1.1.7",
+ "mochawesome": "^4.0.1",
+ "mochawesome-merge": "^2.0.1",
+ "mochawesome-report-generator": "^4.0.1",
  "mustache": "^2.3.0",
  "mutation-observer": "^1.0.3",
  "node-fetch": "^2.1.2",
@@ -164,6 +171,7 @@
  "supertest-as-promised": "^4.0.2",
  "tmp": "0.1.0",
  "tree-kill": "^1.1.0",
+ "ts-loader": "^6.0.4",
  "typescript": "3.5.3",
  "vinyl-fs": "^3.0.2",
  "xml-crypto": "^0.10.1",
@@ -359,5 +367,13 @@
  },
  "engines": {
  "yarn": "^1.10.1"
+ },
+ "workspaces": {
+ "nohoist": [
+ "**/mochawesome",
+ "**/mochawesome/**",
+ "**/mocha-multi-reporters",
+ "**/mocha-multi-reporters/**"
+ ]
  }
-}
+}