Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

T350782 Test System Rewrite #542

Merged
merged 10 commits into from
Dec 20, 2023
Merged

T350782 Test System Rewrite #542

merged 10 commits into from
Dec 20, 2023

Conversation

lorenjohnson
Copy link
Contributor

@lorenjohnson lorenjohnson commented Dec 16, 2023

Replaces PR #525 with a much more clear commit history. This branch also no longer includes linting of docs/diagrams or the property test updates.


Makes possible running tests more atomically, and more. https://phabricator.wikimedia.org/T350782

  • Replaces all test runner shell scripts and with Typescript making more clear and flexible control of the our test environment
  • Update test suite setup to allow for a full set of overrides to the test setup, and running process. See *.conf.js files and review the related Typescript types which self-document much of what is happening.
  • Provides detailed WDIO test run logs in suites/SUITE_NAME/results/wdio*.log. Here we can see all browser console output, as well as detail results of each selector happening in each spec file. Setup log from steps before test runners start is wdio.log, subsequent log files are created for each spec file ran according to the correlated WDIO job runner number, e.g. wdio-0-1.log, etc.
  • Changes local dev test run CLI from make test SUITE=<suite-name> to ./test.sh <suite-name>
  • Adds --setup test runner option to bring-up a test suites services without running tests, and leave them up along with the Selenium instance for manual testing, e.g ./test.sh quickstatements --setup
  • Allows all WDIO Runner options to be added to test runs :./test.sh <suite-name> <WDIO-Runner-CLI-options>. E.g. ./test.sh repo --spec=specs/repo/property.ts --watch, etc
  • Rewrites Example test setup to be in line with other tests. The Example test suite is now ran using simply: ./test.sh example
  • Rewrites Upgrade test setup to be in line with other tests. Run the Upgrade test suite with: ./test.sh upgrade WMDE9 <optional-version-tag-of-upgrade-to-version>
  • Renames all *_SERVER env variables to *_URL in the test environment (the only place we were using these variables) and adds the http:// protocol prefix where it wasn't consistently applied before.
  • Removes linting run before test runs
  • Fix: Removes password requirement on Selenium Grid / Test Browser instance (docs updated)
  • Fix: Linting no longer tries to lint Dockerfiles within node_modules directories
  • Fix: In Github Actions removes lint as a "need" of the build and test jobs . This will have the effect of the checks still failing if linting fails, but build and test continue running and building runs in parallel with linting decreasing test run times by a couple minutes.
    ... etc

Please try out the new Test CLI, here is the help for that which can be retrieved by simply typing ./test.sh or ./test.sh --help:

> ./test.sh --help

./test.sh <suiteName>

Run a test suite

Commands:
  ./test.sh <suiteName>                        Run a test suite                                                [default]
  ./test.sh upgrade <fromVersion> [toVersion]  run upgrade test
  ./test.sh all                                Run all test suites

Positionals:
  suiteName  Name of the test suite to run or setup
   [string] [choices: "repo", "fedprops", "repo_client", "quickstatements", "pingback", "confirm_edit", "elasticsearch",
                                       "base__repo", "base__repo_client", "base__pingback", "base__fedprops", "example"]

Options:
  --setup                  Start and leave-up the test environment without running tests                       [boolean]
  --headedTests, --headed  Run tests in a headed browser                                                       [boolean]
  --spec                   Run only this spec file(s) using the selected test suite setup (WDIO)                 [array]
  --watch                  Automatically re-run tests when spec files in the suite change (WDIO)               [boolean]
  --version                Show version number                                                                 [boolean]
  --help                   Show help                                                                           [boolean]

Also help is available specific to the Upgrade test suite:

> ./test.sh upgrade --help

./test.sh upgrade <fromVersion> [toVersion]

run upgrade test

Positionals:
  fromVersion  Version to upgrade FROM in the upgrade test
  [string] [required] [choices: "WMDE9", "WMDE9_BUNDLE", "WMDE10", "WMDE10_BUNDLE", "WMDE11", "WMDE11_BUNDLE", "WMDE12",
                                        "WMDE12_BUNDLE", "WMDE13", "WMDE13_BUNDLE", "LOCAL_BUILD", "LOCAL_BUILD_BUNDLE"]
  toVersion    Version to upgrade TO in the upgrade test
  [string] [choices: "WMDE9", "WMDE9_BUNDLE", "WMDE10", "WMDE10_BUNDLE", "WMDE11", "WMDE11_BUNDLE", "WMDE12", "WMDE12_BU
                                                  NDLE", "WMDE13", "WMDE13_BUNDLE", "LOCAL_BUILD", "LOCAL_BUILD_BUNDLE"]

Options:
  --setup                  Start and leave-up the test environment without running tests                       [boolean]
  --headedTests, --headed  Run tests in a headed browser                                                       [boolean]
  --spec                   Run only this spec file(s) using the selected test suite setup (WDIO)                 [array]
  --watch                  Automatically re-run tests when spec files in the suite change (WDIO)               [boolean]
  --version                Show version number                                                                 [boolean]
  --help                   Show help                                                                           [boolean]

@lorenjohnson lorenjohnson changed the title T350782 test system rewrite T350782 Test System Re-write Dec 16, 2023
@lorenjohnson lorenjohnson changed the title T350782 Test System Re-write T350782 Test System Rewrite Dec 16, 2023
@lorenjohnson lorenjohnson requested a review from a team December 16, 2023 05:47
@lorenjohnson lorenjohnson marked this pull request as ready for review December 16, 2023 05:47
lint.sh Outdated Show resolved Hide resolved
lint.sh Show resolved Hide resolved
test/cli.js Outdated Show resolved Hide resolved
test/setup/wdio.conf.ts Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
test/setup/loadEnvFiles.ts Outdated Show resolved Hide resolved
test/setup/wdio.conf.ts Outdated Show resolved Hide resolved
flake.nix Outdated Show resolved Hide resolved
test/types/wdio.d.ts Outdated Show resolved Hide resolved
test/package.json Show resolved Hide resolved
@lorenjohnson lorenjohnson force-pushed the T350782-test-system-rewrite branch from f4100de to 0e182f7 Compare December 20, 2023 11:18
@lorenjohnson lorenjohnson force-pushed the T350782-test-system-rewrite branch from 0e182f7 to ca721e3 Compare December 20, 2023 11:20
@lorenjohnson lorenjohnson merged commit 1538078 into main Dec 20, 2023
26 checks passed
@lorenjohnson lorenjohnson deleted the T350782-test-system-rewrite branch December 20, 2023 12:07
lorenjohnson added a commit that referenced this pull request Dec 22, 2023
…r usage in scripts, updates test docs (#553)

- Correctly uses new wikibase/* namespace when checking for already-loaded Docker Images in test
- Shares test-runner setup stuff in common with the lint.sh and test.sh scripts in "test/scripts/test_runner_setup.sh"
- Adds a DEBUG option for "test.sh" which will passthrough Docker build and setup stuff in the case of mysterious issues when running the script
- Adds a "--command", "-c" option for "test.sh" which will allow commands to be ran on test-runner
- "lint.sh" now also builds test-runner like "test.sh" did
- Explicitly creates the Docker network used by test-runner and test-services when it doesn't already exist
- Test documentation update
RickiJay-WMDE pushed a commit that referenced this pull request Feb 19, 2024
…r usage in scripts, updates test docs (#553)

- Correctly uses new wikibase/* namespace when checking for already-loaded Docker Images in test
- Shares test-runner setup stuff in common with the lint.sh and test.sh scripts in "test/scripts/test_runner_setup.sh"
- Adds a DEBUG option for "test.sh" which will passthrough Docker build and setup stuff in the case of mysterious issues when running the script
- Adds a "--command", "-c" option for "test.sh" which will allow commands to be ran on test-runner
- "lint.sh" now also builds test-runner like "test.sh" did
- Explicitly creates the Docker network used by test-runner and test-services when it doesn't already exist
- Test documentation update
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants