Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Command-line improvements #121

Open
cgay opened this issue Dec 14, 2019 · 0 comments
Open

Command-line improvements #121

cgay opened this issue Dec 14, 2019 · 0 comments
Milestone

Comments

@cgay
Copy link
Member

cgay commented Dec 14, 2019

Current Testworks command line:

$ _build/bin/testworks-run --help
Usage: testworks-run [options]

Run test suites.
      --debug WHAT        Enter the debugger on failure: NO|crashes|failures
  -p, --progress TYPE     Show output as the test run progresses: none|DEFAULT|verbose
      --report TYPE       Final report to generate: failures|full|json|log|none|summary|surefire|xml
      --report-file FILE  File in which to store the report.
      --load FILE         Load the given shared library file before searching for test suites. May be repeated.
      --suite SUITE       Run (or list) only these named suites. May be repeated.
      --test TEST         Run (or list) only these named tests. May be repeated.
      --skip-suite SUITE  Skip these named suites. May be repeated.
      --skip-test TEST    Skip these named tests. May be repeated.
  -l, --list WHAT         List components: all|suites|tests|benchmarks
  -t, --tag TAG           Only run tests matching this tag. If tag is prefixed with '-', the test will only run if it does NOT have the tag. May be repeated. Ex: --tag=-slow,-benchmark means don't run benchmarks or tests tagged as slow.
  -h, --help              Display this message.

Off the top of my head, I propose these changes:

  1. Always output the result summary. If someone truly doesn't want to see this (e.g., automation) then >/dev/null is available.
  2. Get rid of --debug=no. This is equivalent to not specifying the --debug flag.
  3. --report should be only about the report format. Therefore the options are json, xml, surefire, and log. (But we should delete log, and maybe xml.)
  4. Replace --progress with --verbose boolean option to display what you currently get from --report full. --report full goes away.)
  5. Replace --suite and --test with --match, which accepts a regular expression and may be repeated. Suites and tests occupy the same namespace so there is no ambiguity.
  6. Replace --skip-test and --skip-suite with --skip, which accepts a regular expression and may be repeated. Ditto.
  7. Make --list a boolean flag (i.e., and list the tests that would be run based on --match and --skip.
  8. Add a few more short options (see below).
  9. Provide a default location for --report-file. Probably ./testworks-report.json etc. Tell user where the report was written.

The end result looks something like this:

Run tests.
  -d, --debug WHAT        Enter the debugger on failure: crashes|failures
  -v, --verbose           Show output as the test run progresses.
      --report TYPE       Final report to generate: json|log|surefire|xml
      --report-file FILE  File in which to store the report.
  -l, --load FILE         Load the given shared library file before searching for tests. May be repeated.
      --match REGEX       Run only tests and suites matching this regular expression. May be repeated.
      --skip REGEX        Skip tests and suites matching this regular expression. May be repeated.
  -l, --list              Only list the tests that would have been run.
  -t, --tag TAG           Only run tests matching this tag. If tag is prefixed with '-', the test will only run if it does NOT have the tag. May be repeated. Ex: --tag=-slow,-benchmark means don't run benchmarks or tests tagged as slow.
  -h, --help              Display this message.
@cgay cgay added this to the Prioritization Test milestone Dec 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant