cucumber-js --help should list available formatters

[aslakhellesoy]

Here is the output from 7.3.0:

Usage: cucumber-js [options] [<GLOB|DIR|FILE[:LINE]>...]

Options:
  -v, --version                                      output the version number
  -b, --backtrace                                    show full backtrace for errors
  -d, --dry-run                                      invoke formatters without executing steps (default: false)
  --exit                                             force shutdown of the event loop when the test run has finished: cucumber will call process.exit (default: false)
  --fail-fast                                        abort the run on first failure (default: false)
  -f, --format <TYPE[:PATH]>                         specify the output format, optionally supply PATH to redirect formatter output (repeatable) (default: [])
  --format-options <JSON>                            provide options for formatters (repeatable) (default: {})
  --i18n-keywords <ISO 639-1>                        list language keywords (default: "")
  --i18n-languages                                   list languages (default: false)
  --language <ISO 639-1>                             provide the default language for feature files (default: "en")
  --name <REGEXP>                                    only execute the scenarios with name matching the expression (repeatable) (default: [])
  --no-strict                                        succeed even if there are pending steps
  --order <TYPE[:SEED]>                              run scenarios in the specified order. Type should be `defined` or `random` (default: "defined")
  -p, --profile <NAME>                               specify the profile to use (repeatable) (default: [])
  --parallel <NUMBER_OF_WORKERS>                     run in parallel with the given number of workers (default: 0)
  --predictable-ids                                  Use predictable ids in messages (option ignored if using parallel) (default: false)
  --publish                                          Publish a report to https://reports.cucumber.io (default: false)
  --publish-quiet                                    Don't print information banner about publishing reports (default: false)
  -r, --require <GLOB|DIR|FILE>                      require files before executing features (repeatable) (default: [])
  --require-module <NODE_MODULE>                     require node modules before requiring files (repeatable) (default: [])
  --retry <NUMBER_OF_RETRIES>                        specify the number of times to retry failing test cases (default: 0) (default: 0)
  --retryTagFilter, --retry-tag-filter <EXPRESSION>  only retries the features or scenarios with tags matching the expression (repeatable).
          This option requires '--retry' to be specified. (default: "")
  -t, --tags <EXPRESSION>                            only execute the features or scenarios with tags matching the expression (repeatable) (default: "")
  --world-parameters <JSON>                          provide parameters that will be passed to the world constructor (repeatable) (default: {})
  -h, --help                                         display help for command
  For more details please visit https://github.com/cucumber/cucumber-js/blob/master/docs/cli.md

For comparison, cucumber-ruby lists the available formatters:

Usage: cucumber [options] [ [FILE|DIR|URL][:LINE[:LINE]*] ]+

Examples:
cucumber examples/i18n/en/features
cucumber @rerun.txt (See --format rerun)
cucumber examples/i18n/it/features/somma.feature:6:98:113
cucumber -s -i http://rubyurl.com/eeCl

        --publish                    Publish a report to https://reports.cucumber.io
        --publish-quiet              Don't print information banner about publishing reports
    -r, --require LIBRARY|DIR        Require files before executing the features. If this
                                     option is not specified, all *.rb files that are
                                     siblings of or below the features will be loaded auto-
                                     matically. Automatic loading is disabled when this
                                     option is specified; all loading becomes explicit.
                                     Files in directories named "support" are still always
                                     loaded first when their parent directories are
                                     required or if the "support" directories themselves are
                                     explicitly required.
                                     This option can be specified multiple times.
        --retry ATTEMPTS             Specify the number of times to retry failing tests (default: 0)
        --i18n-languages             List all available languages
        --i18n-keywords LANG         List keywords for in a particular language
                                     Run with "--i18n help" to see all languages
        --fail-fast                  Exit immediately following the first failing scenario
    -f, --format FORMAT              How to format features (Default: pretty). Available formats:
                                       html     : Outputs HTML report
                                       json     : Prints the feature as JSON.
                                                     The JSON format is in maintenance mode.
                                                     Please consider using the message formatter
                                                     with the standalone json-formatter
                                                     (https://github.com/cucumber/cucumber/tree/master/json-formatter).
                                       junit    : Generates a report similar to Ant+JUnit. Use
                                                     junit,fileattribute=true to include a file attribute.
                                       message  : Outputs protobuf messages
                                       pretty   : Prints the feature as is - in colours.
                                       progress : Prints one character per scenario.
                                       rerun    : Prints failing files with line numbers.
                                       stepdefs : Prints All step definitions with their locations. Same as
                                                     the usage formatter, except that steps are not printed.
                                       summary  : Summary output of feature and scenarios
                                       usage    : Prints where step definitions are used.
                                                     The slowest step definitions (with duration) are
                                                     listed first. If --dry-run is used the duration
                                                     is not shown, and step definitions are sorted by
                                                     filename instead.
                                     Use --format rerun --out rerun.txt to write out failing
                                     features. You can rerun them with cucumber @rerun.txt.
                                     FORMAT can also be the fully qualified class name of
                                     your own custom formatter. If the class isn't loaded,
                                     Cucumber will attempt to require a file with a relative
                                     file name that is the underscore name of the class name.
                                     Example: --format Foo::BarZap -> Cucumber will look for
                                     foo/bar_zap.rb. You can place the file with this relative
                                     path underneath your features/support directory or anywhere
                                     on Ruby's LOAD_PATH, for example in a Ruby gem.
        --init                       Initializes folder structure and generates conventional files for
                                     a Cucumber project.
    -o, --out [FILE|DIR|URL]         Write output to a file/directory/URL instead of STDOUT. This option
                                     applies to the previously specified --format, or the
                                     default format if no format is specified. Check the specific
                                     formatter's docs to see whether to pass a file, dir or URL.

                                     When using a URL, the output of the formatter will be sent as the HTTP request body.
                                     HTTP headers and request method can be set with cURL like options.
                                     Example: --out "http://example.com -X POST -H Content-Type:text/json"
    -t, --tags TAG_EXPRESSION        Only execute the features or scenarios with tags matching TAG_EXPRESSION.
                                     Scenarios inherit tags declared on the Feature level. The simplest
                                     TAG_EXPRESSION is simply a tag. Example: --tags @dev. To represent
                                     boolean NOT preceed the tag with 'not '. Example: --tags 'not @dev'.
                                     A tag expression can have several tags separated by an or which represents
                                     logical OR. Example: --tags '@dev or @wip'. The --tags option can be specified
                                     A tag expression can have several tags separated by an and which represents
                                     logical AND. Example: --tags '@dev and @wip'. The --tags option can be specified
                                     several times, and this also represents logical AND.
                                     Example: --tags '@foo or not @bar' --tags @zap. This represents the boolean
                                     expression (@foo || !@bar) && @zap.

                                     Beware that if you want to use several negative tags to exclude several tags
                                     you have to use logical AND: --tags 'not @fixme and not @buggy'.

                                     Tags can be given a threshold to limit the number of occurrences.
                                     Example: --tags @qa:3 will fail if there are more than 3 occurrences of the @qa tag.
                                     This can be practical if you are practicing Kanban or CONWIP.
    -n, --name NAME                  Only execute the feature elements which match part of the given name.
                                     If this option is given more than once, it will match against all the
                                     given names.
    -e, --exclude PATTERN            Don't run feature files or require ruby files matching PATTERN
    -p, --profile PROFILE            Pull commandline arguments from cucumber.yml which can be defined as
                                     strings or arrays.  When a 'default' profile is defined and no profile
                                     is specified it is always used. (Unless disabled, see -P below.)
                                     When feature files are defined in a profile and on the command line
                                     then only the ones from the command line are used.
    -P, --no-profile                 Disables all profile loading to avoid using the 'default' profile.
    -c, --[no-]color                 Whether or not to use ANSI color in the output. Cucumber decides
                                     based on your platform and the output destination if not specified.
    -d, --dry-run                    Invokes formatters without executing the steps.
    -m, --no-multiline               Don't print multiline strings and tables under steps.
    -s, --no-source                  Don't print the file and line of the step definition with the steps.
    -i, --no-snippets                Don't print snippets for pending steps.
    -I, --snippet-type TYPE          Use different snippet type (Default: cucumber_expression). Available types:
                                     classic: Snippets without parentheses. Note that these cause a warning from modern versions of Ruby. e.g. Given /^I have (\d+) cukes$/
                                     cucumber_expression: Cucumber Expressions         e.g. Given("I have {int} cukes")
                                     percent: Snippets with percent regexp e.g. Given %r{^I have (\d+) cukes$}
                                     regexp : Snippets with parentheses    e.g. Given(/^I have (\d+) cukes$/)
    -q, --quiet                      Alias for --no-snippets --no-source --no-duration --publish-quiet.
        --no-duration                Don't print the duration at the end of the summary
    -b, --backtrace                  Show full backtrace for all errors.
    -S, --[no-]strict                Fail if there are any strict affected results
                                     (that is undefined, pending or flaky results).
        --[no-]strict-undefined      Fail if there are any undefined results.
        --[no-]strict-pending        Fail if there are any pending results.
        --[no-]strict-flaky          Fail if there are any flaky results.
    -w, --wip                        Fail if there are any passing scenarios.
    -v, --verbose                    Show the files and features loaded.
    -g, --guess                      Guess best match for Ambiguous steps.
    -l, --lines LINES                Run given line numbers. Equivalent to FILE:LINE syntax
    -x, --expand                     Expand Scenario Outline Tables in output.
        --order TYPE[:SEED]          Run examples in the specified order. Available types:
                                       [defined]     Run scenarios in the order they were defined (default).
                                       [random]      Shuffle scenarios before running.
                                     Specify SEED to reproduce the shuffling from a previous run.
                                       e.g. --order random:5738
        --version                    Show version.
    -h, --help                       You're looking at it.

[davidjgoss]

The code for defining and describing the CLI options via the commander library is here: https://github.com/cucumber/cucumber-js/blob/main/src/cli/argv_parser.ts

[TomerPacific]

Hey @aslakhellesoy , I'd like to help out with this issue.

[aslakhellesoy]

Hi @TomerPacific excellent! What questions do you have about getting started?

[TomerPacific]

@aslakhellesoy, sorry for the late reply.
I'll fork the repository and try to get everything set up. Once I have that done, I'll will start to look at the link @davidjgoss posted.

[TomerPacific]

@aslakhellesoy , I've got the project up and running and I looked at all the relevant places mentioned in this issue.
I just want to make sure that I understand what needs to be done.

Regarding the formatters, you are referencing this line from argv_parser.ts:

.option(
        '-f, --format <TYPE[:PATH]>',
        'specify the output format, optionally supply PATH to redirect formatter output (repeatable)',
        ArgvParser.collect,
        []
      )

And it should return all available formatters like in here:

     : Outputs HTML report
                                       json     : Prints the feature as JSON.
                                                     The JSON format is in maintenance mode.
                                                     Please consider using the message formatter
                                                     with the standalone json-formatter
                                                     (https://github.com/cucumber/cucumber/tree/master/json-formatter).
                                       junit    : Generates a report similar to Ant+JUnit. Use
                                                     junit,fileattribute=true to include a file attribute.
                                       message  : Outputs protobuf messages
                                       pretty   : Prints the feature as is - in colours.
                                       progress : Prints one character per scenario.
                                       rerun    : Prints failing files with line numbers.
                                       stepdefs : Prints All step definitions with their locations. Same as
                                                     the usage formatter, except that steps are not printed.
                                       summary  : Summary output of feature and scenarios
                                       usage    : Prints where step definitions are used.
                                                     The slowest step definitions (with duration) are
                                                     listed first. If --dry-run is used the duration
                                                     is not shown, and step definitions are sorted by
                                                     filename instead.
                                     Use --format rerun --out rerun.txt to write out failing
                                     features. You can rerun them with cucumber @rerun.txt.
                                     FORMAT can also be the fully qualified class name of
                                     your own custom formatter. If the class isn't loaded,
                                     Cucumber will attempt to require a file with a relative
                                     file name that is the underscore name of the class name.
                                     Example: --format Foo::BarZap -> Cucumber will look for
                                     foo/bar_zap.rb. You can place the file with this relative
                                     path underneath your features/support directory or anywhere
                                     on Ruby's LOAD_PATH, for example in a Ruby gem.

My question is if there is a programmatic way to get all the formatters (so that it will adapt to future changes) or if this should be done in another way?

[aurelien-reeves]

My question is if there is a programmatic way to get all the formatters (so that it will adapt to future changes) or if this should be done in another way?

As far as I know, no, there is no programmatic way to get formatters

[TomerPacific]

I have done the following in argv_parser.ts:

.option( '-f, --format <TYPE[:PATH]>', 'specify the output format, optionally supply PATH to redirect formatter output (repeatable)', ArgvParser.collect, [ 'html : Outputs HTML report', 'json : Prints the feature as JSON. The JSON format is in maintenance mode. Please consider using the message formatter with the standalone json-formatter (https://github.com/cucumber/cucumber/tree/master/json-formatter).', 'junit : Generates a report similar to Ant+JUnit. Use junit,fileattribute=true to include a file attribute.', 'message : Outputs protobuf messages', 'pretty : Prints the feature as is - in colours.', 'progress : Prints one character per scenario.', 'rerun : Prints failing files with line numbers.', 'stepdefs : Prints All step definitions with their locations. Same as the usage formatter, except that steps are not printed.', 'summary : Summary output of feature and scenarios', 'usage : Prints where step definitions are used. The slowest step definitions (with duration) are listed first. If --dry-run is used the duration is not shown, and step definitions are sorted by filename instead.', "Use·--format·rerun·--out·rerun.txt·to·write·out·failing·features.·You·can·rerun·them·with·cucumber·@rerun.txt.·FORMAT·can·also·be·the·fully·qualified·class·name·of·your·own·custom·formatter.·If·the·class·isn't·loaded,·Cucumber·will·attempt·to·require·a·file·with·a·relative·file·name·that·is·the·underscore·name·of·the·class·name.·Example:·--format·Foo::BarZap·->·Cucumber·will·look·for·foo/bar_zap.rb.·You·can·place·the·file·with·this·relative·path·underneath·your·features/support·directory·or·anywhere·on·Ruby's·LOAD_PATH,·for·example·in·a·Ruby·gem.", ] )

I don't know if this is the desired addition you were looking for.
Furthermore, when I am running the tests, 7 tests are failing and I want to make sure that I understand why.
Below is an example of one failing test's output:

`7) Configuration
formatters
does not split absolute windows paths without an output:

  AssertionError: expected [ Array(12) ] to deeply equal [ Array(1) ]
  + expected - actual

     {
       "outputTo": ""
       "type": "C:\\custom\\formatter"
     }
  -  {
  -    "outputTo": " Outputs HTML report"
  -    "type": "html     "
  -  }
  -  {
  -    "outputTo": " Prints the feature as JSON. The JSON format is in maintenance mode. Please consider using the message formatter with the standalone json-formatter (https"
  -    "type": "json     "
  -  }
  -  {
  -    "outputTo": " Generates a report similar to Ant+JUnit. Use junit,fileattribute=true to include a file attribute."
  -    "type": "junit    "
  -  }
  -  {
  -    "outputTo": " Outputs protobuf messages"
  -    "type": "message  "
  -  }
  -  {
  -    "outputTo": " Prints the feature as is - in colours."
  -    "type": "pretty   "
  -  }
  -  {
  -    "outputTo": " Prints one character per scenario."
  -    "type": "progress "
  -  }
  -  {
  -    "outputTo": " Prints failing files with line numbers."
  -    "type": "rerun    "
  -  }
  -  {
  -    "outputTo": " Prints All step definitions with their locations. Same as the usage formatter, except that steps are not printed."
  -    "type": "stepdefs "
  -  }
  -  {
  -    "outputTo": " Summary output of feature and scenarios"
  -    "type": "summary  "
  -  }
  -  {
  -    "outputTo": " Prints where step definitions are used. The slowest step definitions (with duration) are listed first. If --dry-run is used the duration is not shown, and step definitions are sorted by filename instead."
  -    "type": "usage    "
  -  }
  -  {
  -    "outputTo": "·--format·Foo"
  -    "type": "Use·--format·rerun·--out·rerun.txt·to·write·out·failing·features.·You·can·rerun·them·with·cucumber·@rerun.txt.·FORMAT·can·also·be·the·fully·qualified·class·name·of·your·own·custom·formatter.·If·the·class·isn't·loaded,·Cucumber·will·attempt·to·require·a·file·with·a·relative·file·name·that·is·the·underscore·name·of·the·class·name.·Example"
  -  }
   ]

  at Context.<anonymous> (C:\Users\Tomer\Documents\Projects\cucumber-js\src\cli\configuration_builder_spec.ts:349:26)
  at processTicksAndRejections (internal/process/task_queues.js:97:5)`

Taking this test as an example, I think that because there were not formatters before, this test only expected one formatter and now there are more and that is why the test is failing. Should I change the tests accordingly or am I missing something?

[aslakhellesoy]

My question is if there is a programmatic way to get all the formatters (so that it will adapt to future changes) or if this should be done in another way?

There is a list of the available formatters in

cucumber-js/src/formatter/builder.ts

Lines 65 to 84 in 6e958f1

	case 'json':
	return JsonFormatter
	case 'message':
	return MessageFormatter
	case 'html':
	return HtmlFormatter
	case 'progress':
	return ProgressFormatter
	case 'progress-bar':
	return ProgressBarFormatter
	case 'rerun':
	return RerunFormatter
	case 'snippets':
	return SnippetsFormatter
	case 'summary':
	return SummaryFormatter
	case 'usage':
	return UsageFormatter
	case 'usage-json':
	return UsageJsonFormatter

I would suggest refactoring that case statement to a Record<string,typeof Formatter> and export that record as formatters.

Then the formatters record can be used where the case statement is now, and it can also be used in argv_parser.ts to get a list of the available formatters.

The list could be passed to commander's option choices() (see docs).

As for getting the long textual description of what each formatter does - it would be nice if each Formatter had a static documentation field this could be extracted from.

[aurelien-reeves]

@TomerPacific it would be easier if you open a pull request with your changes