Programmatic API for running cucumber-js

[davidjgoss]

Problem

Currently we don't have a good way to programmatically run cucumber-js. The need is from several angles:

• Testing cucumber-js - for the CCK and our acceptance tests in the project
• Testing custom formatters and snippets
• Projects that use cucumber-js as part of a framework (e.g. Serenity, Stryker)
• Projects that want to run cucumber-js programmatically for niche reasons

As-is

What tends to happen at the moment is a new instance of Cli is created with strung-together argv input. It's obviously very unweildy and also isn't on the public API.

Sometimes (possibly due to perceived fragility of the above) frameworks will just rely on the cucumber-js CLI but struggle to find ways to integrate and have their own options.

The Runtime class is currently part of the public API but it's not useful in these contexts, depending on the pickles and support code to be provided by the caller.

Proposal

Two components in the project:

runCucumber

New async function that executes a test run in-process. Responsibilities:

• Accept an options object with a nice interface
• Do the "work" of resolving pickles, loading support code, running test cases, orchestrating formatters
• Return a promise that resolves to a result

This would be part of the public API and we'd encourage framework maintainers to use it when "wrapping" cucumber-js. We'd also use it for our own testing.

As much as possible, it would avoid direct interaction with process, instead accepting normalised options and stream interfaces for output, and leaving it to the caller to decide how to exit based on the result or an unhandled error.

Also Runtime should come off the public API as it's really an internal thing.

CLI

Effectively a "client" of runCucumber. Responsibilities:

• Aggregate options from various sources (argv, env vars, config files) (see comment)
• Call runCucumber with the resolved options
• Exit as appropriate based on the results

This would continue to not be on the public API. Also it would only use functions/interfaces that are on the public API, such that we could easily break it out into its own package at some point, as is a common pattern now with projects like Jest.

This decoupling also paves the way for some interesting new CLI features without having them bleed into the internals, e.g.:

• --gui for the cucumber-electron stuff
• --interactive for quick targeted reruns when TDD'ing

etc

We would also expose functions (consumable by the CLI and by others) for:

• Getting the options
• Handling i18nKeywords and i18nLanguages

Timescale

We'd target this at the upcoming 8.0.0 release.

[davidjgoss]

Paging for initial feedback: @aslakhellesoy @charlierudolph @aurelien-reeves @mattwynne @nicojs @jan-molak

[aslakhellesoy]

I love this proposal! We already an API like this in fake-cucumber's runCucumber

[jan-molak]

@davidjgoss - sounds great!

For your reference, here's how Serenity/JS invokes Cucumber at the moment - CucumberCLIAdapter
And here's the logic around converting configuration parameters to argv - CucumberOptions.

Being able to provide an options object instead of argv would be much nicer 👍🏻

[aurelien-reeves]

Love it!

[aurelien-reeves]

While specifying that new public API, we may also consider what recently happened with issue #1489 and think about providing public APIs to have more and better interaction with the filters and the resulting features under test

[aslakhellesoy]

And to add to that - let's make something that would make it easier for Cucumber-Electron too:

• https://github.com/cucumber/cucumber-electron/blob/1c2faaa77065754d1ca17dce27fff1c421d226d0/src/renderer/main.js#L49
• https://github.com/cucumber/cucumber-electron/blob/0617ab15545ac5ceb4587599e589fefd0b670142/src/renderer/index.js#L41-L50

[nicojs]

Having a public API is better than nothing, so go ahead 👍!

Preferably I would also have an API to load profiles using the same rules as cucumber-js does, so I can mimic the exact behavior of a normal cucumber-js call.

loadProfiles(directory = process.cwd()): Record<string, Profile>

StrykerJS will also rely heavily on the custom_formatters API and the events published by the eventBroadcaster. Could we add those to the public API as well? See: https://github.com/stryker-mutator/stryker-js/blob/03b1f20ed933d3a50b52022cfe363c606c2b16c5/packages/cucumber-runner/src/stryker-formatter.ts#L45-L69

[davidjgoss]

Preferably I would also have an API to load profiles using the same rules as cucumber-js does, so I can mimic the exact behavior of a normal cucumber-js call.

This is a good point. Profiles (in their current form at least) are fundamentally coupled to the CLI so it feels right to keep them on that side of the boundary, but we could still expose a function to load them and generate a partial options object.

[davidjgoss]

While specifying that new public API, we may also consider what recently happened with issue #1489 and think about providing public APIs to have more and better interaction with the filters and the resulting features under test.

I think we could include an option for providing a custom pickle filter when calling the API (in addition to the names, tags etc that drive the built-in filtering).

[aslakhellesoy]

The current syntax for profiles if very command-line-y.

I would ❤️❤️❤️ to be able to specify profiles in a more generic format such as JSON, JavaScript, YAML or environment variables. In JSON it could look like this:

.cucumber.json

{
  "default": {
    "requireModule": ["ts-node/register"],
    "require": ["support/**/*./ts"],
    "worldParameters": {
      "appUrl": "http://localhost:3000/",
    },
    "format": ["progress-bar", "html:./cucumber-report.html"]
  },
  "ci": {
    "requireModule": ["ts-node/register"],
    "require": ["support/**/*./ts"],
    "worldParameters": {
      "appUrl": "http://localhost:3000/",
    },
    "format": ["html:./cucumber-report.html"],
    "publish": true
  }
}

Or, using JavaScript

.cucumber.js

const common = {
  "requireModule": ["ts-node/register"],
  "require": ["support/**/*./ts"],
  "worldParameters": {
    "appUrl": "http://localhost:3000/",
  }
}

module.exports = {
  default: {
    ...common,
    "format": ["progress-bar", "html:./cucumber-report.html"]
  },
  ci: {
    ...common,
    "format": ["html:./cucumber-report.html"],
    "publish": true
  }
}

Or even with environment variables (for example loaded with a tool like dotenv):

.cucumber.env

CUCUMBER_PROFILE_DEFAULT_REQUIREMODULE=ts-node/register
CUCUMBER_PROFILE_DEFAULT_REQUIRE=ts-node/register
CUCUMBER_PROFILE_DEFAULT_WORLDPARAMETERS_APPURL=http://localhost:3000/
CUCUMBER_PROFILE_DEFAULT_FORMAT=progress-bar,html:./cucumber-report.html
CUCUMBER_PROFILE_CI_REQUIREMODULE=ts-node/register
CUCUMBER_PROFILE_CI_REQUIRE=ts-node/register
CUCUMBER_PROFILE_CI_WORLDPARAMETERS_APPURL=http://localhost:3000/
CUCUMBER_PROFILE_CI_FORMAT=progress-bar,html:./cucumber-report.html
CUCUMBER_PROFILE_CI_PUBLISH=true

In fact, the config library does exactly this. We never ended up integrating it in Cucumber-JVM because other stuff got in the way, but maybe we could give it a go with a JavaScript implementation?

[davidjgoss]

@aslakhellesoy agree that would be awesome! I'll try and get a POC going for this proposal so we have something a bit more concrete to talk around, and would love to make profiles right as part of it (4.5 years and counting on #751 😄)

[aurelien-reeves]

Refs. #1004

[nicojs]

This is a good point. Profiles (in their current form at least) are fundamentally coupled to the CLI so it feels right to keep them on that side of the boundary, but we could still expose a function to load them and generate a partial options object.

Yes, that would be awesome and much appreciated from the point of view of plugin creators.

I would ❤️❤️❤️ to be able to specify profiles in a more generic format such as JSON, JavaScript, YAML or environment variables. In JSON it could look like this:

That sounds great! And is also exactly the reason I would appreciate an API to load them the same way as cucumberJS does. Loading a single cucumber.js file is trivial. Replicating a configuration file loading algorithm, including precedence, file format, etc AND maintaining it is something else entirely 😅.

[nicojs]

Q: Would I be able to run runCucumber twice in succession without clearing the require cache? This is important for the mutation testing use case.

We want to load the environment and run the tests multiple times in quick succession while changing a global variable in order to switch the active mutant.

Right now, we're using the cli private API and we need to clear the step definition files from the require.cache between each test run. This is not ideal for CommonJS and won't work at all for esm.

Pseudo code of our use case:

const profiles = await loadProfiles();
const options = {
  ...profiles.default,
  formatter: require.resolve('./our-awesomely-crafted-formatter'),
  some: 'other options we want to override',
}
const cucumber = new Cucumber(options);

// Allow cucumber to load the step definitions once. 
// This is `async`, so support for esm can be added without a breaking change
await cucumber.initialize();

// Initial test run ("dry run"), without mutants active
await cucumber.run();

collectMutantCoveragePerTestFromFormatter();

// Start mutation testing:

global.activeMutant = 1;
await cucumber.run({ features: ['features/a.feature:24']);
collectResultsFromFormatterToDetermineKilledOrSurvivedMutant()

global.activeMutant = 2;
await cucumber.run({ features: ['features/b.feature:24:25:26', 'features/c.feature:12']);
collectResultsFromFormatterToDetermineKilledOrSurvivedMutant()

// etc

[davidjgoss]

@nicojs definitely agree we need this, it's come up a few times before with e.g. people wanting to run cucumber in a lambda, and I'd also like to add an interactive mode which would need it too.

What I had sketched out so far was in more of a functional style but same fundamental concept I think:

const runnerOptions = {
    support: {
        require: ['features/support/**/*.js']
    }
}

// initial run returns support code library
const { support } = await runCucumber(runnerOptions)

// subsequent run reuses support code library
await runCucumber({
    ...runnerOptions,
    support
})

[nicojs]

That works for us 👍

[jan-molak]

Works for us too 👍🏻

[rkrisztian]

I really think that something like this would be extremely useful as an alternative for the great mess we have today with testing tool integrations (Jest, Cypress), for example I found these problems (in the order of importance):

• cypress-cucumber-preprocessor does not support tags on Examples ([feature request] use tags for example tables for scenarios with more example tables badeball/cypress-cucumber-preprocessor#196)
• jest-cucumber does not support Cucumber JSON reporting (Reports in json bencompton/jest-cucumber#27)
• cypress-cucumber-preprocessor generates multiple Cucumber JSON reports with no offical support for aggregation (How to get a single cucumber.json file aggregate all features badeball/cypress-cucumber-preprocessor#423)
• jest-cucumber is not as convenient as jest-cucumber-fusion
• there's also cucumber-jest...
• Karma does not have a working implementation anymore (Cucumber + Karma #1095)

I would rather see some minimal glue code between Jest/Karma/Cypress/etc. and cucumber-js so I don't have to suffer for all those missing features I need to use.

[mattwynne]

Great suggestion @davidjgoss 👍

This separation of concerns between the command-line user interface and the "business logic" of parsing and executing scenarios as tests reminds me of the hexagonal architecture pattern.

In cucumber-ruby we actually split the core domain logic (or "inner hexagon") into a separate gem package, as we were rebuilding it from scratch in a "clean room". I realise that's not the context here, but it might be worth drawing some inspiration from, or feeding back innovations from this design into the Ruby API. There's an example in the cucumber-ruby-core gem's README of how to use that API.

[davidjgoss]

Okay, here's a first pass at the API signature for the "run" bit. It's heavily based on the IConfiguration object we have internally (so shouldn't cause too much refactoring lower down) but just a little less "flat":

export interface IRunCucumberOptions {
  cwd: string
  features: {
    defaultDialect?: string
    paths: string[]
  }
  filters: {
    name?: string[]
    tagExpression?: string
  }
  support:
    | {
        transpileWith?: string[]
        paths: string[]
      }
    | ISupportCodeLibrary
  runtime: {
    dryRun?: boolean
    failFast?: boolean
    filterStacktraces?: boolean
    parallel?: {
      count: number
    }
    retry?: {
      count: number
      tagExpression?: string
    }
    strict: boolean
    worldParameters?: any
  }
  formats: {
    stdout: string
    files: Record<string, string>
    options: IParsedArgvFormatOptions
  }
}

export interface IRunResult {
  success: boolean
  support: ISupportCodeLibrary
}

export async function runCucumber(
  options: IRunCucumberOptions
): Promise<IRunResult> {
  // do stuff
}

And a very contrived example usage:

const result = await runCucumber({
  cwd: process.cwd(),
  features: {
    paths: ['features/**/*.feature'],
  },
  filters: {
    name: ['Acme'],
    tagExpression: '@interesting',
  },
  support: {
    transpileWith: ['ts-node'],
    paths: ['features/support/**/*.ts'],
  },
  runtime: {
    failFast: true,
    retry: {
      count: 1,
      tagExpression: '@flaky',
    },
    strict: true,
    worldParameters: {
      foo: 'bar',
    },
  },
  formats: {
    stdout: '@cucumber/pretty-formatter',
    files: {
      'report.html': 'html',
      'TEST-cucumber.xml': 'junit',
    },
    options: {
      printAttachments: false,
    },
  },
})

Feedback welcome! Note this doesn't cover the profile/config loading stuff which would be another function.

[nicojs]

I think this looks good. Question: How would I configure a custom formatter?

[davidjgoss]

@nicojs sorta like on the CLI

formats: {
    files: {
      'report.html': './my/fancy-reporter.js',
      'other-report.html': '@me/reporter-package',
    }
  },

[davidjgoss]

did we released 8.0.0-rc.1 a little bit early?

Yes, I think so in retrospect.

[davidjgoss]

8.0.0-rc.3 has been released including this new API, there are some docs here - a little thin at the moment but will continue to expand them.

If anyone has any early feedback that'd be fantastic (cc @jan-molak @nicojs @binarymist).

[MarufSharifi]

Hey thanks for doing this great job, I need to use from @cucumber/cucumber: 8.0.0, when it will be available to use?

[davidjgoss]

@MarufSharifi within the next week or two, but you can try 8.0.0-rc.3 in the meantime if that helps.

[binarymist]

Thanks @davidjgoss .

https://github.com/cucumber/cucumber-js/releases/tag/v8.0.0-rc.3 links to https://github.com/cucumber/cucumber-js/blob/v8.0.0-rc.3/docs/cli.md#printing-attachments-details which doesn't appear to exist.

---

I'm assuming https://github.com/cucumber/cucumber-js/blob/main/docs/javascript_api.md#sources-example is what I'm looking for to recreate our testplan functionality?

runConfiguration doesn't seem to contain anything of value.

loadSources(runConfiguration.sources) returns undefined

Can you please explain how this can replace our existing testPlan functionality, maybe even provide some tips to help with the guess work?

Thanks.

[aurelien-reeves]

Thanks @davidjgoss .

https://github.com/cucumber/cucumber-js/releases/tag/v8.0.0-rc.3 links to https://github.com/cucumber/cucumber-js/blob/v8.0.0-rc.3/docs/cli.md#printing-attachments-details which doesn't appear to exist.

Fixed, thanks for the notification

[aurelien-reeves]

I'm assuming https://github.com/cucumber/cucumber-js/blob/main/docs/javascript_api.md#sources-example is what I'm looking for to recreate our testplan functionality?

runConfiguration doesn't seem to contain anything of value.

loadSources(runConfiguration.sources) returns undefined

Can you please explain how this can replace our existing testPlan functionality, maybe even provide some tips to help with the guess work?

Thanks.

loadConfiguration returns a runConfiguration and a useConfiguration. useConfiguration is the resolution of the configuration that the API has computed from the configuration file, environment, and other given parameters.

In your case, how does it looks like? If it does not looks as expected, give the proper argument to loadConfiguration method.

Refs to the documentation of the API:

• IResolvedConfiguration: https://github.com/cucumber/cucumber-js/blob/main/docs/api/cucumber.iresolvedconfiguration.md
• LoadConfiguration method: https://github.com/cucumber/cucumber-js/blob/main/docs/api/cucumber.loadconfiguration.md

If you are still not able to make it work, please provide more info to us like your piece of code, what is expected, and what is the actual result

[binarymist]

What is the configuration file in this context? We have a set of environment application config files with the following relevant cucumber properties in them:

cucumber: {
    features: {
      doc: 'The location of the feature files.',
      format: String,
      default: 'src/features'
    },
    steps: {
      doc: 'The location of the step files.',
      format: String,
      default: 'src/steps'
    },
    binary: {
      doc: 'The location of the Cucumber binary.',
      format: String,
      // default: `${process.cwd()}/node_modules/.bin/cucumber-js`
      default: path.join(process.cwd(), '/bin/purpleteamCucumber.js')
    },
    timeout: {
      doc: 'The value used to set the timeout (https://github.com/cucumber/cucumber-js/blob/master/docs/support_files/timeouts.md)',
      format: 'duration',
      default: 5000
    }
  },

Is this what you're refering to as "the configuration file"? We have no other cucumber specific configuration file. Sadly the file documentation tells us nothing of what should be inside it. Is the contents described else where?

It's still not clear how to get from where we currently are (as clearly specified above) and reiterated again below for your convenience to the new API:

1. For the testplan command we...
2. createCucumberArgs
3. We pass the new CucCli instance to getActiveFeatureFileUris
4. Which calls the legacy getConfiguration method of the cucumber CLI instance which no longer exists

I'd imagine that because we've had the same functionality working for a long time previously then we obviously have all the information that cucumber needs, it's just a matter of how do we get the information into a form that the new API can deal with?

So it looks like the new loadConfiguration can replace the legacy CLI getConfiguration. I'm not sure if what it returns though provides us with the pickleFilterOptions and featurePaths that we need as seen in our getActiveFeatureFileUris?

I'm also unsure how we get from the current not passing anything to the CLI's getConfiguration method to the arguments (loadConfigurationOptions and runEnvironment) that the new loadConfiguration expects?

I've looked at both ILoadConfigurationOptions and IRunEnvironment docs.

• I don't understand what the file is, I don't think we have one of these, so how do we get to that from out current state?
• I'm not sure what is meant by profiles, I don't think we've used these before?
• It's also unclear what providedis?
• The cwd is obvious enough
• env kind of makes sense
• I'm not sure why stderr and stdout are required if they are? We havent been specifying these before that I'm aware of.

Thanks.

[davidjgoss]

@binarymist so for the test plan case specifically you shouldn't need to use the Cli class at all. loadSources from the new API should, I think, give you what you need.

The loadConfiguration method is designed to source configuration from the standard cucumber configuration file that a user might have in their project, apply some defaults where nothing is specified, and return it in a shape that works for the other API functions. For your case it seems configuration is managed in its own way and you already have it to hand at runtime, so you could use the provided option to include that directly, and omit the file and profiles which wouldn't matter. This would be similar to what you do with createCucumberArgs and then passing that to the Cli, except it's in object form rather than an argv-style string, something like:

const { runConfiguration } = await loadConfiguration({
  provided: {
    paths: [`${this.#cucumber.features}/${sessionProps.sUtType}`],
    require: [`${this.#cucumber.steps}/${sessionProps.sUtType}`],
    format: [`message:${this.#results.dir}result_appScannerId-${sessionProps.testSession ? sessionProps.testSession.id : 'noSessionPropsAvailable'}_${this.#strings.NowAsFileName('-')}.NDJSON`],
    // etc
  }
})

runConfiguration.sources should then be what you need to give to loadSources, which returns an object with a plan property, which is a (pre-filtered by paths, tags, names etc) array of objects each representing a pickle (test case) to be run.

Both functions have an optional environment argument which you mentioned - you don't need to provide this unless you need any of the values to be different from the defaults, which are:

{
  cwd: process.cwd(), // files would be loaded relative to this dir
  stdout: process.stdout,
  stderr: process.stderr,
  env: process.env,
}

I hope this helps. I appreciate the docs are a little thin right now but they'll improve quickly and feedback like this is helpful.

[garfieldnate]

I might be too late to the party to give input, but the API still seems fairly similar to running cucumber from the CLI, and I'm having some trouble understanding how to apply this for my use case.

I want to write a test file that contains all of the steps, then run those steps on one or more associated feature-files. I'm using TS, and I want to be able to compile the test and then run it repeatedly as I develop the feature file without having to re-do the transpilation step repeatedly, which slows down the development loop.

It's already been mentioned that I can use loadConfiguration({provided: {...}}) to define the config inline. I'm unsure of how to do the same for loadSupport; I need to create an ISupportCodeLibrary from the steps and world in the current file, but there's no documentation on how to construct this.

[binarymist]

Thanks @davidjgoss that was really helpful to get testPlan working.

It looks like the update from rc.2 to rc.3 may have broken our test functionaly?

Have you seen this error before:

Error [ERR_REQUIRE_ESM]: require() of ES Module /usr/src/app/src/steps/BrowserApp/app_scan_steps.js from /usr/src/app/node_modules/@cucumber/cucumber/lib/api/support.js not supported.
 Instead change the require of app_scan_steps.js in /usr/src/app/node_modules/@cucumber/cucumber/lib/api/support.js to a dynamic import() which is available in all CommonJS modules.
     at /usr/src/app/node_modules/@cucumber/cucumber/lib/api/support.js:14:32
     at Array.map (<anonymous>)
     at getSupportCodeLibrary (/usr/src/app/node_modules/@cucumber/cucumber/lib/api/support.js:14:18)
     at runCucumber (/usr/src/app/node_modules/@cucumber/cucumber/lib/api/run_cucumber.js:33:53)
     at async Cli.run (/usr/src/app/node_modules/@cucumber/cucumber/lib/cli/index.js:45:29)
     at async run (file:///usr/src/app/src/scripts/runCuc.js:75:14) {
   code: 'ERR_REQUIRE_ESM'
 }

app_scan_steps.js is here
runCuc.js is here

I've created a diff where the only difference is rc.2 vs rc.3 which has confirmed that this error doesn't exist with rc.2 and is introduced with rc.3.

Thanks.

[binarymist]

There doesn't appear to be a lib/api/support.js in the root of the cucumber repo, but I found a src/api/support.ts which appears to be new in rc.3 and it appears to be requireing our ESM, so as expected the error makes sense.

So it seems that cucumber in rc.2 supports ESM but rc.3 has regressed to only supporting CJS?

[davidjgoss]

@binarymist it’s still there but uses a specfic option for import vs require now - previously it was inferred but the heuristics were too fuzzy so we changed it to be explicit:

• esm: use specific option for import vs require for support code #1931
• https://github.com/cucumber/cucumber-js/blob/main/docs/esm.md

[davidjgoss]

(We should add better messaging for that, thanks for raising.)

[davidjgoss]

@garfieldnate just to make sure I understand, you want to:

• Load (including transpile) the support code once
• Use that support code on multiple test runs, triggered when you change a feature file

If so, I think you could implement this now. You can't create the support code library inline, but you can load it once and use it multiple times. Something like:

import {loadConfiguration, loadSupport, runCucumber} from '@cucumber/cucumber/api'

const {runConfiguration} = await loadConfiguration()
const support = await loadSupport(runConfiguration)
const merged = {...runConfiguration, support}

// do this bit as many times as you want e.g. in a loop or callback
await runCucumber(merged)

[binarymist]

Thanks @davidjgoss.

Initial testing has the import working. It seems that the colour characters (green ticks, red crosses, dashes, etc) that are usually written are no longer colour. We haven't changed the '--format-options', '{"colorsEnabled": true}',. Has something else been changed around the formaters?

Thanks.

[aurelien-reeves]

That seems to be an issue.
Indeed there have been recent changes under the hood, but not related to the API

Do you think you could submit a new issue for that, with a minimal reproducible example?

[davidjgoss]

Worth noting we switched from using colors to chalk in a hurry for well-known reasons, could be related to that.

[binarymist]

Do you think you could submit a new issue for that, with a minimal reproducible example?

Currently we're going through the release phase with no colours (as a known issue). We're unable to keep PurpleTeam users waiting any longer. Will try and circle back to this. I'm not sure how minimal a seperate reproducable example would be. My guess is that everyone else expecting colours on this version will also be facing the same problem. It's just running the CLI with '--format-options', '{"colorsEnabled": true}'
Nothing else special, we've been running the CLI this way for years.

[aurelien-reeves]

It's just running the CLI with '--format-options', '{"colorsEnabled": true}'
Nothing else special, we've been running the CLI this way for years.

I already tried that, but I cannot reproduce your issue

How do you pass the format options to cucumber? Through the CLI directly? Using a config file?

I am puzzled with the format you are using. Why such format? I mean: '--format-options', '{"colorsEnabled": true}': how do you pass those parameters to cucumber?

First of all, that option is true per default, so you should be able to not specifying that option.
Also, why that coma between format-options and the options? And the quotes? I would suggest trying this instead: --format-options '{"colorsEnabled": true}'

[davidjgoss]

@aurelien-reeves I think that's from running the Cli programmatically where you provide argv as string[]

I think I might see the/an issue here, going to raise something new though @binarymist as it's not API related per se.