Provide a way to selectively execute tests

[ryust-ax]

What are you trying to achieve?

Selectively run a subset of tests without "skip"ing all the rest

What do you get instead?

At the moment, I have a mechanism in a hook to skip a test if it matches a set of criteria, one of which being a relevant tag being present in a list.
example: #661

Provide console output if related. Use --verbose mode for more details.
n/a

Provide test source code if related
n/a - this is extremely broadly applicable to any and all tests, it's only specific to the use of @tag notations in test scenario strings.

But a basic example:

const { I } = inject();
Feature("Some Feature");

Scenario("A scenario description @tag1 @skip @tag2", async () => {
  console.log("Execute some code here");
});

@skip @otherTags
Scenario: Some scenario
  Given I do something first
  When I do another thing
  Then I see a result

Details

• CodeceptJS version: 3.3.6
• NodeJS Version: 14.19.2
• Operating System: Ubuntu 20.04
• puppeteer || webdriverio || testcafe version (if related): possibly related: playwright
• Configuration file: n/a

Hack Implementation

I'm not really sure how to do all the things necessary to offer a PR on this project. I'm also not sure if this project is appropriate, or if it's more appropriate to offer a solution to the Allure project (https://github.com/allure-framework/allure-js) or the Mocha project (https://github.com/mochajs/mocha) given the motivation and maybe draft? solution(s) for mocha and/or CodeceptJS as referenced https://docs.qameta.io/allure-testops/integrations/test-frameworks/.

So, I've built what will roughly do what I need in my own codebase, though I'll need to patch it after every update.

A test scenario, (bdd or plain-code, doesn't matter)

@tag1
Scenario: Some scenario
  Given I do something first
  When I do another thing
  Then I see a result

In codecept.conf.js (where the property value is an array, but could also be a function result after importing and transforming an external file content).

module.exports = {
  ...
  selective: ["tag1", "tag2"],
  ...
}

In lib/container.js, add the if (config.selective) block

...
class Container {
  ...
  static create(config, opts) {
    const mochaConfig = config.mocha || {};
    if (config.grep && !opts.grep) {
      mochaConfig.grep = config.grep;
    }
    if (config.selective) {
      mochaConfig.selective = config.selective;
    }
    this.createMocha = () => {
      container.mocha = MochaFactory.create(mochaConfig, opts || {});
    };
    ...
  }
 ...
}
...

In lib/mochaFactory::create(), added a new prototype definition on the Mocha object and then call the function; on or about line 44/45 in CodeceptJS v3.3.6

        Mocha.prototype.selective = function(selective) {
          if (!selective) {
            return this;
          }
          if (!Array.isArray(selective)) {
            throw new Error("Must provide an array.");
          }
          const re = selective.join("|");
          const regex = new RegExp(re);
          this.grep(regex);

          return this;
        };

        mocha.selective(config.selective);

Weakness/Uknowns

• I don't know whether this is the "right" place or even the correct way of adding this code, but it's the most concise I've found that seems to at least work with the codeceptjs run command and both BDD and plain-code tests.
• I'd love to be able to use a command line parameter like --selective, maybe even with a file path, but I couldn't figure how to modify the command prompt for codecept and have it follow through into mocha.
• This leverages regex/grep which mocha seems to really advise against; preferring instead that developers utilize the .only() methodology. However, I couldn't figure out how to call the .only() method or any related functionality via a programatic before.test or before.suite event; like I was able to programmatically skip() as described in issue 661.
• This is important because programmatically skipping 450 out of 500 tests when I only want to execute a specific list of 50 makes the reporting a huge mess. Plus, I couldn't find this capability so as to be able to fully implement Allure / Allure TestOps.

Summary

Hopefully someone with greater experience in CodeceptJS and/or Mocha can see this and maybe this will provide some inspiration for a quick(er) patch. If not, then anyone else coming along might be able to use this to patch their own installation.

[ryust-ax]

On second (or third) examination, this can be done via bootstrap in the codecept.conf.js:

const { container } = require("codeceptjs");

exports.config = {
  ...
  bootstrap: async function () {
    const mocha = container.mocha();
    mocha.selective = function(selective) {
      if (!selective) {
        return false;
      }
      if (!Array.isArray(selective)) {
        throw new Error("Must provide an array.");
      }
      const re = selective.join("|");
      const regex = new RegExp(re);
      mocha.grep(regex);

    };

    mocha.selective(["tag1", "tag2"]);

    return null;
  },
  ...
}

Where the array in selective() could be provided programmatically with or without an environment variable

UPDATE
I cleaned up this code to avoid having to attach a new method to the mocha object and my implementation involved extracting this code into a standalone object w/ its own methods. I'm not posting the final result (it's cluttered with console.log for debugging purposes), but I'd encourage anyone messing with this to experiment with cleaner code or implementation. Leaving as is for reference (and something more like what I've got here might be needed if implemented deeper than the configuration bootstrapping).

[DavertMik]

Looks like a good approach and nice example of using bootstrap!
Thank you!