Can we run the tests by cucumber tags like cucumber-js --tags "@foo or @bar"

[niroshagithub]

No description provided.

[vitalets]

Short answer: no (for now)
Long answer:
Playwright itself has an opportunity to run tests by tags like follows:

npx playwright test --grep "@foo|@bar"

The problem here is that Playwright's tags can be defined only in test titles:

test('Test login page @fast', async ({ page }) => { ... });
test('Test full report @slow', async ({ page }) => { ... });

With playwright-bdd you can append tags to scenario name and it will work:

Feature: my feature

    Scenario: Test login page @fast
       ...

Run with tags:

npx bddgen && npx playwright test --grep "@fast"

But this is not how we used to work with tags in Cucumber. What I want is to define tags in usual way:

Feature: my feature
    
    @fast
    Scenario: Test login page
       ...

Currently playwright-bdd reads such Cucumber tags but does not append them to test titles for Playwright. This is done intentionally to keep test titles unchanged, as they are defined by user. From BDD perspective it is not expected that tags will be concatenated to scenario name. This also can break some reporting.

The ideal solution is to have alternative way of defining tags in Playwright - without modifying test titles. I've already created issue for that microsoft/playwright#23180, you can add thumbs there :)

Until this is done I think of option in playwright-bdd to let user define which tags should be appended to test titles. For example, in config:

const testDir = defineBddConfig({
  appendTagsToTitle: ['@fast'],
  ...
});

Then the following scenario

Feature: my feature

    @fast
    Scenario: Test login page
       ...

will be converted into:

test('Test login page @fast', async ({ page }) => { ... });

and you will be able to run only @fast tests:

npx bddgen && npx playwright test --grep "@fast"

[jzaratei]

In my project, I'm using the playwright tagging approach. At the moment, appending the tag in the scenario name does not look so bad, in fact, it adds a little more information about the scenario. But I'm agree if I wanted to add more tags, it would look overloaded.

Additionally, my scenarios already have a [testId] tag because, I'm using alexneo lib to do the integration with azure testplans.

    Scenario: [442904] @project_create Create a project with optional info
        When I select add project
...
    Scenario: [443111] @project_edit Edit project files

On the other hand, appendTagsToTitle could be useful. Let's say, I have a global tag called login in the feature name. What would I expect is to have that tag in each of every scenario related to that feature.

@login
 Feature: Do login
  Scenario: Login ok
   ...
  Scenario: Login ko

test('Login ok @login', async ({ page }) => { ... });
test('Login ko @login', async ({ page }) => { ... });

[vitalets]

Will add appendTagsToTitle in the future release.
Btw, for @login case - will not it be enough to append this tag to test.describe only, not to every test? E.g.:

test.describe('Do login @login', () => {
  test('Login ok', async ({ page }) => { ... });
  test('Login ko', async ({ page }) => { ... });
});

Instead of:

test('Login ok @login', async ({ page }) => { ... });
test('Login ko @login', async ({ page }) => { ... });

[jzaratei]

@vitalets you're right, I did a test appending the tag in the test.describe() and it seems to work.

[magarwal19]

Looking forward for this feature, as this would also be helpful in case we want to run multiple scenarios based on tags like
@smoke or @regression

[vitalets]

Looking forward for this feature, as this would also be helpful in case we want to run multiple scenarios based on tags like @smoke or @regression

Ok, already in progress.

For now you can workaround it by adding tags manually to feature/scenario names

@smoke
Feature: Do login @smoke
...

[alescinskis]

also really looking forward to this one, this was very useful in other company where we've used cucumber+webdriverio, and were able to launch specific groups of tests based on provided tags.
similar for playwright-bdd would be immensely appreciated.

[NikkTod]

@alescinskis, @magarwal19, just as information, you already have that possibility, as vitalets wrote above, you can append the tags in the scenario title and use this example command to run only those tests that has @fast tag in the scenario name.

npx bddgen && npx playwright test --grep "@fast"

[vitalets]

@alescinskis, @magarwal19, just as information, you already have that possibility, as vitalets wrote above, you can append the tags in the scenario title and use this example command to run only those tests that has @fast tag in the scenario name.

npx bddgen && npx playwright test --grep "@fast"

I think this solution is ok for newly created features.
But for large existing codebase of features this is not a good way to go as you need to update a lot of files with a strange change - copy tag into title.
Hope to release appendTagsToTitle very soon.

[magarwal19]

@vitalets @NikkTod
thank you for quick solutions!

I am using grep for a single tag, but i am also looking for a option where i can give multiple tags,
for eg:

let us say i want to executed four tests which has tag
@test-1
@test-2
@Test-3
and @test-4

there are other tests also in repository with tags 5/6/7/8 and so on.
So how can i run these four tests only?

[magarwal19]

Looking forward for this feature, as this would also be helpful in case we want to run multiple scenarios based on tags like @smoke or @regression

Ok, already in progress.

For now you can workaround it by adding tags manually to feature/scenario names

@smoke
Feature: Do login @smoke
...

For my framework, we already have tagging done at scenario level, but need support to run scenarios which match any of given tags

[vitalets]

Released v5.1.0 with tags filtering support.

While implementing I've realized that instead of Playwright's --grep we can filter scenarios with --tags option during bddgen call. At the end it will output only filtered test files and filtering in Playwright call is not needed.
Personally, I like this approach more, because:

1. clean syntax of tags expressions instead of Playwright's regexp
2. keeping test titles unmodified
3. only needed files are generated (that is faster)

Example command:

npx bddgen --tags "@foo or @bar" && npx playwright test

@magarwal19 for your case with several tags:

npx bddgen --tags "@test-1 or @test-2 or @test-3 or @test-4" && npx playwright test

Although appendTagsToTitle is basically ready I didn't merge it for now.
I suggest to go with that simple approach, feel free to share your experience.
If we find any cases not covered by --tags option, I will merge it for sure.
Thanks in advance!

[NikkTod]

@vitalets, that sounds great, I very much like the --tags option approach

[alescinskis]

I have an issue when trying to run tests by tags, it checks incorrect path for screenshots.

when trying to launch without specifying tags:

arturslescinskis@Arturs-Lescinskis automation % npx bddgen && npx playwright test autoplayModified.feature --reporter=list,allure-playwright --workers=2 --project=firefox

  ✓  1 [firefox] › popups/autoplayModified.feature.spec.js:6:7 › Popups › Autoplay Modified: headerless enabled (1.2s)

  1 passed (3.6s)

When adding a @smoke tag to that particular scenario, it checks screenshot folder with different path, and since it doesn't exist, creates screenshot (I'm asserting with .toHaveScreenshot():

arturslescinskis@Arturs-Lescinskis automation % npx bddgen --tags "@smoke" && npx playwright test autoplayModified.feature --reporter=list,allure-playwright --workers=2 --project=firefox

Running 1 test using 1 worker

  ✘  1 [firefox] › autoplayModified.feature.spec.js:6:7 › Popups › Autoplay Modified: headerless enabled (1.1s)
  ✘  2 [firefox] › autoplayModified.feature.spec.js:6:7 › Popups › Autoplay Modified: headerless enabled (retry #1) (2.2s)


  1) [firefox] › autoplayModified.feature.spec.js:6:7 › Popups › Autoplay Modified: headerless enabled

    Error: A snapshot doesn't exist at /Users/arturslescinskis/GIT/pp-experiment/libraries/automation/tests/visual-regression/screenshots/autoplayModified.feature.spec.js-snapshots/Autoplay-Modified-headerless-enabled-firefox-darwin.png.

screenshots are actually saved under libraries/automation/tests/visual-regression/screenshots/popups/autoplayModified.feature.spec.js-snapshots
feature file is located: libraries/automation/tests/visual-regression/features/popups/autoplayModified.feature
in playwright.config.ts I have:

const testDir = defineBddConfig({
  paths: ['tests/visual-regression/features'],
  require: ['tests/visual-regression/steps/*.ts'],
  outputDir: 'tests/visual-regression/screenshots/',
});

[vitalets]

@alescinskis yes, it's a bug, thanks for reporting!
Common parent directory for test files should be the same as it is without tags filtering.
Will make a hotfix soon.

[vitalets]

@alescinskis could you check with playwright-bdd@5.1.1 ?

[alescinskis]

previous issue no longer reproducible

without tags

arturslescinskis@Arturs-Lescinskis automation % npx bddgen && npx playwright test autoplayModified.feature --reporter=list,allure-playwright --workers=2 --project=firefox

Running 31 tests using 2 workers

  ✓  1 [firefox] › popups/autoplayModified.feature.spec.js:6:7 › Popups › Autoplay Modified: headerless enabled (1.4s)
  ✓  2 [firefox] › popups/autoplayModified.feature.spec.js:13:9 › Popups › Autoplay Modified: translations › Autoplay 
  ...
  ✓  33 [firefox] › popups/autoplayModified.feature.spec.js:158:9 › Popups › Autoplay Modified: translations › Autoplay Modified: translations, locale set to vi (640ms)

  2 flaky
  29 passed (16.7s)

with tags

arturslescinskis@Arturs-Lescinskis automation % npx bddgen --tags @smoke  && npx playwright test autoplayModified.feature --reporter=list,allure-playwright --workers=2 --project=firefox

Running 1 test using 1 worker

  ✓  1 [firefox] › popups/autoplayModified.feature.spec.js:6:7 › Popups › Autoplay Modified: headerless enabled (1.0s)

  1 passed (3.5s)

[vitalets]

Closing as fixed.