Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests in parameterized 'test.describe' block are reported as retries #732

Closed
mihalichzh opened this issue Jul 20, 2023 · 1 comment · Fixed by #761
Closed

Tests in parameterized 'test.describe' block are reported as retries #732

mihalichzh opened this issue Jul 20, 2023 · 1 comment · Fixed by #761
Labels
theme:playwright type:bug Something isn't working

Comments

@mihalichzh
Copy link

mihalichzh commented Jul 20, 2023

Hi guys!

Describe the bug
Tests are unexpectedly reported as retries when wrapped with parameterized describe block.

To Reproduce

  1. Clone repo https://github.com/mihalichzh/allure_playwright_issue
  2. Run tests using 'npx playwright test'
  3. Generate and open allure report

Expected behavior
Tests in parameterized describe block are not reported as retries. Each describe block is expected to be treated as a separate test item if parameterized (as it is in native 'html' reporter).

Additional context
Reproducible with (at least) playwright 1.34.3/1.36.1 and allure-playwright 2.4.0.

Playwright html reporter:
Screenshot 2023-07-20 at 23 19 16

Allure:
Screenshot 2023-07-20 at 23 18 38

@vovsemenv vovsemenv added theme:playwright type:bug Something isn't working labels Jul 21, 2023
@nbaldzhiev
Copy link

nbaldzhiev commented Aug 15, 2023

I can confirm that tests in a parameterised describe block are reported as retries ("allure-playwright": "^2.5.0","@playwright/test": "^1.36.2")

This is very confusing because when you open the Tree of the Launch, for the given file you only see the last parameterised run, so it looks like only those tests ran. However, when you click on a given test and go to retries, you actually see these are not "retries", but are the other parameterised tests. In our case, there's a file with 1 describe block, which contains 4 tests and is parameterised against 3 values (to be fully precise, parameterised values are 5, i.e. we have an array of 5 values, but the last two are skipped, so we run against the 0-2 values). In Allure TestOps launch tree view, this file is reported to have ran 8 tests; but when you click each test and go to retries, there are 2 more tests, which are the other 2 parameterised values. We only see the run against the last parameterised value in the launch tree view for each browser and the runs against the previous 2 parameterised values are only visible in the "retries" section. So, when we ran against 2 browsers, the launch's tree view showed only 8 ran tests in that file, but in reality 24 tests were ran (2 browsers * 3 parameterised values * 4 tests) - the rest of the tests were only visible in the retries section of each test visible in the launch tree view.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
theme:playwright type:bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants