Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(junit): fix testsuites time to be sum of all testsuite items #6985

Merged

Conversation

saitonakamura
Copy link
Contributor

@saitonakamura saitonakamura commented Nov 29, 2024

not the total time of execution

Description

This addresses 2 problems, one that is general to junit reporter itself, but more debatable, and the other one is merge-reports specific, but is clearly a bug. Let's start with the first one.

  1. I haven't found an official junit.xml spec (and I'm not sure if it exists at all), but this document https://github.com/testmoapp/junitxml?tab=readme-ov-file#complete-junit-xml-example suggests that testsuites.time is an

time Aggregated time of all tests in this file in seconds

which makes sense
But the current implementation uses new Date().getTime() - this._timeStart.getTime(), so like, total execution time, including transform, prepare and so on. Which IMO should be changed to aggregated sum

  1. When using --merge-reports --reporter=junit the testsuites.time would be completely wrong (due to it being execution time of reports merging, not the tests times aggregation). This is not a junit-only problem, it also affect text reporters, but I'm not sure what is the correct behaviour for them, so I focused on junit here.

I specifically found it when merging reports, since we calculate mean test time by doing testsuites.time / (testsuites.total - testsuites.skipped), and noticed that mean time suddenly dropped down to nanoseconds (I wish it would, though!)

Xml tests are just best-effort, I'd like to use a proper parser, so let me know if they will suffice or not

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

  • It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
  • Ideally, include a test that fails without this PR but passes with it.
  • Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

  • Run the tests with pnpm test:ci.

Documentation

  • If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

  • Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

Copy link

netlify bot commented Nov 29, 2024

Deploy Preview for vitest-dev ready!

Built without sensitive environment variables

Name Link
🔨 Latest commit 225c9a4
🔍 Latest deploy log https://app.netlify.com/sites/vitest-dev/deploys/67498358c94c470008e3e634
😎 Deploy Preview https://deploy-preview-6985--vitest-dev.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@saitonakamura saitonakamura force-pushed the fix-junit-testsuites-time branch from 225c9a4 to fcb62e9 Compare November 29, 2024 11:33
Copy link
Member

@sheremet-va sheremet-va left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems correct to me

@saitonakamura
Copy link
Contributor Author

Anything else required from my side?

@sheremet-va sheremet-va merged commit ca37a06 into vitest-dev:main Dec 3, 2024
17 checks passed
@saitonakamura saitonakamura deleted the fix-junit-testsuites-time branch December 3, 2024 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Approved
Development

Successfully merging this pull request may close these issues.

2 participants