Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not count skipped as passed in test cmd #230

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ihoro
Copy link
Member

@ihoro ihoro commented Sep 25, 2024

It changes only output of 'kyua test' CLI command. Hence, other outputs
like junit are kept intact for CI and other use cases. It's meant to
improve UX of attended use cases.

The issue is that the following can be tricky to interpret:

  222/222 passed (0 failed)

It can be read as all tests are passed, but it might be a summary line
of all tests skipped due to some requirement is not met.

It's reworked to easily distinguish such cases:

  222/222 passed (0 broken, 0 failed, 0 skipped)
  0/222 passed (0 broken, 0 failed, 222 skipped)

The overall formula is:

  <actually passed>/<total> (<details about not actually passed ones>)

It changes only output of 'kyua test' CLI command. Hence, other outputs
like junit are kept intact for CI and other use cases. It's meant to
improve UX of attended use cases.

The issue is that the following can be tricky to interpret:

  222/222 passed (0 failed)

It can be read as all tests are passed, but it might be a summary line
of all tests skipped due to some requirement is not met.

It's reworked to easily distinguish such cases:

  222/222 passed (0 broken, 0 failed, 0 skipped)
  0/222 passed (0 broken, 0 failed, 222 skipped)

The overall formula is:

  <actually passed>/<total> (<details about not actually passed ones>)
@ihoro
Copy link
Member Author

ihoro commented Sep 25, 2024

It mirrors https://reviews.freebsd.org/D46653.

@emaste
Copy link
Member

emaste commented Nov 13, 2024

I prefer the reporting that @kprovost suggested in the FreeBSD review. "7/7 passed (and 3 were skipped)" is more immediately obvious to me than "7/10 passed (the three that did not pass were skipped)." Or don't print as a ratio. "7 passed, 0 failed, 3 skipped, 10 total"

Copy link
Contributor

@ngie-eign ngie-eign left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm on board with @emaste . This format change can result in user confusion about how many failures there actually were when executing tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants