Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add aborted tests to TRX summary #4083

Closed
wants to merge 1 commit into from

Conversation

mfkl
Copy link

@mfkl mfkl commented Oct 21, 2022

Description

The current TRX report summary is missing information about various test status, such as aborted, timeout, inconclusive, etc.

Full list is here

_abortedTests = countForNonExistingResults;
_errorTests = countForNonExistingResults;
_timeoutTests = countForNonExistingResults;
_inconclusiveTests = countForNonExistingResults;
_passedButRunAborted = countForNonExistingResults;
_notRunnable = countForNonExistingResults;
_notExecuted = countForNonExistingResults;
_disconnected = countForNonExistingResults;
_warning = countForNonExistingResults;
_completed = countForNonExistingResults;
_inProgress = countForNonExistingResults;
_pending = countForNonExistingResults;

Public repro here https://github.com/mfkl/trx-missing-attributes-repro

It would be nice to have this information as part of the TRX summary, which currently only reports failed/passed test information.

I've started by adding the aborted test result info to the TRX summary. If the approach seems fine with you, I'll add the other properties so that the TRX summary is completely filled.

To write the test, I had to modify the following public API src/Microsoft.TestPlatform.ObjectModel/TestOutcome.cs which does not match the internal API src/Microsoft.TestPlatform.Extensions.TrxLogger/ObjectModel/TestOutcome.cs

Related issue

Contribution towards #2506

@mfkl
Copy link
Author

mfkl commented Oct 21, 2022

@microsoft-github-policy-service agree

@mfkl
Copy link
Author

mfkl commented Nov 18, 2022

Hi, could anyone please take a look and provide their thoughts on this?

@MarcoRossignoli
Copy link
Contributor

MarcoRossignoli commented Nov 18, 2022

Thanks @mfkl for the PR, TRX is used in more than one tool and we need to ensure that we're not breaking anybody, at the moment we're busy on higher priority tasks, we'll review and let you know soon as possible but we don't have an ETA at the moment. Thanks for the patience.

If you need it and you're in rush you can fork and use your custom implementation.

@mfkl
Copy link
Author

mfkl commented Nov 18, 2022

Understood, thanks. Will wait for your feedback!

@MarcoRossignoli
Copy link
Contributor

We did some check and for the moment we decided to not take this PR for risky edge cases, we're planning to work on a full review of the TRX format and implementation. Thanks for your contribution.

@topherhansen
Copy link

I think it's really annoying that the TRX says that all of these attributes are zero when it's just making it up. When a sessiontimeout happens (I tell it 300 seconds and it's only run half my tests) it lies and says that the total was some small part of my tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants