17.11 - error output is treated as error and fails tests

[birarroshan]

ℹ️ workarounds

Set ForwardStandardOutput to false in your runsettings:

<RunSettings>
  <RunConfiguration>
    <ForwardStandardOutput>false</ForwardStandardOutput>
  </RunConfiguration>
</RunSettings>

or

set VSTEST_DISABLE_STANDARD_OUTPUT_FORWARDING=1 environment variable.

---

Description

We are running Appium-WinAppdriver UI tests with ADO pipeline. After upgrading to 17.11.0, VStest task is reporting appium debug logs as error.

Steps to reproduce

1. Install 17.11.0 for VSTestPlatform - Latest Stable is 17.11.0

• task: VisualStudioTestPlatformInstaller@1
  inputs:
  packageFeedSelector: 'nugetOrg'
  versionSelector: 'latestStable'

2. Run test using
   • task: VSTest@2
     inputs:
     testSelector: 'testAssemblies'
     testAssemblyVer2: "Test.dll"
     searchFolder: $(Pipeline.Workspace)
     testFiltercriteria: "$(testFilterExpression)"
     vsTestVersion: 'toolsInstaller'

Expected behavior

Tests should run as usual without any error logs

Actual behavior

Appium DEBUG logs are being reported as Error.

Note - We have not enabled any debug logging for appium

Also, specFlow logs are reported as Warning.

Diagnostic logs

Could not collect Diagnostic logs. Will upload it one collected.

Environment

OS - Hosted Pool with windows 10 base image

Aditional Notes -

ADO job is passing, but all errors are reported under Error tab of pipeline run

[shapea]

I can confirm the same issue in vstest.console.exe as of version 17.11.0. As soon as the update was released, I had ADO build pipelines with no other changes start to fail with stderr output. Digging further into it, there seems to be a bug with some of the new output forwarding changes in the 17.11.0 release as noted here: https://github.com/microsoft/vstest/releases.

@nohwnd if bugs were ducks, this certainly walks, talks and quacks like a duck. From what I can tell, it appears that stdout output from tests gets erroneously forwarded to stderr instead. This causes benign test output to become errors. I can even verify this by manually calling vstest.console.exe with tests exhibiting this behaving and sending stderr output to nul (i.e. 2> nul).

I managed to sleuth out a workaround for this issue using the information here: #4998

For folks impacted by this, it seems you can update your test .runsettings to include this to disable the fowarding causing the issues:

<RunSettings>
  <RunConfiguration>
    <ForwardStandardOutput>false</ForwardStandardOutput>
  </RunConfiguration>
</RunSettings>

@birarroshan this may help unblock you, it did for me in ADO for the same reason. :)

[nohwnd]

The change was introduced here:

135b4f2#diff-d494c9e9058f3e044a79a007c308ad13da804fd3fb6d404462c09e3cd8003e50R59

This change then gets enabled to default in 17.11.

The ForwardStandardOutput option set to false is effective because it prevents forwarding the error stream as error messages.

it appears that stdout output from tests gets erroneously forwarded to stderr instead.

The appium logs go into error stream, not into standard output. All the streams get forwarded correctly, but you are right that this is not the desired behavior and the error stream should be sent as info messages instead, not as errors, because applications like to use error stream as output for logs, or info messages.

@birarroshan there is also VSTEST_DISABLE_STANDARD_OUTPUT_FORWARDING=1 env variable, if that is easier for you to set. as documented in #4998

[nohwnd]

Re-opening for awareness.

This bug affects VS17.11, TestPlatform 17.11, net8.0, and all newer versions.

Each product has it's own servicing calendar, will update this thread when fixes are released.

• release 17.11.1 to nuget
• insert 17.11.1 in VS
• insert 17.11.1 into net8
• insert 17.12.x into net9
• insert 17.12.x into net10

[shapea]

@nohwnd in my case it's not coming from Appium, but rather an internal-only MS thing.

Edit: It took a while to pull in symbols and sources to step deeper into their native pinvoke'd code, but it's doing the same sort of thing and writing non-error output to stderr as well.

[nohwnd]

@nohwnd in my case it's not coming from Appium, but rather an internal-only MS thing.

Edit: It took a while to pull in symbols and sources to step deeper into their native pinvoke'd code, but it's doing the same sort of thing and writing non-error output to stderr as well.

Thanks for confirming.

[bradyclifford]

If you add VSTEST_DISABLE_STANDARD_OUTPUT_FORWARDING: "1" to your runner, it does fix the issue.

However, it does prevent output from ITestOutputHelper - if you are even using it. Which is helpful when running tests because it groups the logs and shows them in each test, segregated from the other logs. It does the same when creating a report in GitHub on the runner.

If you use GitHub Actions, you can instead use the suggested GitHub Actions test logger mentioned in .Net's docs. Using this test logger, I was able to avoid declaring the VSTEST_DISABLE_STANDARD_OUTPUT_FORWARDING env. var and still output errors in my logs.

[nohwnd]

@bradyclifford this should be fixed in 17.11.1 do you still see the issue in that version?

[carstencodes]

Hi @nohwnd

we had the same issue here. The tests were failing with .NET SDK 8.0.401, which is shipping VsTest 17.11.0.

All tests passed, but the entire test suite was reported as failed due to messages on stderr.

I was not able to reproduce the behavior on my machine using .NET SDK 8.0.403, which ships VsTest 17.11.1

To me, the issue looks solved.

Regards

Carsten