[WIP]Add coverlet smoke test #2330

MarcoRossignoli · 2020-02-15T10:07:56Z

MarcoRossignoli · 2020-02-15T17:25:16Z

@nohwnd @jakubch1 I have strange failure running smoke test on CI and local with script

test.cmd -f netcoreapp2.1 -p smoke -filter coverlet
...
... .. .. .. .StdOut:
IntegrationTestBase.Execute: Starting dotnet.exe
IntegrationTestBase.Execute: Path = D:\git\vstest\artifacts\testArtifacts\dotnet\dotnet.exe
IntegrationTestBase.Execute: Arguments = test --collect:"XPlat Code Coverage" D:\git\vstest\test\TestAssets\CoverletCoverageTestProject\CoverletCoverageTestProject.csproj --diag:coverletcoverage.a41eac5e1afb450188c78916d35d293e.log
IntegrationTestBase.Execute: Total execution time: 00:00:02.2606459
IntegrationTestBase.Execute: stdError =

IntegrationTestBase.Execute: stdOut = D:\git\vstest\artifacts\testArtifacts\dotnet\sdk\3.1.100\Microsoft.TestPlatform.targets(61,5): error MSB4018: The "Microsoft.TestPlatform.Build.Tasks.VSTestLogsTask" task failed unexpectedly. [D:\git\vstest\test\TestAssets\CoverletCoverageTestProject\CoverletCoverageTestProject.csproj]
D:\git\vstest\artifacts\testArtifacts\dotnet\sdk\3.1.100\Microsoft.TestPlatform.targets(61,5): error MSB4018: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.CodeCoverage.Shim, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified. [D:\git\vstest\test\TestAssets\CoverletCoverageTestProject\CoverletCoverageTestProject.csproj]
D:\git\vstest\artifacts\testArtifacts\dotnet\sdk\3.1.100\Microsoft.TestPlatform.targets(61,5): error MSB4018: File name: 'Microsoft.VisualStudio.CodeCoverage.Shim, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' [D:\git\vstest\test\TestAssets\CoverletCoverageTestProject\CoverletCoverageTestProject.csproj]
D:\git\vstest\artifacts\testArtifacts\dotnet\sdk\3.1.100\Microsoft.TestPlatform.targets(61,5): error MSB4018:    at Microsoft.TestPlatform.Build.Tasks.VSTestLogsTask.set_LogType(String value) [D:\git\vstest\test\TestAssets\CoverletCoverageTestProject\CoverletCoverageTestProject.csproj]
D:\git\vstest\artifacts\testArtifacts\dotnet\sdk\3.1.100\Microsoft.TestPlatform.targets(61,5): error MSB4018:  [D:\git\vstest\test\TestAssets\CoverletCoverageTestProject\CoverletCoverageTestProject.csproj]    D:\git\vstest\artifacts\testArtifacts\dotnet\sdk\3.1.100\Microsoft.TestPlatform.targets(61,5): error MSB4018:  [D:\git\vstest\test\TestAssets\CoverletCoverageTestProject\CoverletCoverageTestProject.csproj]    D:\git\vstest\artifacts\testArtifacts\dotnet\sdk\3.1.100\Microsoft.TestPlatform.targets(61,56): error MSB4026: The "LogType=BuildStarted" parameter for the "Microsoft.TestPlatform.Build.Tasks.VSTestLogsTask" task is invalid. [D:\git\vstest\test\TestAssets\CoverletCoverageTestProject\CoverletCoverageTestProject.csproj]
D:\git\vstest\artifacts\testArtifacts\dotnet\sdk\3.1.100\Microsoft.TestPlatform.targets(61,5): error MSB4063: The "Microsoft.TestPlatform.Build.Tasks.VSTestLogsTask" task could not be initialized with its input parameters.  [D:\git\vstest\test\TestAssets\CoverletCoverageTestProject\CoverletCoverageTestProject.csproj]

It works inside VS and also if I run command by hand on console, but fail with test script/CI

D:\git\vstest\artifacts\testArtifacts\dotnet\dotnet.exe --info                                                                                                                                                 .NET Core SDK (reflecting any global.json):
 Version:   3.1.100
 Commit:    cd82f021f4

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18363
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   D:\git\vstest\artifacts\testArtifacts\dotnet\sdk\3.1.100\

Host (useful for support):
  Version: 3.1.0
  Commit:  65f04fb6db

.NET Core SDKs installed:
  3.1.100 [D:\git\vstest\artifacts\testArtifacts\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.0 [D:\git\vstest\artifacts\testArtifacts\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.1.0 [D:\git\vstest\artifacts\testArtifacts\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.0 [D:\git\vstest\artifacts\testArtifacts\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.0 [D:\git\vstest\artifacts\testArtifacts\dotnet\shared\Microsoft.WindowsDesktop.App]
...
D:\git\vstest\artifacts\testArtifacts\dotnet\dotnet.exe test --collect:"XPlat Code Coverage" D:\git\vstest\test\TestAssets\CoverletCoverageTestProject\CoverletCoverageTestProject.csproj --diag:coverletcoverage.a41eac5e1afb450188c78916d35d293e.log
...
Starting test execution, please wait...
Logging Vstest Diagnostics in file: D:\git\vstest\coverletcoverage.a41eac5e1afb450188c78916d35d293e.log

A total of 1 test files matched the specified pattern.

Attachments:
  D:\git\vstest\test\TestAssets\CoverletCoverageTestProject\TestResults\45cdee0f-740f-40b4-8fd2-db9f7741f4ad\coverage.cobertura.xml
Test Run Successful.
Total tests: 1
     Passed: 1
 Total time: 1,5941 Seconds

Is it expected, am I missing something?

nohwnd · 2020-02-21T08:05:54Z

test/TestAssets/CoverletCoverageTestProject/CoverletCoverageTestProject.csproj

+  </PropertyGroup>
+
+  <ItemGroup>
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.5.0" />


This should use the variables from the TestPlatform.Dependencies to keep the versions up to date. I am currently making changes to run the acceptance tests against the version that is built in the repo, not against the previous version. We will need to rebase this on top of the other change to allow this test to pass, othewise it would use a way old version of the dependencies.

Ok I'll rebase when update will be in place...or do you want that I create a new prop NETTestSdkBuildVersion and for now fix the version and you'll update with var?

That's not necessary I would have to go and remove that variable afterwards. I am fighting with the acceptance tests to pass on the server, but hopefully it won't take more than a day more to get it to pass... 🤞 (or at least I hope because it is boring and exhausting 😁 )

#2340 is now in master

Great rebasing!Any news on strange error #2330 (comment)?

test/Microsoft.TestPlatform.SmokeTests/DataCollectorTests.Coverlet.cs

nohwnd · 2020-02-21T08:09:47Z

test/Microsoft.TestPlatform.SmokeTests/DataCollectorTests.Coverlet.cs

+            this.InvokeDotnetTest($"--collect:\"XPlat Code Coverage\" {projectFileName} --diag:coverletcoverage.{logId}.log");
+            var currentDir = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
+
+            // Verify vstest.console.dll CollectArgumentProcessor fix codeBase for coverlet package


Not sure what this comment means, could you rephrase it to be more concrete, or explain to me what it means in a comment here?

With this PR #2288 we removed custom code to allow coverlet lib to load in-proc collector.
That PR added some code inside CollectArgumentProcessor where we looking for "base path" of coverlet.collector.dll expected inside nuget package default folder, passed with VSTestTestAdapterPath https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/build/netstandard1.0/coverlet.collector.targets#L21
When we find the lib we'll emit a log https://github.com/microsoft/vstest/pull/2288/files#diff-6f2506b5205b023825192043bd33a256R138
This assert check that we're sure that we've updated collector setting code base with full path, otherwise without "custom coverlet code" inside ProxyExecutionManager coverlet dll won't be resolved inside testhost.
https://github.com/microsoft/vstest/pull/2288/files#diff-3e65931fa6e723b1e2669b6ac7d6bcddL44
https://github.com/microsoft/vstest/pull/2288/files#diff-3e65931fa6e723b1e2669b6ac7d6bcddL272

nohwnd · 2020-02-21T08:15:02Z

scripts/build.ps1

@@ -228,6 +228,10 @@ function Publish-PatchedDotnet {
    Write-Log "Publish-PatchedDotnet: Copy VSTest task artifacts to local dotnet installation to allow `dotnet test` to run with it"
    $buildArtifactsPath = "$env:TP_ROOT_DIR\src\Microsoft.TestPlatform.Build\bin\$TPB_Configuration\$TPB_TargetFrameworkNS2_0\*"
    Copy-Item $buildArtifactsPath $dotnetTestArtifactsSdkPath -Force
+
+    Write-Log "Publish-PatchedDotnet: Copy vstest.console.dll artifact to local dotnet installation to allow `dotnet test` to run with it"


During my current work on the acceptance tests I found that this is falling short, because I am patching the task definition and the build but not the console and testhost. Would this be able to run via the InvokeVstest (or the other alike methods that run vsconsole but not dotnet test)? I tried taking the contents of the package that we ultimately insert into dotnet test (the CLI package), and it broke the runner, and I would rather avoid copying "random" dlls into the local copy of dotnet because then we can't really be sure about what we are testing against and if it reflects the real world.

I cannot use InvokeVsTest...because the base path of coverlet directory(nuget package) is injected through msbuild prop https://github.com/tonerdo/coverlet/blob/master/src/coverlet.collector/build/netstandard1.0/coverlet.collector.targets
I should pass /testadapterpath: with path #2278 (comment) but it depends on where we run tests and on which plat(this is a less important issue, coverlet pack at the moment has got only one tfm netstandard1.0)
Do you know a way?

Let's figure that out later, imho finding the adapter should not be that hard, there will be some tests already doing it. But I would definitely love to get the dotnet test working with the currently built taks + console + testhost, because testing dotnet test more extensively would be nice. Maybe I was just doing something wrong.

I'll take a look if InvokeVstest is used by other tests.

MarcoRossignoli · 2020-02-29T09:36:52Z

#2340 was pretty invasive and changed project guid...I don't want to lose too much time on fixing merging issue(I get some build error related to sln guids), I'll open new one with new test since this PR is low code one.

add coverlet smoke test

8d0c07d

MarcoRossignoli requested review from nohwnd and AbhitejJohn and removed request for nohwnd February 15, 2020 10:09

MarcoRossignoli requested a review from jakubch1 February 15, 2020 18:42

nohwnd reviewed Feb 21, 2020

View reviewed changes

MarcoRossignoli changed the title ~~Add coverlet smoke test~~ [WIP]Add coverlet smoke test Feb 21, 2020

Merge branch 'master' into coverletsmoketest

a32fd43

MarcoRossignoli closed this Feb 29, 2020

MarcoRossignoli deleted the coverletsmoketest branch February 29, 2020 09:37

MarcoRossignoli mentioned this pull request Mar 1, 2020

Add coverlet smoke test #2348

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP]Add coverlet smoke test #2330

[WIP]Add coverlet smoke test #2330

MarcoRossignoli commented Feb 15, 2020 •

edited

Loading

MarcoRossignoli commented Feb 15, 2020 •

edited

Loading

nohwnd Feb 21, 2020

MarcoRossignoli Feb 21, 2020

nohwnd Feb 21, 2020

MarcoRossignoli Feb 21, 2020

nohwnd Feb 24, 2020

MarcoRossignoli Feb 24, 2020 •

edited

Loading

nohwnd Feb 21, 2020

MarcoRossignoli Feb 21, 2020 •

edited

Loading

nohwnd Feb 21, 2020

MarcoRossignoli Feb 21, 2020 •

edited

Loading

nohwnd Feb 21, 2020

MarcoRossignoli Feb 21, 2020

MarcoRossignoli commented Feb 29, 2020 •

edited

Loading

[WIP]Add coverlet smoke test #2330

[WIP]Add coverlet smoke test #2330

Conversation

MarcoRossignoli commented Feb 15, 2020 • edited Loading

MarcoRossignoli commented Feb 15, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoRossignoli Feb 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoRossignoli Feb 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoRossignoli Feb 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoRossignoli commented Feb 29, 2020 • edited Loading

MarcoRossignoli commented Feb 15, 2020 •

edited

Loading

MarcoRossignoli commented Feb 15, 2020 •

edited

Loading

MarcoRossignoli Feb 24, 2020 •

edited

Loading

MarcoRossignoli Feb 21, 2020 •

edited

Loading

MarcoRossignoli Feb 21, 2020 •

edited

Loading

MarcoRossignoli commented Feb 29, 2020 •

edited

Loading