Microsoft.Sbom.Targets.E2E.Tests is failing with VS 17.12

[DaveTryon]

With the update to Visual Studio 17.12, the tests in the Microsoft.Sbom.Targets.E2E.Tests project are failing in the PR build. This does not repro from inside Visual Studio, but repros when running from the command line. The error is coming from the .NET 8 version of SbomGenerationSucceedsForMultiTargetedProject. I'll stub it out for now to unblock the PR builds, but this needs to be fixed.

Here is the output from running dotnet test test\Microsoft.Sbom.Targets.E2E.Tests\Microsoft.Sbom.Targets.E2E.Tests.csproj from the root of the solution:

.>dotnet test test\Microsoft.Sbom.Targets.E2E.Tests\Microsoft.Sbom.Targets.E2E.Tests.csproj 
  Determining projects to restore...
  Restored .\src\Microsoft.Sbom.Common\Microsoft.Sbom.Common.csproj (in 230 ms).
  Restored .\src\Microsoft.Sbom.Contracts\Microsoft.Sbom.Contracts.csproj (in 230 ms).
  Restored .\src\Microsoft.Sbom.Parsers.Spdx22SbomParser\Microsoft.Sbom.Parsers.Spdx22SbomParser.csproj (in 230 ms).
  Restored .\src\Microsoft.Sbom.Extensions\Microsoft.Sbom.Extensions.csproj (in 230 ms).
  Restored .\src\Microsoft.Sbom.Adapters\Microsoft.Sbom.Adapters.csproj (in 230 ms).
  Restored .\src\Microsoft.Sbom.Api\Microsoft.Sbom.Api.csproj (in 369 ms).
  Restored .\src\Microsoft.Sbom.Extensions.DependencyInjection\Microsoft.Sbom.Extensions.DependencyInjection.csproj (in 369 ms).
  Restored .\test\Microsoft.Sbom.Targets.E2E.Tests\Microsoft.Sbom.Targets.E2E.Tests.csproj (in 371 ms).
  Restored .\src\Microsoft.Sbom.Targets\Microsoft.Sbom.Targets.csproj (in 494 ms).
  Restored .\src\Microsoft.Sbom.Tool\Microsoft.Sbom.Tool.csproj (in 500 ms).
  Microsoft.Sbom.Targets -> .\src\Microsoft.Sbom.Targets\bin\Debug\net472\Microsoft.Sbom.Targets.dll
  Microsoft.Sbom.Contracts -> .\src\Microsoft.Sbom.Contracts\bin\Debug\net8.0\Microsoft.Sbom.Contracts.dll
  Microsoft.Sbom.Extensions -> .\src\Microsoft.Sbom.Extensions\bin\Debug\net8.0\Microsoft.Sbom.Extensions.dll
  Microsoft.Sbom.Adapters -> .\src\Microsoft.Sbom.Adapters\bin\Debug\net8.0\Microsoft.Sbom.Adapters.dll
  Microsoft.Sbom.Targets.E2E.Tests -> .\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net472\Microsoft.Sbom.Targets.E2E.Tests.dll
  Building .\test\Microsoft.Sbom.Targets.E2E.Tests\..\..\src\Microsoft.Sbom.Tool\bin\Debug\net8.0
  Microsoft.Sbom.Common -> .\src\Microsoft.Sbom.Common\bin\Debug\net8.0\Microsoft.Sbom.Common.dll
  Microsoft.Sbom.Parsers.Spdx22SbomParser -> .\src\Microsoft.Sbom.Parsers.Spdx22SbomParser\bin\Debug\net8.0\Microsoft.Sbom.Parsers.Spdx22SbomParser.dll
  Microsoft.Sbom.Api -> .\src\Microsoft.Sbom.Api\bin\Debug\net8.0\Microsoft.Sbom.Api.dll
  Microsoft.Sbom.Extensions.DependencyInjection -> .\src\Microsoft.Sbom.Extensions.DependencyInjection\bin\Debug\net8.0\Microsoft.Sbom.Extensions.DependencyInjection.dll
  Microsoft.Sbom.Tool -> .\src\Microsoft.Sbom.Tool\bin\Debug\net8.0\Microsoft.Sbom.Tool.dll
  Microsoft.Sbom.Tool -> .\src\Microsoft.Sbom.Tool\bin\Debug\net8.0\publish\
  Microsoft.Sbom.Targets -> .\src\Microsoft.Sbom.Targets\bin\Debug\net8.0\Microsoft.Sbom.Targets.dll
  Microsoft.Sbom.Targets.E2E.Tests -> .\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\Microsoft.Sbom.Targets.E2E.Tests.dll
Test run for .\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\Microsoft.Sbom.Targets.E2E.Tests.dll (.NETCoreApp,Version=v8.0)
Test run for .\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net472\Microsoft.Sbom.Targets.E2E.Tests.dll (.NETFramework,Version=v4.7.2)
VSTest version 17.12.0 (x64)

VSTest version 17.12.0 (x64)

Starting test execution, please wait...
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
A total of 1 test files matched the specified pattern.
  Failed SbomGenerationSucceedsForMultiTargetedProject [260 ms]
  Error Message:
   Assert.IsTrue failed. Failed to restore the project
  Stack Trace:
     at Microsoft.Sbom.Targets.E2E.Tests.GenerateSbomE2ETests.RestoreBuildPack(Project sampleProject) in .\test\Microsoft.Sbom.Targets.E2E.Tests\GenerateSbomE2ETests.cs:line 118
   at Microsoft.Sbom.Targets.E2E.Tests.GenerateSbomE2ETests.SbomGenerationSucceedsForMultiTargetedProject() in .\test\Microsoft.Sbom.Targets.E2E.Tests\GenerateSbomE2ETests.cs:line 327
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

  Standard Output Messages:
 Build started 11/27/2024 2:39:45 PM.
 __________________________________________________
 Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (Restore target(s)):
 
 Target _FilterRestoreGraphProjectInputItems:
     __________________________________________________
     Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (_IsProjectRestoreSupported target(s)):
 
 Target _GetAllRestoreProjectPathItems:
     Determining projects to restore...
     __________________________________________________
     Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (_GenerateRestoreProjectPathWalk target(s)):
 
     Target _GenerateRestoreProjectPathItemsAllFrameworks:
         __________________________________________________
         Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (_GenerateRestoreProjectPathItemsPerFramework target(s)):
 
         __________________________________________________
         Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (_GenerateRestoreProjectPathItemsPerFramework target(s)):
 
     __________________________________________________
     Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (_IsProjectRestoreSupported target(s)):
 
 Target _GenerateRestoreGraph:
     __________________________________________________
     Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (_GenerateRestoreGraphProjectEntry target(s)):
 
     Target _GetRestoreSettingsAllFrameworks:
         __________________________________________________
         Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (_GetRestoreSettingsPerFramework target(s)):
 
         __________________________________________________
         Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (_GetRestoreSettingsPerFramework target(s)):
 
     __________________________________________________
     Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (_GenerateProjectRestoreGraph target(s)):
 
     Target GetAllRuntimeIdentifiers:
         __________________________________________________
         Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (GetAllRuntimeIdentifiers target(s)):
 
         __________________________________________________
         Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (GetAllRuntimeIdentifiers target(s)):
 
     Target _GenerateProjectRestoreGraphAllFrameworks:
         __________________________________________________
         Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (_GenerateProjectRestoreGraphPerFramework target(s)):
 
         __________________________________________________
         Project ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" is building ".\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj" (_GenerateProjectRestoreGraphPerFramework target(s)):
 
 Target Restore:
     Restoring packages for .\test\Microsoft.Sbom.Targets.E2E.Tests\bin\Debug\net8.0\ProjectSamples\ProjectSample1\ProjectSample1.csproj...
       CACHE https://api.nuget.org/v3-flatcontainer/system.runtime.compilerservices.unsafe/index.json
       CACHE https://api.nuget.org/v3-flatcontainer/system.runtime.compilerservices.unsafe/4.5.0/system.runtime.compilerservices.unsafe.4.5.0.nupkg
     C:\Program Files\dotnet\sdk\8.0.404\NuGet.targets(174,5): error : Could not load file or assembly 'System.Security.Cryptography.Pkcs, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. Could not find or load a specific file. (0x80131621)
     C:\Program Files\dotnet\sdk\8.0.404\NuGet.targets(174,5): error :   Could not load file or assembly 'System.Security.Cryptography.Pkcs, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
 Done building target "Restore" in project "ProjectSample1.csproj" -- FAILED.
 
 Done building project "ProjectSample1.csproj" -- FAILED.
 
 Build FAILED.
 
 C:\Program Files\dotnet\sdk\8.0.404\NuGet.targets(174,5): error : Could not load file or assembly 'System.Security.Cryptography.Pkcs, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. Could not find or load a specific file. (0x80131621)
 C:\Program Files\dotnet\sdk\8.0.404\NuGet.targets(174,5): error :   Could not load file or assembly 'System.Security.Cryptography.Pkcs, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
     0 Warning(s)
     1 Error(s)
 
 Time Elapsed 00:00:00.14

Failed!  - Failed:     1, Passed:     6, Skipped:     0, Total:     7, Duration: 18 s - Microsoft.Sbom.Targets.E2E.Tests.dll (net8.0)

Passed!  - Failed:     0, Passed:     7, Skipped:     0, Total:     7, Duration: 28 s - Microsoft.Sbom.Targets.E2E.Tests.dll (net472)

[sfoslund]

Spent some time investigating this and had several findings:

• I was not able to repro this locally, but it does still repro in main in the GH actions CI build
• This repros in GH actions whenever we include net472 in the target frameworks, but does not repro with only net8.0. Since we were able to build net472 projects with net 6, this seems to me to indicate an issue with the .NET version we're building with as opposed to the .NET version we're targeting in the sample project
• This issue does not repro in our ADO CI pipeline. This is not the first time we have seen different behavior between GH actions and ADO pipelines when running theoretically equivalent actions, which makes me wonder if we should switch to using ADO pipelines exclusively. This PR build is the only place across all of our projects which uses GH actions so switching to ADO pipelines were would also create consistency.
• As a side effect of this effort I noticed that we do not currently run unit tests in our CI pipeline. I created Add running unit tests to CI pipeline #835 to fix this.