Skip to content

Commit

Permalink
Make host tests use single copy of shared framework (#100588)
Browse files Browse the repository at this point in the history
- Only create one .NET install layout to be shared by all host tests
-  Add `pretest.proj` for `host.pretest` subset that builds all test project assets and creates the single .NET install layout
- Fix `NativeHostApis` tests that were editing the .NET install layout directly (instead of creating a copy to edit)
  - Remove some unnecessary copying/creating of SDKs and frameworks by sharing the fixture across tests
- Update host testing doc with simpler setup instructions and more details around investigating test failures
  • Loading branch information
elinor-fung authored Apr 4, 2024
1 parent 7f2be68 commit 960849b
Show file tree
Hide file tree
Showing 7 changed files with 167 additions and 146 deletions.
21 changes: 15 additions & 6 deletions docs/workflow/testing/host/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,16 +37,16 @@ dotnet build src\installer\tests\HostActivation.Tests

The host tests depend on:
1. Pre-built [test project](/src/installer/tests/Assets/Projects) output which will be copied and run by the tests. The `host.pretest` subset builds these projects.
2. Product binaries in a directory layout matching that of a .NET install
3. TestContextVariables.txt file with property and value pairs which will be read by the tests
2. Product binaries in a directory layout matching that of a .NET install. The `host.pretest` subset creates this layout.
3. TestContextVariables.txt files with property and value pairs which will be read by the tests. The `host.tests` subset creates these files as part of building the tests.

When [running all tests](#running-all-tests), the build is configured such that these are created/performed before the start of the test run.

In order to create (or update) these dependencies without running all tests:
1. Build the `host.pretest` subset. By default, this is included in the `host` subset. This corresponds to (1) above.
2. Run the `SetUpSharedFrameworkPublish` and `SetupTestContextVariables` targets for the desired test project. This corresponds to (2) and (3) above. For example:
1. Build the `host.pretest` subset. By default, this is included in the `host` subset. This corresponds to (1) and (2) above.
2. Build the desired test project. This corresponds to (3) above. Building the test itself will run the `SetupTestContextVariables` target, but it can also be run independently - for example:
```
dotnet build src\installer\tests\HostActivation.Tests -t:SetUpSharedFrameworkPublish;SetupTestContextVariables -p:RuntimeConfiguration=Release -p:LibrariesConfiguration=Release
dotnet build src\installer\tests\HostActivation.Tests -t:SetupTestContextVariables -p:RuntimeConfiguration=Release -p:LibrariesConfiguration=Release
```

## Running tests
Expand Down Expand Up @@ -86,6 +86,15 @@ If you built the runtime or libraries with a different configuration from the ho
build.cmd -vs Microsoft.DotNet.CoreSetup -rc Release -lc Release
```

## Investigating failures

When [running all tests](#running-all-tests), reports with results will be generated under `<repo_root>\artifacts\TestResults`. When [running individual tests](#running-specific-tests), results will be output to the console by default and can be configured via [`dotnet test` options](https://learn.microsoft.com/dotnet/core/tools/dotnet-test#options).

In order to test the hosting components, the tests launch a separate process (e.g. `dotnet`, apphost, native host) and validate the expected output (standard output and error) of the launched process. This usually involves copying or creating test artifacts in the form of an application to run or a .NET install to run against.

On failure, tests will report the file, arguments, and environment for the launched process that failed validation. With [preserved test artifacts](#preserving-test-artifacts), this information can be used to directly debug the specific scenario that the test was running.

### Preserving test artifacts

In order to test the hosting components, the tests launch a separate process (e.g. `dotnet`, apphost, native host) and validate the expected output (standard output and error) of the launched process. This usually involves copying or creating test artifacts in the form of an application to run or a .NET install to run against. The tests will delete these artifacts after the test finishes. To allow inspection or usage after the test finishes, set the environment variable `PRESERVE_TEST_RUNS=1` to avoid deleting the test artifacts.
The tests will delete any generated test artifacts after the test finishes. To allow inspection or usage after the test finishes, set the environment variable `PRESERVE_TEST_RUNS=1` to avoid deleting the test artifacts.

2 changes: 1 addition & 1 deletion eng/Subsets.props
Original file line number Diff line number Diff line change
Expand Up @@ -500,7 +500,7 @@

<!-- Host.pretest subset (consumes live built libraries assets so needs to come after libraries) -->
<ItemGroup Condition="$(_subset.Contains('+host.pretest+'))">
<ProjectToBuild Include="$(InstallerProjectRoot)tests\Assets\Projects\**\*.csproj" Category="host" />
<ProjectToBuild Include="$(InstallerProjectRoot)tests\pretest.proj" Category="host" />
</ItemGroup>

<!-- Host.tests subset (consumes live built libraries assets so needs to come after libraries) -->
Expand Down
20 changes: 0 additions & 20 deletions src/installer/tests/Directory.Build.targets
Original file line number Diff line number Diff line change
@@ -1,25 +1,5 @@
<Project>

<Target Name="SetUpSharedFrameworkPublish"
DependsOnTargets="DetermineTestOutputDirectory"
BeforeTargets="RunTests">

<!--
Explicitly restore before PublishToDisk. IsRestoring property ensures that this evaluation won't be reused.
See https://github.com/dotnet/msbuild/issues/2811
-->
<MSBuild Projects="$(InstallerProjectRoot)pkg\sfx\Microsoft.NETCore.App\Microsoft.NETCore.App.Runtime.sfxproj"
Targets="Restore"
Properties="IsRestoring=true" />
<!--
Set up the shared framework copy this set of tests should use. There's no known reason to have
one per test project, but RepoDirectoriesProvider may need some tweaking to share.
-->
<MSBuild Projects="$(InstallerProjectRoot)pkg\sfx\bundle\Microsoft.NETCore.App.Bundle.bundleproj"
Targets="PublishToDisk"
Properties="OutputPath=$(TestsOutputDir)sharedFrameworkPublish/" />
</Target>

<Target Name="SetupTestContextVariables"
Condition="'$(IsTestProject)' == 'true'"
DependsOnTargets="
Expand Down
Loading

0 comments on commit 960849b

Please sign in to comment.