SDK templates testing in dotnet/installer #6026
Labels
8.0
area: infra
The issue is related to engineering infrastructure.
Cost:M
Work that requires one engineer up to 2 weeks
Priority:1
Work that is critical for the release, but we could probably ship without
triaged
The issue was evaluated by the triage team, placed on correct area, next action defined.
User Story
A single user-facing feature. Can be grouped under an epic.
Milestone
Improving template tests in dotnet/installer
Background
One of the most common problems with .NET templates shipped within .NET SDK are the issues which defining grouping metadata: identity, group identity, precedence.
This issue is commonly reproduced only when multiple .NET SDKs are installed.
Invalid definition results in the conflict that is shown only when the template is attempted to be run and the user cannot resolve it.
During delivery of .NET 5 test templates and common templates had this issue.
During delivery of .NET 6 ASP.NET templates and WPF templates had this issue. Issue with item templates in asp.net was discovered post preview 7.
The problem is amplified because the templates are delivered by 5 teams as of now, using different approaches for authoring, infrastructure and testing their templates.
dotnet/installer seems to be the only common point for them.
Proposal
Since the dotnet/installer is the first and last point where the template packages are inserted, create the tests that can catch the issues described above and disallow releasing the build until they are solved.
The tests will be written by template engine team, the maintenance of them may be negotiated between template engine team, dotnet/installer team or template authors.
Technical details
The tests are suggested as the separate job in existing pipeline. Running tests on a single platform is enough - should the issue occur, it is identically reproduced on all the platforms.
The tests will do the following actions:
Preparation steps:
artifacts\packages\<configuration>\Shipping\dotnet-sdk-<version>.zip
)Test action:
dotnet new <template> --framework <framework> --no-restore
.There are 2 ways how to ensure the test matrix:
prepare test data programmatically
dotnet new list
to get all available templatesdotnet new <template> --help
to see if template supports--framework
and its valuesprepare test data manually
dotnet new list
(anddotnet new <template> --help
)Planned number of test cases (maximum): 40 templates x 2 languages (average) x 4 TFMs = 320 cases.
Note: it is a maximum number of tests as not all the templates support all the frameworks.
Predicted test run time: 320 cases * 2 sec = 640 sec ~ 11 minutes
Note: doesn't include build, test preparation steps, test matrix preparation.
Other possible improvements:
Tasks
Justification
Engineering Impact
The text was updated successfully, but these errors were encountered: