Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SDK templates testing in dotnet/installer #6026

Open
Tracked by #6025
donJoseLuis opened this issue Jan 31, 2023 · 1 comment
Open
Tracked by #6025

SDK templates testing in dotnet/installer #6026

donJoseLuis opened this issue Jan 31, 2023 · 1 comment
Labels
8.0 area: infra The issue is related to engineering infrastructure. Cost:M Work that requires one engineer up to 2 weeks Priority:1 Work that is critical for the release, but we could probably ship without triaged The issue was evaluated by the triage team, placed on correct area, next action defined. User Story A single user-facing feature. Can be grouped under an epic.
Milestone

Comments

@donJoseLuis
Copy link
Contributor

donJoseLuis commented Jan 31, 2023

Improving template tests in dotnet/installer

Background

One of the most common problems with .NET templates shipped within .NET SDK are the issues which defining grouping metadata: identity, group identity, precedence.
This issue is commonly reproduced only when multiple .NET SDKs are installed.
Invalid definition results in the conflict that is shown only when the template is attempted to be run and the user cannot resolve it.

During delivery of .NET 5 test templates and common templates had this issue.
During delivery of .NET 6 ASP.NET templates and WPF templates had this issue. Issue with item templates in asp.net was discovered post preview 7.

The problem is amplified because the templates are delivered by 5 teams as of now, using different approaches for authoring, infrastructure and testing their templates.
dotnet/installer seems to be the only common point for them.

Proposal

Since the dotnet/installer is the first and last point where the template packages are inserted, create the tests that can catch the issues described above and disallow releasing the build until they are solved.
The tests will be written by template engine team, the maintenance of them may be negotiated between template engine team, dotnet/installer team or template authors.

Technical details

The tests are suggested as the separate job in existing pipeline. Running tests on a single platform is enough - should the issue occur, it is identically reproduced on all the platforms.

The tests will do the following actions:

Preparation steps:

  • download and install currently supported SDK versions using dotnet-install scripts (at the moment 3.1, 6.0, 7.0). This can be done as part of the build or from C# code.
  • unpack build install on top (artifacts\packages\<configuration>\Shipping\dotnet-sdk-<version>.zip)

Test action:

  • run all the available templates for all available frameworks (in case template support framework parameter), without restore - dotnet new <template> --framework <framework> --no-restore.
  • check if template instantiated successfully, based on output or/and exit code.

There are 2 ways how to ensure the test matrix:

  • prepare test data programmatically

    • run dotnet new list to get all available templates
    • run dotnet new <template> --help to see if template supports --framework and its values
    • prepare test matrix based on discovered data
    • pros: low maintenance costs, however if list of help output changes the test might need adjustments
    • cons: the risk of bugs in logic, the risk of missing the cases, the risk that templates are not supporting all the frameworks they normally should
  • prepare test data manually

    • the test matrix to be hardcoded in code
    • when adding new template or supported TFM, the matrix has to be manually adjusted accordingly
    • to verify that the matrix is up-to-date, the sanity test will be written failing if matrix doesn't contain all the results from dotnet new list (and dotnet new <template> --help)
    • pros: eliminated risk of option 1
    • cons: possibly higher maintenance costs comparing to option 1, if list of help output changes the santiy test might need adjustments

Planned number of test cases (maximum): 40 templates x 2 languages (average) x 4 TFMs = 320 cases.
Note: it is a maximum number of tests as not all the templates support all the frameworks.

Predicted test run time: 320 cases * 2 sec = 640 sec ~ 11 minutes
Note: doesn't include build, test preparation steps, test matrix preparation.

Other possible improvements:

  • Existing template tests may benefit from matrix generation / sanity tests to ensure that all the templates are being tested.

Tasks

Justification

Engineering Impact

  1. Time savings from streamlined test execution in the installer pipelines.
@donJoseLuis donJoseLuis added triaged The issue was evaluated by the triage team, placed on correct area, next action defined. User Story A single user-facing feature. Can be grouped under an epic. 8.0 labels Jan 31, 2023
@donJoseLuis donJoseLuis added the Cost:M Work that requires one engineer up to 2 weeks label Feb 1, 2023
@vlada-shubina
Copy link
Member

vlada-shubina commented Feb 1, 2023

From #5463

Would it be possible to include concrete examples of the issues described in the Background section? Have there been cases where a template was successfully run (in terms of output and exit code) but the generated artifacts had a bug?

Concrete examples from .NET 7
dotnet/wpf#7204 (comment)
dotnet/aspnetcore#44479
dotnet/aspnetcore#44095
dotnet/test-templates#118

.NET 6:
#4296
#4299
dotnet/test-templates#118

@vlada-shubina vlada-shubina added the area: infra The issue is related to engineering infrastructure. label Feb 1, 2023
@donJoseLuis donJoseLuis added the Priority:1 Work that is critical for the release, but we could probably ship without label Feb 6, 2023
@donJoseLuis donJoseLuis added this to the 8.0RC1 milestone Feb 21, 2023
@marcpopMSFT marcpopMSFT modified the milestones: 8.0RC1, Backlog Jun 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
8.0 area: infra The issue is related to engineering infrastructure. Cost:M Work that requires one engineer up to 2 weeks Priority:1 Work that is critical for the release, but we could probably ship without triaged The issue was evaluated by the triage team, placed on correct area, next action defined. User Story A single user-facing feature. Can be grouped under an epic.
Projects
None yet
Development

No branches or pull requests

3 participants