Add "dotnet new" benchmark project template

[CodeTherapist]

The template supports the following parameters:

Usage example:

dotnet new benchmark -n=MyCompany.Namespace.Benchmarks -b=MyBechmark -f="netstandard2.0;net472"

The template is a library, not as proposed a exe - I suggest that users should take advantage of the dotnet global tool to run the benchmarks, instead of creating a console app.
The config parameter will conditionally add a benchmark config class and reference it inside the the benchmark class.

• Initial release
• Documentation

The documentation will be delivered as soon the template is approved.

Closes #1028

[CatGirlsAreLife]

Nice, can I test this template locally on Windows 10 build 18317 x64?

[CodeTherapist]

@CatGirlsAreLife Yes, you could test this on windows. To do so install the template from the directory location like this:

dotnet new -i [path-to-the-nuspec-directory-of-the-template]

After that, you can use it as I described already.

[Therzok]

@adamsitnik @AndreyAkinshin is there interest in publishing this?

@CodeTherapist mind if I add some commits on top of yours in another PR?

[CodeTherapist]

@Therzok Thanks for helping me out, very appreciated.
@dzmitry-lahoda and @TheAngryByrd Thanks for the F# template, used it as base line - included it in my last commit. How you like it?

Added a VB.NET and a F# template as well. Even I'm primarily using C# to get cool things done - I decided to provide them with the exact same capability and behavior as the C# template.

After installation:

Building them as is:

Create the template nuget packages (using nuget 5.x)

nuget pack BenchmarkDotNet.BenchmarkProjectTemplate.CSharp.nuspec
nuget pack BenchmarkDotNet.BenchmarkProjectTemplate.VB.nuspec
nuget pack BenchmarkDotNet.BenchmarkProjectTemplate.FSharp.nuspec

Installing the template nuget packages

dotnet new -i BenchmarkDotNet.BenchmarkProjectTemplate.CSharp
dotnet new -i BenchmarkDotNet.BenchmarkProjectTemplate.VB
dotnet new -i BenchmarkDotNet.BenchmarkProjectTemplate.FSharp

Using the template

dotnet new benchmark -n=MyCompany.Namespace.CSharp.Benchmarks -b=MyBechmark -c=true -f="netstandard2.0;net472" -lang C#

Accordingly, you can use -lang F# or -lang VB.

[TheAngryByrd]

I had some issues with the F# template. I sent a PR to your branch to fix them CodeTherapist#1

[AndreyAkinshin]

@CodeTherapist thanks for the pull request and sorry for such a late reply. I was pretty busy during the last several months and didn't have enough time for BenchmarkDotNet.

In general, it looks and works great! It would be awesome to have such templates in the terminal. However, I have several concerns:

1. Currently, we have three different NuGet packages with templates which should be installed independently. It will be better to have only one NuGet package to them all. You can use the NUnit 3 Test Project Template as an example.
2. It's better to use the current version of BenchmarkDotNet in the csproj file instead of 0.1*. We didn't release the 1.0 version, which allows us to make some breaking changes from time to time (of course, we prefer to make backward compatible releases when it's possible). So, I don't really like the floating NuGet package reference versions because a new release of BenchmarkDotNet can spoil the existing solutions.
3. Other templates (like console, classlib, xunit, mvc) use the --no-restore option for skipping restore. It would be nice to use the same name to be consistent.
4. It will be better to reference BenchmarkDotNet.Diagnostics.Windows only if a benchmark is created on Windows.
5. I prefer to use the current BenchmarkDotNet version as a template version (in the nuspec file). It will simplify syncing between templates and new versions of the library.
6. Since we don't have proper support of BenchmarkDotNet projects in IDEs (VS, Rider) as we have for unit tests, it will be better to create a netcoreapp project by default with the following Main method:

class Program
{
    static void Main(string[] args)
    {
        BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
    }
}

It will help developers to easily run benchmarks from an IDE during the benchmark design stage.

[CodeTherapist]

thanks for the pull request and sorry for such a late reply. I was pretty busy during the last several months and didn't have enough time for BenchmarkDotNet.

No issue, we are all usually busy 👍 Thanks for your answers and your time, reviewing my PR.

I agree on all your statements except for number six: I think we should encourage the users to use the awesome dotnet tool (#213) to run the benchmarks quickly - I will add it as optional option.
Beside that I think a first-class integration into the most used IDE's (VS, Rider and VS Code) would make sense and would increase acceptance to use benchmark dotnet.

[AndreyAkinshin]

I agree on all your statements except for number six: I think we should encourage the users to use the awesome dotnet tool

@adamsitnik what do you think?

Beside that I think a first-class integration into the most used IDE's (VS, Rider and VS Code) would make sense and would increase acceptance to use benchmark dotnet.

Yeah, it will be great. I already working on the advanced BenchmarkDotNet support in Rider.

[CodeTherapist]

• 1. Currently, we have three different NuGet packages with templates which should be installed independently. It will be better to have only one NuGet package to them all.

• 2. It's better to use the current version of BenchmarkDotNet in the csproj file instead of 0.1*.

• 3. Other templates (like console, classlib, xunit, mvc) use the --no-restore option for skipping restore. It would be nice to use the same name to be consistent.

• 4. It will be better to reference BenchmarkDotNet.Diagnostics.Windows only if a benchmark is created on Windows.

• 5. I prefer to use the current BenchmarkDotNet version as a template version (in the nuspec file). It will simplify syncing between templates and new versions of the library.

• 6. Since we don't have proper support of BenchmarkDotNet projects in IDEs (VS, Rider) as we have for unit tests, it will be better to create a netcoreapp project by default with the following Main method [...]

@AndreyAkinshin Thanks for your review - everything done, except number 6.
Added a basic documentation for the templates as well.

[AndreyAkinshin]

except number 6

What do you think about "netcoreapp with Main method" as an option? (We can keep the existing behaviour with netstandard/classlib as the default)

[CodeTherapist]

except number 6

What do you think about "netcoreapp with Main method" as an option? (We can keep the existing behaviour with netstandard/classlib as the default)

Sure! Also thought about that and mentioned this few comments earlier as well.
How about --consoleapp as options name?

[AndreyAkinshin]

How about --consoleapp as options name?

LGTM.

Could you please also merge master into your branch and resolve the conflicts?

[AndreyAkinshin]

@CodeTherapist in general, it works great! I will merge the PR as soon as you introduce the --consoleapp option.

[CodeTherapist]

@AndreyAkinshin: The --console-app is implemented, docs are updated and conflicts are resolved.

The templates are not tested at the moment during the build. I think we should do this to ensure further changes do not break the templates. What do you think?

[AndreyAkinshin]

@CodeTherapist everything works great! Thank you very much for your contribution!

[ulex]

@CodeTherapist first of all, thanks for the template, very appreciated!
let me share the experience of using it. I've found this this issue when I was wondering why the default isn't a console application. The first place where I found the llink to the global tool was the current issue. It is nice, that there is a console tool for running benchmark, but there is no way to figure about it from documentation. I checked these places:

1. Getting Started shows example with command-line application
2. How to run your benchmarks explictly states, that

   There are several ways to run your benchmarks. What is important is that BenchmarkDotNet works only with Console Apps.

3. Even the page about templates didn't mention global tool at all

IMO it seems like either more promotion of global tools is required, or default value should be changed.

[GF-Huang]

After install the template, I create a BenchmarkDotNet project in VS2022, it creates a solution with an empty project folder, without any project files.

references:

• https://benchmarkdotnet.org/articles/guides/dotnet-new-templates.html

[GF-Huang]

I try to create project by command line dotnet new benchmark, it creates a library project, it's crazy to the docs!!!