Initial v0.11.5 changelog

docs/_changelog/header/v0.11.5.md

@@ -0,0 +1,170 @@
+## Highlights
+
+* **Features and noticeable improvements**
+  * **Power plan management**  
+    Now BenchmarkDotNet executes all benchmarks with enabled High-Performance power plan (configurable, Windows-only).
+    You can find some details below.
+    [#68](https://github.com/dotnet/BenchmarkDotNet/issues/68)
+    [#952](https://github.com/dotnet/BenchmarkDotNet/pull/952)
+  * **Better Environment Variables API**  
+    Now we have some additional extension methods which allow defining environment variables in user jobs.
+    In the previous version, users always had to set an array of environment variables like this:
+      `job.With(new[] { new EnvironmentVariable("a", "b") })`.
+    Now it's possible to define an environment variable like
+      `job.With(new EnvironmentVariable("a", "b"))` or
+      `job.WithEnvironmentVariable("a", "b")`.
+    Also, it's possible to clear the list of environment variables via
+      `job.WithoutEnvironmentVariables()`.
+    [#1069](https://github.com/dotnet/BenchmarkDotNet/issues/1069)
+    [#1080](https://github.com/dotnet/BenchmarkDotNet/pull/1080)
+  * **Better outlier messages**  
+    The previous version of BenchmarkDotNet printed information about detected or removed outliers like this:
+      "3 outliers were detected".
+    It was nice, but it didn't provide additional information about these outliers
+      (users had to read the full log to find the outliers values).
+    Now BenchmarkDotNet prints additional information about outlier values like this:
+       "3 outliers were detected (2.50 us..2.70 us)".
+    [e983cd31](https://github.com/dotnet/BenchmarkDotNet/commit/e983cd3126e64f82fe59bc1bc45d1a870a615e87)
+  * **Support modern CPU architecture names**  
+    In the environment information section, BenchmarkDotNet prints not only the processor brand string, but also its architecture
+      (e.g., "Intel Core i7-4770K CPU 3.50GHz (Haswell)").
+    However, it failed to recognize some recent processors.
+    Now it's able to detect the architecture for modern Intel processors correctly
+      (Kaby Lake, Kaby Lake R, Kaby Lake G, Amber Lake Y, Coffee Lake, Cannon Lake, Whiskey Lake).
+    [995e053d](https://github.com/dotnet/BenchmarkDotNet/commit/995e053d14a61cdadc417149480f23ebf679bcb7)
+  * **Introduce BenchmarkDotNet.Annotations**  
+    Currently, BenchmarkDotNet targets .NET Standard 2.0.
+    It makes some users unhappy because they want to define benchmarks in projects with lower target framework.
+    We decided to start working on the `BenchmarkDotNet.Annotations` NuGet package which targets .NET Standard 1.0
+      and contains classes that users need to define their benchmarks.
+    However, it's not easy to refactor the full source code base and move all relevant public APIs to this package.
+    In v0.11.5, we did the first step and moved some of these APIs to `BenchmarkDotNet.Annotations`.
+    We want to continue moving classes to this package and get full-featured annotation package in the future.
+    [#1084](https://github.com/dotnet/BenchmarkDotNet/pull/1084)
+    [#1096](https://github.com/dotnet/BenchmarkDotNet/pull/1096)
+  * **Use InProcessEmitToolchain by default in InProcess benchmarks**  
+    In BenchmarkDotNet 0.11.4, we introduced `InProcessEmitToolchain`.
+    It's a new, full-featured InProcess toolchain which allows executing benchmarks in the current process
+        without spawning additional process per benchmark.
+    It supports `[Arguments]`, `[ArgumentsSource]`, passing the arguments by `out`, `ref` and returning stack-only types like `Span<T>`.
+    However, in v0.11.4, it can be activated only if `InProcessEmitToolchain` is declared explicitly.
+    Now it's enabled by default when `[InProcessAttribute]` is used.
+    [#1093](https://github.com/dotnet/BenchmarkDotNet/pull/1093)
+  * **Introduce an option which prevents overwriting results**  
+    Currently, BenchmarkDotNet overwrites results each time when the benchmarks are executed.
+    It allows avoiding tons of obsolete files in the `BenchmarkDotNet.Artifacts` folder.
+    However, the behavior doesn't fit all use cases:
+      sometimes users want to keep results for old benchmark runs.
+    Now we have a special option for it.
+    The option can be activated via `--noOverwrite` console line argument or
+      `DontOverwriteResults` extension method for `IConfig`
+    [#1074](https://github.com/dotnet/BenchmarkDotNet/issues/1074)
+    [#1083](https://github.com/dotnet/BenchmarkDotNet/pull/1083)
+* **Other improvements and bug fixes**
+  * **Diagnostics and validation**
+    * **Better benchmark declaration error processing**  
+      In the previous version, BenchmarkDotNet threw an exception when some benchmark methods had an invalid declaration
+        (e.g., invalid signature or invalid access modifiers).
+      Now it prints a nice error message without ugly stack traces.
+      [#1107](https://github.com/dotnet/BenchmarkDotNet/issues/1107)
+    * **Better error message for users who want to debug benchmarks**  
+      [#1073](https://github.com/dotnet/BenchmarkDotNet/issues/1073)
+    * **Don't show the same validation error multiple times**  
+      Now each error will be printed only once.
+      [#1079](https://github.com/dotnet/BenchmarkDotNet/issues/1079)
+    * **Restrict MemoryDiagnoserAttribute usage to class**  
+      Now it's impossible to accidentally mark a method with this attribute.
+      [#1119](https://github.com/dotnet/BenchmarkDotNet/issues/1119)
+      [#1122](https://github.com/dotnet/BenchmarkDotNet/pull/1122)
+  * **Export**
+    * **Better indentation in disassembly listings**  
+      Now DissassemblyDiagnoser correctly process source code which contains tab as the indentation symbol
+      [#1110](https://github.com/dotnet/BenchmarkDotNet/issues/1110)
+    * **Fix incorrect indentation for StackOverflow exporter**  
+      Previously, StackOverflow exporter doesn't have a proper indent for job runtimes in the environment information.
+      Now it's fixed.
+      [#826](https://github.com/dotnet/BenchmarkDotNet/issues/826)
+      [#1104](https://github.com/dotnet/BenchmarkDotNet/pull/1104)
+    * **Fix StackOverflowException in XmlExporter.Full**  
+      [#1086](https://github.com/dotnet/BenchmarkDotNet/issues/1086)
+      [#1090](https://github.com/dotnet/BenchmarkDotNet/pull/1090)
+    * **Shortify MemoryDiagnoser column titles**  
+      Now we use the following column titles:
+        "Allocated" instead of "Allocated Memory/Op",
+        "Gen 0" instead of "Gen 0/1k Op".
+      The full description of each column can be found in the legend section below the summary table.
+      [#1081](https://github.com/dotnet/BenchmarkDotNet/pull/1081)
+  * **Benchmark generation and execution**
+    * **Fixed broken Orderers**  
+      The previous version has a nasty bug with custom orderers.
+      Now it's fixed.
+      [#1070](https://github.com/dotnet/BenchmarkDotNet/issues/1070)
+      [#1109](https://github.com/dotnet/BenchmarkDotNet/issues/1109)
+    * **Better overhead evaluation**  
+      In the previous version, BenchmarkDotNet evaluated the benchmark overhead as a mean value of all overhead iteration.
+      It was fine in most cases, but in some cases, the mean value can be spoiled by outliers.
+      Now BenchmarkDotNet uses the median value.
+      [#1116](https://github.com/dotnet/BenchmarkDotNet/issues/1116)
+    * **Respect CopyLocalLockFileAssemblies**  
+      Now BenchmarkDotNet respect the `CopyLocalLockFileAssemblies` value and copies it
+        to the generated benchmark project.
+      [#1068](https://github.com/dotnet/BenchmarkDotNet/issues/1068)
+      [#1108](https://github.com/dotnet/BenchmarkDotNet/pull/1108)
+    * **Disable CodeAnalysisRuleSet for generated benchmarks**  
+      Previously, generated benchmarks may fail if the `CodeAnalysisRuleSet` is defined in `Directory.Build.Props`.
+      [#1082](https://github.com/dotnet/BenchmarkDotNet/pull/1082)
+    * **Supported undefined enum values**  
+      [#1020](https://github.com/dotnet/BenchmarkDotNet/issues/1020)
+      [#1071](https://github.com/dotnet/BenchmarkDotNet/issues/1071)
+  * **Other minor improvements and bug fixes**
+
+## Power plans
+
+In [#952](https://github.com/dotnet/BenchmarkDotNet/pull/952), power plan management was implemented.
+It resolves a pretty old issue [#68](https://github.com/dotnet/BenchmarkDotNet/issues/68) which was created more than three years ago.
+Now BenchmarkDotNet forces OS to execute a benchmark on the High-Performance power plan.
+You can disable this feature by modifying PowerPlanMode property.
+Here is an example where we are playing with this value:
+
+```cs
+[Config(typeof(Config))]
+public class IntroPowerPlan
+{
+    private class Config : ManualConfig
+    {
+        public Config()
+        {
+            Add(Job.MediumRun.WithPowerPlan(PowerPlan.HighPerformance));
+            Add(Job.MediumRun.WithPowerPlan(PowerPlan.UserPowerPlan));
+        }
+    }
+
+    [Benchmark]
+    public int IterationTest()
+    {
+        int j = 0;
+        for (int i = 0; i < short.MaxValue; ++i)
+            j = i;
+        return j;
+    }
+
+    [Benchmark]
+    public int SplitJoin()
+        => string.Join(",", new string[1000]).Split(',').Length;
+}
+```
+
+And here is an example of the summary table on plugged-off laptop:
+
+```md
+        Method |       PowerPlan |     Mean |     Error |    StdDev |
+-------------- |---------------- |---------:|----------:|----------:|
+ IterationTest | HighPerformance | 40.80 us | 0.4168 us | 0.6109 us |
+     SplitJoin | HighPerformance | 13.24 us | 0.2514 us | 0.3763 us |
+ IterationTest |   UserPowerPlan | 79.72 us | 2.5623 us | 3.8352 us |
+     SplitJoin |   UserPowerPlan | 24.54 us | 2.1062 us | 3.1525 us |
+```
+
+As you can see, the power plan produces a noticeable effect on the benchmark results.
+
+This feature is available on Windows only.