Performance changes when I use WithId

[BorisTheBrave]

I've been investigating the performance of a benchmark i've created. I've been seeing a lot of weird behaviour, but I noticed one thing that appears to be specific to Benchmark.NET.

The following code demonstrates the problem: BorisTheBrave/DeBroglie@0b0f994

It runs a single test called Benchmarks.Free, under 3 different configurations:

            BenchmarkRunner
                .Run<Benchmarks>(
                    DefaultConfig.Instance
                        .With(Job.Default.WithId("A"))
                        .With(Job.Default)
                        .With(Job.Default.WithId("Z"))
                        );

All three should logically give the same result, but instead, the unnamed one is significantly faster.

On my machine:

BenchmarkDotNet=v0.12.0, OS=Windows 10.0.18362
Intel Core i5-8400 CPU 2.80GHz (Coffee Lake), 1 CPU, 6 logical and 6 physical cores
  [Host]     : .NET Framework 4.8 (4.8.4075.0), X64 RyuJIT
  DefaultJob : .NET Framework 4.8 (4.8.4075.0), X64 RyuJIT
  A          : .NET Framework 4.8 (4.8.4075.0), X64 RyuJIT
  Z          : .NET Framework 4.8 (4.8.4075.0), X64 RyuJIT


| Method |     Job |     Mean |     Error |    StdDev |
|------- |-------- |---------:|----------:|----------:|
|   Free | Default | 5.671 ms | 0.0240 ms | 0.0213 ms |
|   Free |       A | 6.187 ms | 0.0311 ms | 0.0291 ms |
|   Free |       Z | 5.967 ms | 0.0218 ms | 0.0204 ms |

Default is always under 5.7 ms, the others around 6ms. I get similar results in .NET Core 2.1.12.

Running the jobs in separate processes separately doesn't change performance (so it's not a JIT warming thing).

Note that this behaviour is very sensitive to changes - I've found changing random unused lines of code, turning on debug symbols etc causes the Default to take 6ms instead. I presume that WithId is making some similar subtle difference to the compiled assembly. This makes it hard for me to supply you with a smaller test case.

Using any With method seems to increase the runtime - I'm just illustrating with WithId as it seems clear to me it shouldn't have any effect.

---

In conclusion
Expected behaviour: Jobs Default and A should have similar performance
Actual behaviour: Default is significantly faster
Steps to reproduce: Checkout BorisTheBrave/DeBroglie@0b0f994, build DeBrogilie.Benchmark.exe and run it.

[abelbraaksma]

I don't know if it is caused by "WithId" itself, but I did notice an oddity yesterday using separate builds using WithId and WithCustomBuildConfiguration. I think that, if you use WithId, it always ends up in separate build dirs, and sometimes this isn't fully cleaned up or something.

For instance, I had this run ("After" is supposed to be faster here):

Then, when I ran each job individually, it showed that "After" was indeed faster. Then I cleaned the bin and tried again with the original config, and lo and behold:

I have not yet managed to make this behavior repeatable, though.

[adamsitnik]

Hi @BorisTheBrave

From my experience (I am the person who was working on ensuring that the last few .NET Core releases have not regressed performance) such differences between benchmark runs are typically caused by memory and|or loop body alignment changes (which are quite random and can be caused by adding some additional source code).

I am currently working on a Workshop for the .NET Team and preparing a doc that provides some examples and describes how to deal with that:

https://github.com/adamsitnik/performance/blob/e4987916c83bef9c2203c256043dc4d9f6cc6a29/docs/regressions.md#memory-alignment
https://github.com/adamsitnik/performance/blob/e4987916c83bef9c2203c256043dc4d9f6cc6a29/docs/regressions.md#code-alignment

I hope that this doc answers your question and provides you some way to move on from here (get the disassembly and compare alignment etc). Since there is nothing actionable on the BDN side and we already have two issues that track the alignment ideas (#756 and #1513) I am closing this one.

The WithCustomBuildConfiguration problem mentioned by @abelbraaksma was solved in #1494 and was caused by a lack of optimizations enabled.

Thanks,
Adam

[BorisTheBrave]

Thanks, I think #1494 was what I was experiencing. I'll look into alignment issues, but I agree that is not a BDN problem.