Question: How do we do performance test?

[sherry-ummen]

I had a small conversation with @karelz on the Gitter Coreclr channel. Here is the chat transcript

Hi,
Just wondering how the performance test is done for the core clr

Karel Zikmund @karelz Nov 08 23:05
We have a perf lab with perf test cases. The perf lab and its results is internal only at this moment.
What exactly are you interested in?

@karelz I am more interested in how it’s done. What OS? And what parameters we are monitoring?. Knowing OS is important because in my office we have guy coming from a well know anti virus software company and he said that his company has profiled that Windows 7 is much better than Windows 10 for doing perf tests and it's stats because windows 10 has lot of background services running which hinders the stats. Its hard to believe for me. So thought of knowing how your team does and what's the rational

Karel Zikmund @karelz 06:46
@sherry-ummen our perf lab removes all noise from machines - disables all kind of background services and OS features (PreFetcher, etc.), which could interfere with our measurements and introduce noise. Key thing for performance is to focus on things which matter and remove noise/variance of the environment.
For micro-benchmarks we measure time and memory. .NET Core perf tests use xunit-performance harness (see Microsoft/xunit-performance) and we are discussing merging it with BenchmarkDotNet.
However, it does not contain all the perf lab scripts to set up the machines, etc.

Sherry Ummen @sherry-ummen 06:52
So then I assume you are using windows OS for perf test? Which OS? And do you have some scripts to disable all the background noise? Any plans to share those?

Karel Zikmund @karelz 08:55
We run perf on Linux and Mac as well. Windows is just something we run for 10+ years ...I am not sure if the infra scripts are open or not (I know that few years ago the perf test harness was way too involved to be open-sourced). It would be best to ask on GH - I can loop in perf team if you tag me (I haven't been involved in perf team work for last 2 years)

Sherry Ummen @sherry-ummen 09:07
I assume GH means Github? And which Repo in Github I should ask?
Coreclr Repo?

Karel Zikmund @karelz 09:11
coreclr or corefx will both work fine

---

So in short I am interested in the said infra scripts which are used to setup a machine. In my case a Windows machine. Whether its open sourced if not then any plans to open source it?

[karelz]

@DrewScoggins @brianrob @valenis can you please help answer the question here?
Did we publish scripts / machine setup instructions we use in perf lab?

[DrewScoggins]

Also tagging @jorive

[karelz]

@jorive @DrewScoggins @brianrob @valenis any insights?

[DrewScoggins]

Sorry, meant to write something here and got sidetracked.

So first and foremost you can find all of our performance runs at ci2.dot.net. They live under dotnet_coreclr and dotnet_corefx. All of the jobs there are handled by a perf.groovy that is checked in the respective coreclr and corefx repos. That should give you a good idea of the general shape of the infrastructure that we use to run the tests.

As for more machine setup specific steps and configurations, we run all of our Windows testing on Windows Server 2016. The scripts that we use to setup the machines lives here, https://github.com/DrewScoggins/perf-infra/blob/AddMachineSetupScript/machineSetup/setupPerfMachine.ps1. That does all of the installation of components that we need to build the products that we test as well as putting the machines in a good state for collecting performance information. Also for all of our microbenchmarks we set the priority of the running tests as high, and affinitize the process to run on one core. This helps to reduce the amount of noise that we see in the results.

Finally, we also do daily stability testing to ensure that we are not seeing machine drift over time, and that machines in the pool that we expect to be homogeneous actually are. You can find the job definition for that here, https://github.com/dotnet/perf-infra/blob/master/stability.groovy, and the supporting files are in the same repo. Sadly, the tool that we use to visualize all of this data is not yet available to the public, but we are working on getting that done.

Hope this helps!

[sherry-ummen]

Thank you very much. You say "as well as putting the machines in a good state for collecting performance information" which line in that script does that, as I could not find that.

[DrewScoggins]

In the machine setup script we set a bunch of power configurations at the beginning of the script. That is some of what we do for machine setup. Also if you look in perf.groovy files in the CoreCLR and CoreFX repo you can find the stability prefix that we use when calling the performance tests. This sets the affinity of our process to one core, as well as setting our priority to high.