-
-
Notifications
You must be signed in to change notification settings - Fork 963
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Inaccurate results reported for small methods #1802
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Unless doing instruction level profiling, the highest precision timer on a modern computer is about
25-32 cycles
(which is, under ideal circumstances, about6ns
on a 5GHz processor at best and32ns
on a 1GHz processor).Due to platform specific differences, the maximum reported difference in the high precision timer APIs exposed by the OS is about
100ns
. Additionally, it is well documented that due to the latency between calls and other factors on the OS or hardware, the latency for such a call can be much worse, such as closer to300ns
when a CPU level timer such asRDTSC
is not available: https://docs.microsoft.com/en-us/windows/win32/sysinfo/acquiring-high-resolution-time-stamps#resolution-precision-accuracy-and-stability.While Benchmark.NET does try to account for small methods and while it also tries to account for noise due to call overhead and the like, there are many cases where the numbers it reports are of questionable accuracy.
One such example is the following:
In particular, if we look at the first entry
GetShortName_opt
is reporting a time of0.2082 ns
. Even in an "ideal" scenario where the JIT is able to fully optimize the comparison against a constant value and optimize it to simple bexor rax, rax
, this is still reporting that it takes approximately 1 cycle on a 5GHz CPU.It would be beneficial, IMO, if Benchmark.NET was more proactive about labeling potentially problematic results and had guidance on how to optimally write a test in a way that will provide accurate results.
The text was updated successfully, but these errors were encountered: