Noticeable performance penalty after "interpolated" benchmarks on MacOS

[bvdmitri]

After conversation in the Julia's helpedesk slack channel I decided to open this issue, but I'm not sure it has something to do with either Julia or BenchmarkTools.jl. Just something to keep in mind, but apparently on MacOS "interpolated" benchmarks have a serious and noticeable performance penalties for all future subsequent benchmarks.

julia> for i in 1:5
           @btime sum(a) setup=(a=zeros(512^3))
       end
  58.671 ms (0 allocations: 0 bytes)
  58.371 ms (0 allocations: 0 bytes)
  58.723 ms (0 allocations: 0 bytes)
  59.898 ms (0 allocations: 0 bytes)
  59.217 ms (0 allocations: 0 bytes)

julia> a = zeros(512^3);

julia> @btime sum($a);
  63.254 ms (0 allocations: 0 bytes)

julia> for i in 1:5
           @btime sum(a) setup=(a=zeros(512^3))
       end
  66.799 ms (0 allocations: 0 bytes)
  66.359 ms (0 allocations: 0 bytes)
  66.929 ms (0 allocations: 0 bytes)
  67.528 ms (0 allocations: 0 bytes)
  66.260 ms (0 allocations: 0 bytes)

In this example difference is not that large, but I observed performance slowdown from 58ms up to 81ms after more benchmark rounds. Just in case, I restarted my laptop and only Julia (and slack) was running. On my machine I have 16Gb of physical memory and 8-9Gb were still free. So I don't think I hit any physical memory limits.

We did not have a clear explanation of that. But probably MacOS has some limits for memory allocations. Perhaps, once some limit is reached all subsequent benchmarks are slowed down a lot. This has to do something with the fact that BenchmarkTools does not release memory of interpolated blocks. That means that if some has a lot of performed "interpolated" benchmark in their Julia session in MacOS - they will face some OS limits and all their subsequent benchmarks will be skewed.

I'm not entirely sure if BenchmarkTools can fix that. IMO it should be documented at least.

Reproducable on all version of the following Julia versions: 1.6.3, 1.6.4, 1.70-rc2 and 1.8.0-DEV.

Interesting fact, I cannot reproduce this in the official Julia docker environment on the same machine.

julia> versioninfo()
Julia Version 1.6.3
Commit ae8452a9e0 (2021-09-23 17:34 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin19.5.0)
  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.1 (ORCJIT, skylake)