[benchmark] Gather more independent samples for changes

[palimondo]

Multimodal benchmarks with significant delta between the modes can report false performance changes when we gather too few independent samples. This increases the minimal number of independent samples for potential performance changes from 5 to 10.

Resolves SR-9907.

[palimondo]

@swift-ci please benchmark

[palimondo]

@swift-ci please smoke test

[palimondo]

@eeckstein Please review 🙏

[swift-ci]

Performance: -O

TEST	OLD	NEW	DELTA	RATIO
Regression
DataCreateMediumArray	2780	3040	+9.4%	0.91x (?)

Performance: -Onone

TEST	OLD	NEW	DELTA	RATIO
Regression
DataAppendArray	5100	5700	+11.8%	0.89x (?)

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

[palimondo]

Oh man! 🤦‍♂️
Let me see how these other fake changes profile in detail...

[palimondo]

@swift-ci please benchmark

[palimondo]

DataCreateMediumArray has weakly bi-modal runtime distribution, but nowhere near the DataAppendLargeToLarge severity. I'd say this false change is more of an argument against using minimum as the typical value for a benchmark… and more evidence that somebody should have a hard look at high variance of the new Data implementation's performance even beyond SR-9911.

[swift-ci]

Performance: -O

TEST	OLD	NEW	DELTA	RATIO
Improvement
DataAppendDataSmallToSmall	4560	4040	-11.4%	1.13x (?)

Performance: -Osize

TEST	OLD	NEW	DELTA	RATIO
Improvement
SortLettersInPlace	554	513	-7.4%	1.08x (?)

Performance: -Onone

TEST	OLD	NEW	DELTA	RATIO
Improvement
DataAppendArray	5600	5200	-7.1%	1.08x (?)

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

[palimondo]

Humph… Increasing the number of independent samples seems to get rid of the false changes from DataAppendLargeToLarge, but it also exposes false changes from the rest of the DataBenchmarks… 🤷‍♂️ I've seen the same two benchmarks jump out before (hidden comment on #21848), so it's hard to say if gathering more independent samples increases the chance of false change reports... let's try again.

@swift-ci please benchmark

[swift-ci]

Performance: -Onone

TEST	OLD	NEW	DELTA	RATIO
Improvement
DataAppendArray	5600	5100	-8.9%	1.10x (?)

How to read the data

The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview

  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

[eeckstein]

I'm fine with this change.

But if we cannot fix some unstable benchmarks, we should mark them as unstable.

[palimondo]

The DataAppendArray is jumpy in -Onone because of setup overhead from the fillBuffer function. I'll fix that one in the upcoming Janitor Duty.