Skip to content

[benchmark] Janitor Duty: Troublemaker Legacy #22508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Feb 11, 2019
Merged

[benchmark] Janitor Duty: Troublemaker Legacy #22508

merged 8 commits into from
Feb 11, 2019

Conversation

palimondo
Copy link
Contributor

This PR follows-up #20861, #21413, #21516, #21794, #22026 and #22296 in clean-up efforts to enable robust performance measurements by adjusting workloads to run in reasonable time (< 1000 μs), minimizing the accumulated error. To maintain long-term performance tracking, it applies legacy factor where necessary.

This one picks up on troublemakers skipped from #22026 and re-adds a few previously disabled benchmarks. Remember, there never were truly unstable benchmarks, it was our measurement methodology that was flawed before.

@palimondo
Copy link
Contributor Author

@swift-ci Please benchmark

@swift-ci

This comment has been minimized.

Sign in to view
@palimondo
Copy link
Contributor Author

@swift-ci please benchmark

@palimondo
Copy link
Contributor Author

@swift-ci please smoke test

@palimondo palimondo changed the title [WIP][benchmark] Troublemaker Legacy [benchmark] Troublemaker Legacy Feb 11, 2019
@palimondo palimondo changed the title [benchmark] Troublemaker Legacy [benchmark] Janitor Duty: Troublemaker Legacy Feb 11, 2019
@swift-ci
Copy link
Contributor

Performance: -O

TEST OLD NEW DELTA RATIO
Improvement
DataAppendDataLargeToLarge 50400 37600 -25.4% 1.34x (?)
Added
Fibonacci 145 147 146 —
ObjectiveCBridgeFromNSDictionaryAnyObjectForced 9550 9800 9650 —
ObjectiveCBridgeFromNSDictionaryAnyObjectToString 75500 77500 76167 —
ObjectiveCBridgeFromNSDictionaryAnyObjectToStringForced 83500 85500 84167 —
ObjectiveCBridgeFromNSSetAnyObjectToStringForced 111500 115500 112833 —
ObjectiveCBridgeStubDateAccess 228 230 229 —
ObjectiveCBridgeStubFromNSDate 7050 7070 7060 —
ObjectiveCBridgeStubFromNSDateRef 4190 4200 4193 —
ObjectiveCBridgeStubFromNSStringRef 170 172 171 —
ObjectiveCBridgeStubNSDateMutationRef 1684 1684 1684 —
ObjectiveCBridgeStubNSDateRefAccess 371 423 406 —
ObjectiveCBridgeStubToNSDateRef 4220 4320 4280 —
ProtocolDispatch 314 314 314 —
ProtocolDispatch2 357 357 357 —

Code size: -O

TEST OLD NEW DELTA RATIO
Regression
ObjectiveCNoBridgingStubs.o 7972 8868 +11.2% 0.90x
ProtocolDispatch2.o 1888 1920 +1.7% 0.98x

Performance: -Osize

TEST OLD NEW DELTA RATIO
Regression
ObjectiveCBridgeStubToNSStringRef 112 124 +10.7% 0.90x (?)
Added
Fibonacci 159 161 160 —
ObjectiveCBridgeFromNSDictionaryAnyObjectForced 9000 9200 9083 —
ObjectiveCBridgeFromNSDictionaryAnyObjectToString 76000 77500 76500 —
ObjectiveCBridgeFromNSDictionaryAnyObjectToStringForced 84500 86500 85333 —
ObjectiveCBridgeFromNSSetAnyObjectToStringForced 156000 189500 176333 —
ObjectiveCBridgeStubDateAccess 238 238 238 —
ObjectiveCBridgeStubFromNSDate 8850 9330 9017 —
ObjectiveCBridgeStubFromNSDateRef 5300 5380 5353 —
ObjectiveCBridgeStubFromNSStringRef 166 172 169 —
ObjectiveCBridgeStubNSDateMutationRef 1728 1736 1732 —
ObjectiveCBridgeStubNSDateRefAccess 351 403 369 —
ObjectiveCBridgeStubToNSDateRef 4240 4520 4333 —
ProtocolDispatch 342 342 342 —
ProtocolDispatch2 414 414 414 —

Code size: -Osize

TEST OLD NEW DELTA RATIO
Regression
ObjectiveCNoBridgingStubs.o 7743 8511 +9.9% 0.91x
Improvement
ObjectiveCBridgingStubs.o 18508 17892 -3.3% 1.03x
ObjectiveCBridging.o 48595 48067 -1.1% 1.01x

Performance: -Onone

TEST OLD NEW DELTA RATIO
Regression
Radix2CooleyTukeyf 34254 42480 +24.0% 0.81x
Radix2CooleyTukey 40491 48432 +19.6% 0.84x
ObjectiveCBridgeFromNSSetAnyObjectToString 79824 88500 +10.9% 0.90x (?)
Improvement
CharIndexing_russian_unicodeScalars_Backwards 590000 548520 -7.0% 1.08x (?)
Added
Fibonacci 705 705 705 —
ObjectiveCBridgeFromNSDictionaryAnyObjectForced 11300 11700 11517 —
ObjectiveCBridgeFromNSDictionaryAnyObjectToString 80000 81500 80667 —
ObjectiveCBridgeFromNSDictionaryAnyObjectToStringForced 84000 85500 84500 —
ObjectiveCBridgeFromNSSetAnyObjectToStringForced 107500 110000 108333 —
ObjectiveCBridgeStubDateAccess 6396 6505 6433 —
ObjectiveCBridgeStubFromNSDate 7650 7720 7680 —
ObjectiveCBridgeStubFromNSDateRef 5650 5660 5653 —
ObjectiveCBridgeStubFromNSStringRef 333 334 333 —
ObjectiveCBridgeStubNSDateMutationRef 1792 1804 1799 —
ObjectiveCBridgeStubNSDateRefAccess 6048 6113 6071 —
ObjectiveCBridgeStubToNSDateRef 5040 5180 5113 —
ProtocolDispatch 914 933 920 —
ProtocolDispatch2 4703 4773 4730 —
✅ Benchmark Check Report
⚠️🔤 ObjectiveCBridgeFromNSDictionaryAnyObjectToStringForced name is composed of 9 words.
Split ObjectiveCBridgeFromNSDictionaryAnyObjectToStringForced name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⛔️🔤 ObjectiveCBridgeFromNSDictionaryAnyObjectToStringForced name is 55 characters long.
Benchmark name should not be longer than 40 characters.
⚠️🔤 ObjectiveCBridgeStubFromNSStringRef name is composed of 6 words.
Split ObjectiveCBridgeStubFromNSStringRef name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⚠️Ⓜ️ ObjectiveCBridgeStubFromNSStringRef has very wide range of memory used between independent, repeated measurements.
ObjectiveCBridgeStubFromNSStringRef mem_pages [i1, i2]: min=[227, 232] 𝚫=5 R=[23, 20]
⚠️🔤 ObjectiveCBridgeStubNSDateMutationRef name is composed of 6 words.
Split ObjectiveCBridgeStubNSDateMutationRef name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⚠️🔤 ObjectiveCBridgeStubFromNSDateRef name is composed of 6 words.
Split ObjectiveCBridgeStubFromNSDateRef name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⚠️Ⓜ️ ObjectiveCBridgeStubFromNSDateRef has very wide range of memory used between independent, repeated measurements.
ObjectiveCBridgeStubFromNSDateRef mem_pages [i1, i2]: min=[227, 229] 𝚫=2 R=[18, 16]
⚠️🔤 ObjectiveCBridgeStubNSDateRefAccess name is composed of 6 words.
Split ObjectiveCBridgeStubNSDateRefAccess name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⚠️🔤 ObjectiveCBridgeFromNSSetAnyObjectToStringForced name is composed of 9 words.
Split ObjectiveCBridgeFromNSSetAnyObjectToStringForced name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⛔️🔤 ObjectiveCBridgeFromNSSetAnyObjectToStringForced name is 48 characters long.
Benchmark name should not be longer than 40 characters.
⚠️🔤 ObjectiveCBridgeStubDateAccess name is composed of 5 words.
Split ObjectiveCBridgeStubDateAccess name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⚠️🔤 ObjectiveCBridgeStubFromNSDate name is composed of 5 words.
Split ObjectiveCBridgeStubFromNSDate name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⚠️Ⓜ️ ObjectiveCBridgeStubFromNSDate has very wide range of memory used between independent, repeated measurements.
ObjectiveCBridgeStubFromNSDate mem_pages [i1, i2]: min=[235, 230] 𝚫=5 R=[16, 18]
⚠️🔤 ObjectiveCBridgeStubToNSDateRef name is composed of 6 words.
Split ObjectiveCBridgeStubToNSDateRef name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⚠️Ⓜ️ ObjectiveCBridgeStubToNSDateRef has very wide range of memory used between independent, repeated measurements.
ObjectiveCBridgeStubToNSDateRef mem_pages [i1, i2]: min=[228, 225] 𝚫=3 R=[24, 24]
⚠️🔤 ObjectiveCBridgeFromNSDictionaryAnyObjectToString name is composed of 8 words.
Split ObjectiveCBridgeFromNSDictionaryAnyObjectToString name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⛔️🔤 ObjectiveCBridgeFromNSDictionaryAnyObjectToString name is 49 characters long.
Benchmark name should not be longer than 40 characters.
⚠️🔤 ObjectiveCBridgeFromNSDictionaryAnyObjectForced name is composed of 7 words.
Split ObjectiveCBridgeFromNSDictionaryAnyObjectForced name into dot-separated groups and variants. See http://bit.ly/BenchmarkNaming
⛔️🔤 ObjectiveCBridgeFromNSDictionaryAnyObjectForced name is 47 characters long.
Benchmark name should not be longer than 40 characters.
How to read the data The tables contain differences in performance which are larger than 8% and differences in code size which are larger than 1%.

If you see any unexpected regressions, you should consider fixing the
regressions before you merge the PR.

Noise: Sometimes the performance results (not code size!) contain false
alarms. Unexpected regressions which are marked with '(?)' are probably noise.
If you see regressions which you cannot explain you can try to run the
benchmarks again. If regressions still show up, please consult with the
performance team (@eeckstein).

Hardware Overview 
  Model Name: Mac Pro
  Model Identifier: MacPro6,1
  Processor Name: 12-Core Intel Xeon E5
  Processor Speed: 2.7 GHz
  Number of Processors: 1
  Total Number of Cores: 12
  L2 Cache (per Core): 256 KB
  L3 Cache: 30 MB
  Memory: 64 GB

@palimondo palimondo requested a review from eeckstein February 11, 2019 18:01
@palimondo
Copy link
Contributor Author

@eeckstein Please review 🙏

@palimondo palimondo merged commit fa9b52e into swiftlang:master Feb 11, 2019
@palimondo
Copy link
Contributor Author

@eeckstein Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eeckstein eeckstein eeckstein approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@palimondo @swift-ci @eeckstein