test: Run Performance Tests in CI

[philipphofmann]

The unit tests contain a few performance tests. These tests need baselines stored in xcbaselines. Xcode only uses these baselines if the type of machine in the Info.plist and the machine running the tests match. Furthermore, there is no way of defining a generic combo so the tests run on multiple machines. There is no official Apple documentation confirming this, but personal experience and this article strengthen this statement. As the current baselines are taken from a local computer the performance tests don't run in CI.

We could use system_profiler to get the specifications of the machine in CI and update Info.plist accordingly so Xcode runs the performance tests.

Example usage of system_profiler.

system_profiler -detailLevel mini -xml > ~/Desktop/system_report_mini.spx

Specifications needed for the Info.plist file.

<key>1C0AB51E-7DAD-4469-8384-B6DE6A8E9023</key>
<dict>
	<key>localComputer</key>
	<dict>
		<key>busSpeedInMHz</key>
		<integer>400</integer>
		<key>cpuCount</key>
		<integer>1</integer>
		<key>cpuKind</key>
		<string>Quad-Core Intel Core i7</string>
		<key>cpuSpeedInMHz</key>
		<integer>2800</integer>
		<key>logicalCPUCoresPerPackage</key>
		<integer>8</integer>
		<key>modelCode</key>
		<string>MacBookPro15,2</string>
		<key>physicalCPUCoresPerPackage</key>
		<integer>4</integer>
		<key>platformIdentifier</key>
		<string>com.apple.platform.macosx</string>
	</dict>
	<key>targetArchitecture</key>
	<string>x86_64</string>
</dict>

Points to consider:

• <key>1C0AB51E-7DAD-4469-8384-B6DE6A8E9023</key> needs to have a matching 1C0AB51E-7DAD-4469-8384-B6DE6A8E9023.plist file.

• sentry-cocoa/scripts/xcode-test.sh

  Line 27 in 7e975ea

  DESTINATION="platform=iOS Simulator,OS=latest,name=iPhone 8"

  needs to match

  sentry-cocoa/Sentry.xcodeproj/xcshareddata/xcbaselines/63AA76641EB8CB2F00D153DE.xcbaseline/Info.plist

  Lines 56 to 57 in 7e975ea

  	<key>modelCode</key>
  	<string>iPhone10,4</string>

  Keep in mind that the model code and the device name iPhone 8 are not the same.
• Ideally, we create a script and put it into scripts/.

[philipphofmann]

As pointed out by @bruno-garcia, running performance tests in CI is maybe not the best idea, as you don't have full control over how busy the CI machine is. Maybe it's better to have a dedicated self-hosted machine only running the performance tests. When having remote access to this machine setting up the xcbaselines is going to be straightforward.

[philipphofmann]

We now have performance metrics for the SDK #2067 and also benchmarking workflows. We can close this.