Skip to content

Conversation

@philipphofmann
Copy link
Member

@philipphofmann philipphofmann commented Jun 27, 2025

XCTest isn't built for crashing during tests. Instead of using XCTest to press a button and let a test app crash, we now use UserDefaults to tell the test app to crash during launch. We then simply launch the app again and wait to see if it keeps running. This is basically the same as the testCrash of the SwiftUITestSample without using XCTests.

The plan is to also replace the ui_critical_tests_ios_swiftui_envelope with this approach if this looks acceptable.

I still need to use Xcodegen for the test project and refine the script a bit so it's more reliable.

#skip-changelog

@github-actions
Copy link
Contributor

github-actions bot commented Jun 27, 2025

Performance metrics 🚀

  Plain With Sentry Diff
Startup time 1215.73 ms 1240.37 ms 24.64 ms
Size 23.75 KiB 866.69 KiB 842.94 KiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
8047b99 1226.37 ms 1246.63 ms 20.26 ms
55f739c 1226.06 ms 1248.78 ms 22.71 ms
8ea5293 1242.70 ms 1262.25 ms 19.55 ms
5cfc768 1220.74 ms 1245.06 ms 24.32 ms
701b301 1226.10 ms 1245.57 ms 19.47 ms
db9572a 1223.13 ms 1241.60 ms 18.47 ms
db9572a 1200.27 ms 1234.80 ms 34.53 ms
f97a070 1218.88 ms 1253.12 ms 34.24 ms
db9572a 1212.61 ms 1237.73 ms 25.13 ms
acac774 1217.76 ms 1253.29 ms 35.52 ms

App size

Revision Plain With Sentry Diff
8047b99 23.75 KiB 855.37 KiB 831.62 KiB
55f739c 23.75 KiB 858.73 KiB 834.98 KiB
8ea5293 23.75 KiB 852.24 KiB 828.49 KiB
5cfc768 23.75 KiB 850.73 KiB 826.98 KiB
701b301 23.75 KiB 867.16 KiB 843.41 KiB
db9572a 23.75 KiB 858.64 KiB 834.89 KiB
db9572a 23.75 KiB 858.69 KiB 834.93 KiB
f97a070 23.75 KiB 858.68 KiB 834.93 KiB
db9572a 23.75 KiB 858.65 KiB 834.90 KiB
acac774 23.75 KiB 866.51 KiB 842.76 KiB

Previous results on branch: ci/swiftui-crash-test

Startup times

Revision Plain With Sentry Diff
1c0ea5a 1225.48 ms 1247.63 ms 22.15 ms
c41bae2 1232.57 ms 1250.98 ms 18.41 ms
56d7ced 1229.54 ms 1245.23 ms 15.69 ms
37b5a6f 1218.55 ms 1244.44 ms 25.89 ms

App size

Revision Plain With Sentry Diff
1c0ea5a 23.75 KiB 866.68 KiB 842.93 KiB
c41bae2 23.75 KiB 866.67 KiB 842.93 KiB
56d7ced 23.75 KiB 866.51 KiB 842.76 KiB
37b5a6f 23.75 KiB 866.51 KiB 842.76 KiB

@codecov
Copy link

codecov bot commented Jun 27, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 86.146%. Comparing base (25f2d2c) to head (472305e).
Report is 3 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@              Coverage Diff              @@
##              main     #5512       +/-   ##
=============================================
- Coverage   86.250%   86.146%   -0.105%     
=============================================
  Files          403       403               
  Lines        34911     34900       -11     
  Branches     15158     14951      -207     
=============================================
- Hits         30111     30065       -46     
- Misses        4752      4792       +40     
+ Partials        48        43        -5     

see 15 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 25f2d2c...472305e. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Contributor

@noahsmartin noahsmartin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks great, exactly what I had in mind re: #5422

If we wanted to have more control of the app we could start a HTTP server in the app being tested and make requests to it from the test host. This is what we do in FaultOrdering and in general is a pattern we've used a lot at emerge. The idea to use user defaults is great though! So much simpler for something like this. Thanks @philipphofmann !

@philipphofmann philipphofmann marked this pull request as ready for review June 30, 2025 14:26
@philipphofmann philipphofmann merged commit fc6557e into main Jun 30, 2025
103 of 106 checks passed
@philipphofmann philipphofmann deleted the ci/swiftui-crash-test branch June 30, 2025 14:29
philipphofmann added a commit that referenced this pull request Jul 2, 2025
We now use xcrun for testing the SDK when crashing with #5512. We can
remove the unused crash testing logic for the SwiftUITestSample.
philipphofmann added a commit that referenced this pull request Jul 2, 2025
We now use xcrun for testing the SDK when crashing with #5512. We can
remove the unused crash testing logic for the SwiftUITestSample.
philipsawyerdd added a commit to justin-doordash/sentry-cocoa that referenced this pull request Sep 25, 2025
philipsawyerdd added a commit to justin-doordash/sentry-cocoa that referenced this pull request Sep 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants