-
-
Notifications
You must be signed in to change notification settings - Fork 372
ci: Use xcrun for crash UI test #5512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 8047b99 | 1226.37 ms | 1246.63 ms | 20.26 ms |
| 55f739c | 1226.06 ms | 1248.78 ms | 22.71 ms |
| 8ea5293 | 1242.70 ms | 1262.25 ms | 19.55 ms |
| 5cfc768 | 1220.74 ms | 1245.06 ms | 24.32 ms |
| 701b301 | 1226.10 ms | 1245.57 ms | 19.47 ms |
| db9572a | 1223.13 ms | 1241.60 ms | 18.47 ms |
| db9572a | 1200.27 ms | 1234.80 ms | 34.53 ms |
| f97a070 | 1218.88 ms | 1253.12 ms | 34.24 ms |
| db9572a | 1212.61 ms | 1237.73 ms | 25.13 ms |
| acac774 | 1217.76 ms | 1253.29 ms | 35.52 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 8047b99 | 23.75 KiB | 855.37 KiB | 831.62 KiB |
| 55f739c | 23.75 KiB | 858.73 KiB | 834.98 KiB |
| 8ea5293 | 23.75 KiB | 852.24 KiB | 828.49 KiB |
| 5cfc768 | 23.75 KiB | 850.73 KiB | 826.98 KiB |
| 701b301 | 23.75 KiB | 867.16 KiB | 843.41 KiB |
| db9572a | 23.75 KiB | 858.64 KiB | 834.89 KiB |
| db9572a | 23.75 KiB | 858.69 KiB | 834.93 KiB |
| f97a070 | 23.75 KiB | 858.68 KiB | 834.93 KiB |
| db9572a | 23.75 KiB | 858.65 KiB | 834.90 KiB |
| acac774 | 23.75 KiB | 866.51 KiB | 842.76 KiB |
Previous results on branch: ci/swiftui-crash-test
Startup times
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 1c0ea5a | 1225.48 ms | 1247.63 ms | 22.15 ms |
| c41bae2 | 1232.57 ms | 1250.98 ms | 18.41 ms |
| 56d7ced | 1229.54 ms | 1245.23 ms | 15.69 ms |
| 37b5a6f | 1218.55 ms | 1244.44 ms | 25.89 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 1c0ea5a | 23.75 KiB | 866.68 KiB | 842.93 KiB |
| c41bae2 | 23.75 KiB | 866.67 KiB | 842.93 KiB |
| 56d7ced | 23.75 KiB | 866.51 KiB | 842.76 KiB |
| 37b5a6f | 23.75 KiB | 866.51 KiB | 842.76 KiB |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #5512 +/- ##
=============================================
- Coverage 86.250% 86.146% -0.105%
=============================================
Files 403 403
Lines 34911 34900 -11
Branches 15158 14951 -207
=============================================
- Hits 30111 30065 -46
- Misses 4752 4792 +40
+ Partials 48 43 -5 see 15 files with indirect coverage changes Continue to review full report in Codecov by Sentry.
🚀 New features to boost your workflow:
|
noahsmartin
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this looks great, exactly what I had in mind re: #5422
If we wanted to have more control of the app we could start a HTTP server in the app being tested and make requests to it from the test host. This is what we do in FaultOrdering and in general is a pattern we've used a lot at emerge. The idea to use user defaults is great though! So much simpler for something like this. Thanks @philipphofmann !
We now use xcrun for testing the SDK when crashing with #5512. We can remove the unused crash testing logic for the SwiftUITestSample.
We now use xcrun for testing the SDK when crashing with #5512. We can remove the unused crash testing logic for the SwiftUITestSample.
This reverts commit fc6557e.
This reverts commit fc6557e.
XCTest isn't built for crashing during tests. Instead of using XCTest to press a button and let a test app crash, we now use UserDefaults to tell the test app to crash during launch. We then simply launch the app again and wait to see if it keeps running. This is basically the same as the
testCrashof the SwiftUITestSample without using XCTests.The plan is to also replace the
ui_critical_tests_ios_swiftui_envelopewith this approach if this looks acceptable.I still need to use Xcodegen for the test project and refine the script a bit so it's more reliable.
#skip-changelog