Add LaunchUITests using Maestro Mobile

[krystofwoldrich]

📜 Description

This PR adds a new UI test executed by Maestro in GH CI. This is part of the exploration to make UI Tests more stable.

💚 How did you test it?

Programatically rerun the Maestro Crash test 100 times.

1 Test (crash recovery only)

Results: 1/100 tests failed

Record of the results -> https://github.com/getsentry/sentry-cocoa/actions/runs/11162209598/job/31068682803?pr=4395

2 Test (crash recovery on iOS 15, 16, 17 and Obj-C, Swift and SwiftUI matrix)

Result: (1,7%) 4/234 tests failed (4/39 runs, in a run only 1 test job failed)
https://github.com/getsentry/sentry-cocoa/actions/runs/11216856513/job/31199896066

3 Test (matrix + LaunchIUTestSuite transferred to Maestro syntax)

Result: (4%) 7/174 tests failed (7/29, in a run only 1 test job failed))
https://github.com/getsentry/sentry-cocoa/actions/runs/11241823034/attempts/29?pr=4408

The script:

# Current version for one workflow
#!/bin/bash

GITHUB_REPO_OWNER="getsentry"
GITHUB_REPO_NAME="sentry-cocoa"
WORKFLOW_RUN="11241823034"
DEBOUNCE_TIMEOUT_MS=300 #5 minutes

for i in {1..2000}; do
  workflow=$(gh api \
    -H "Accept: application/json" \
    -H "X-GitHub-Api-Version: 2022-11-28" \
    "/repos/${GITHUB_REPO_OWNER}/${GITHUB_REPO_NAME}/actions/runs/${WORKFLOW_RUN}")

  workflow_name=$(echo "$workflow" | jq -r ".name")
  workflow_status=$(echo "$workflow" | jq -r ".status")
  workflow_conclusion=$(echo "$workflow" | jq -r ".conclusion")
  workflow_run_attempt=$(echo "$workflow" | jq -r ".run_attempt")

  echo "Attempt number $workflow_run_attempt of $workflow_name, status $workflow_status, conclusion $workflow_conclusion"

  if [[ "$workflow_status" == "completed" ]]; then
    rerun=$(gh api \
      --method POST \
      -H "Accept: application/json" \
      -H "X-GitHub-Api-Version: 2022-11-28" \
      /repos/${GITHUB_REPO_OWNER}/${GITHUB_REPO_NAME}/actions/runs/${WORKFLOW_RUN}/rerun)
    echo "Rerun requested for $workflow_name"
  fi

  sleep $DEBOUNCE_TIMEOUT_MS
done

# Previous version for single job

GITHUB_REPO_OWNER="getsentry"
GITHUB_REPO_NAME="sentry-cocoa"
WORKFLOW_RUN="11162209598"
JOB_TO_REREUN_INDEX="1"
DEBOUNCE_TIMEOUT_MS=300 #5 minutes

for i in {1..1000}; do
  workflow=$(gh api \
    -H "Accept: application/json" \
    -H "X-GitHub-Api-Version: 2022-11-28" \
    "/repos/${GITHUB_REPO_OWNER}/${GITHUB_REPO_NAME}/actions/runs/${WORKFLOW_RUN}/jobs")

  job_id=$(echo "$workflow" | jq -r ".jobs[${JOB_TO_REREUN_INDEX}].id")
  job_status=$(echo "$workflow" | jq -r ".jobs[${JOB_TO_REREUN_INDEX}].status")
  job_conclusion=$(echo "$workflow" | jq -r ".jobs[${JOB_TO_REREUN_INDEX}].conclusion")
  job_run_attempt=$(echo "$workflow" | jq -r ".jobs[${JOB_TO_REREUN_INDEX}].run_attempt")

  echo "Attempt number $job_run_attempt ($job_id), status $job_status, conclusion $job_conclusion"

  if [[ "$job_status" == "completed" ]]; then
    rerun=$(gh api \
      --method POST \
      -H "Accept: application/json" \
      -H "X-GitHub-Api-Version: 2022-11-28" \
      /repos/${GITHUB_REPO_OWNER}/${GITHUB_REPO_NAME}/actions/jobs/${job_id}/rerun)
    echo "Rerun requested"
  fi

  sleep $DEBOUNCE_TIMEOUT_MS
done

📝 Checklist

You have to check all boxes before merging:

• I reviewed the submitted code.
• I added tests to verify the changes.
• No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
• I updated the docs if needed.
• Review from the native team if needed.
• No breaking change or entry added to the changelog.
• No breaking change for hybrid SDKs or communicated to hybrid SDKs.

#skip-changelog

🔮 Next steps

• Add more iOS versions and other sample apps to fully replicated the current UI Tests for the Crash test
• Add the remaining UI Tests

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.296%. Comparing base (4eea4d1) to head (60fefae).
Report is 23 commits behind head on main.

Additional details and impacted files

@@              Coverage Diff              @@
##              main     #4395       +/-   ##
=============================================
- Coverage   91.424%   91.296%   -0.128%     
=============================================
  Files          628       610       -18     
  Lines        50530     49615      -915     
  Branches     18255     17881      -374     
=============================================
- Hits         46197     45297      -900     
+ Misses        4240      4226       -14     
+ Partials        93        92        -1     

see 75 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4eea4d1...60fefae. Read the comment docs.

[github-actions]

Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1228.88 ms	1246.51 ms	17.63 ms
Size	21.90 KiB	705.31 KiB	683.41 KiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
5eaadc5	1236.24 ms	1245.45 ms	9.20 ms
ec879f7	1304.84 ms	1337.04 ms	32.20 ms
7cd187e	1243.04 ms	1244.79 ms	1.75 ms
bb71736	1244.18 ms	1248.11 ms	3.92 ms
5e66a38	1224.16 ms	1237.76 ms	13.60 ms
d8926bf	1223.31 ms	1235.36 ms	12.05 ms
f1a6589	1253.67 ms	1269.06 ms	15.39 ms
4b08ceb	1237.75 ms	1249.61 ms	11.86 ms
62c15d4	1235.30 ms	1260.82 ms	25.52 ms
8ef07fb	1231.45 ms	1243.98 ms	12.53 ms

App size

Revision	Plain	With Sentry	Diff
5eaadc5	21.58 KiB	651.06 KiB	629.48 KiB
ec879f7	21.58 KiB	669.68 KiB	648.10 KiB
7cd187e	20.76 KiB	401.65 KiB	380.89 KiB
bb71736	21.58 KiB	683.64 KiB	662.06 KiB
5e66a38	22.85 KiB	408.88 KiB	386.03 KiB
d8926bf	21.58 KiB	670.39 KiB	648.81 KiB
f1a6589	22.85 KiB	408.87 KiB	386.02 KiB
4b08ceb	20.76 KiB	431.99 KiB	411.23 KiB
62c15d4	22.85 KiB	411.14 KiB	388.29 KiB
8ef07fb	21.58 KiB	707.42 KiB	685.84 KiB

Previous results on branch: kw-explore-new-e2e

Startup times

Revision	Plain	With Sentry	Diff
8d54369	1214.57 ms	1239.19 ms	24.62 ms

App size

Revision	Plain	With Sentry	Diff
8d54369	21.90 KiB	705.30 KiB	683.41 KiB

[krystofwoldrich]

closing in favor of #4421

We can revisit migrating the rest of the test suit to standalone app with maestro after the initial impl proves more stable.