fix(test): Flaky E2E Android Metrics App Start Time

[antonis]

📢 Type of change

• Bugfix
• New feature
• Enhancement
• Refactoring

⚠️ Depends on: getsentry/action-app-sdk-overhead-metrics#23

📜 Description

Since the main activity of the application is displayed before RN start executing and thus before the JS SDK inits this PR:

• Creates a new AppReadyActivity and exposes it to RN
• Starts AppReadyActivity when JS loads
• Measures startup time with the AppReadyActivity
• Reverts diffMin workaround introduced with chore: Extends the Android metrics diffMin similar to iOS #4159

💡 Motivation and Context

Fixes #3413

💚 How did you test it?

CI

📝 Checklist

• I reviewed submitted code
• I added tests to verify changes
• No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled
• All tests passing
• No breaking changes

🔮 Next steps

#skip-changelog

[github-actions]

iOS (legacy) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1225.24 ms	1231.35 ms	6.10 ms
Size	2.36 MiB	3.10 MiB	753.36 KiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
77680ec+dirty	1231.81 ms	1238.35 ms	6.54 ms
1faf8e3+dirty	1214.87 ms	1222.83 ms	7.97 ms
7bc4d75+dirty	1233.40 ms	1229.56 ms	-3.83 ms
d7401ac+dirty	1252.38 ms	1275.04 ms	22.66 ms
8ae23a7+dirty	1230.02 ms	1227.62 ms	-2.40 ms
5a22220+dirty	1209.49 ms	1220.94 ms	11.45 ms
80b2ce3+dirty	1265.92 ms	1268.60 ms	2.69 ms
b95b8af+dirty	1221.39 ms	1228.52 ms	7.13 ms
d0bf494+dirty	1289.40 ms	1298.40 ms	9.00 ms
f06c879+dirty	1252.64 ms	1259.66 ms	7.02 ms

App size

Revision	Plain	With Sentry	Diff
77680ec+dirty	2.36 MiB	3.10 MiB	753.42 KiB
1faf8e3+dirty	2.36 MiB	3.08 MiB	736.75 KiB
7bc4d75+dirty	2.36 MiB	3.10 MiB	752.58 KiB
d7401ac+dirty	2.36 MiB	2.83 MiB	481.14 KiB
8ae23a7+dirty	2.36 MiB	3.10 MiB	752.42 KiB
5a22220+dirty	2.36 MiB	2.92 MiB	570.21 KiB
80b2ce3+dirty	2.36 MiB	2.84 MiB	486.98 KiB
b95b8af+dirty	2.36 MiB	3.14 MiB	793.32 KiB
d0bf494+dirty	2.36 MiB	2.83 MiB	481.15 KiB
f06c879+dirty	2.36 MiB	2.88 MiB	530.42 KiB

[github-actions]

iOS (new) Performance metrics 🚀

	Plain	With Sentry	Diff
Startup time	1235.94 ms	1229.96 ms	-5.98 ms
Size	2.92 MiB	3.66 MiB	758.63 KiB

Baseline results on branch: main

Startup times

Revision	Plain	With Sentry	Diff
77680ec+dirty	1237.65 ms	1236.24 ms	-1.41 ms
1faf8e3+dirty	1223.38 ms	1220.56 ms	-2.82 ms
7bc4d75+dirty	1222.13 ms	1216.39 ms	-5.74 ms
d7401ac+dirty	1288.10 ms	1289.54 ms	1.44 ms
8ae23a7+dirty	1233.67 ms	1229.52 ms	-4.15 ms
5a22220+dirty	1246.18 ms	1249.61 ms	3.43 ms
80b2ce3+dirty	1245.12 ms	1262.04 ms	16.92 ms
b95b8af+dirty	1235.60 ms	1242.06 ms	6.46 ms
d0bf494+dirty	1266.20 ms	1267.52 ms	1.32 ms
f06c879+dirty	1285.14 ms	1285.86 ms	0.72 ms

App size

Revision	Plain	With Sentry	Diff
77680ec+dirty	2.92 MiB	3.66 MiB	758.54 KiB
1faf8e3+dirty	2.92 MiB	3.64 MiB	742.61 KiB
7bc4d75+dirty	2.92 MiB	3.66 MiB	757.15 KiB
d7401ac+dirty	2.92 MiB	3.40 MiB	488.06 KiB
8ae23a7+dirty	2.92 MiB	3.66 MiB	757.67 KiB
5a22220+dirty	2.92 MiB	3.48 MiB	575.81 KiB
80b2ce3+dirty	2.92 MiB	3.40 MiB	492.75 KiB
b95b8af+dirty	2.92 MiB	3.69 MiB	794.16 KiB
d0bf494+dirty	2.92 MiB	3.40 MiB	488.08 KiB
f06c879+dirty	2.92 MiB	3.44 MiB	533.24 KiB

[antonis]

Though the AppReadyActivity is launched I failed to detect it with Appium getsentry/action-app-sdk-overhead-metrics#23. I'll stop for now and revisit the issue later.

[antonis]

A better simpler approach could be to just emit the app ready timestamp from RN and then parse it from appium to calculate the total load time.

2024-11-20 20:46:18.098 16742-16837 ReactNativeJS           com.testappplain                     I  'AppReadyTimestampMs:', 1732128378092

[antonis]

A better simpler approach could be to just emit the app ready timestamp from RN and then parse it from appium to calculate the total load time.

This failed too since capturing the logs on CI did not work (metric lib branch https://github.com/antonis/action-app-sdk-overhead-metrics/commits/antonis/add-log-timestamp-regex/).

I'm stopping this effort for now.

[vaind]

A better simpler approach could be to just emit the app ready timestamp from RN and then parse it from appium to calculate the total load time.

Yes, this looks like a good idea. We may not even need to post the timestamp, just any message and we can grab log timestamps from logcat

[antonis]

Yes, this looks like a good idea.

Thank you for iterating on this @vaind 🙇

What I tried with this approach is:

• Log a message when RN is loaded
• Pass a regex to capture the timestamp in the metrics lib and calculate the load time.

This didn't work either since the log wasn't emitted or captured on ci (it worked on my machine with a release build 😅).

I guess the reason might be similar to why capturing the 2nd activity did not work.

We may not even need to post the timestamp, just any message and we can grab log timestamps from logcat

Good idea. This can simplify the implementation further.

Any ideas on what I might be missing are more than welcome 🙇