E2E Github Action is flaky

[mikehardy]

This is a tracking issue for a work effort I intend to undertake personally

I will greatly appreciate any collaboration with anyone that has thoughts or help to offer, so feel free to chime in!

Observation: The E2E android test fails frequently, unknown root cause

Recent example:

detox[4758] INFO:  [DetoxServer.js] server listening on localhost:49713...
detox[4758] ERROR: [exec.js/EXEC_FAIL, #23] ""/Users/runner/Library/Android/sdk/platform-tools/adb" -s emulator-5554 shell "ps | grep \"com\.invertase\.testing$\""" failed with code = 1, stdout and stderr:

detox[4758] ERROR: [exec.js/EXEC_FAIL, #23] 
detox[4758] ERROR: [exec.js/EXEC_FAIL, #23] 
  1) "before all" hook in "{root}"
detox[4758] INFO:  at e2e/init.js:53:11 
  ✨ Tests Complete ✨ 
detox[4758] ERROR: at internal/process/warning.js:25:20 
 (node:4758) UnhandledPromiseRejectionWarning: DetoxRuntimeError: Failed to run application on the device

I have noticed that it fails sometimes locally as well, unknown cause.

I'd like to figure out exactly why.

Initial Thoughts:

• reproduce locally, fix (ideally)
• Removing the jet dependency (and possibly using cavy.app) is a stretch goal

[mikehardy]

initial ADB logs containing extra info show something that seems racy in the app/detox-testee side react native context bringup + websocket init

This could use some upstream attention, but we can at least hack out the synchronization now

1. Detox has no way to turn off synchronization in Android, and it should (can hang on UIManager and Timers)
2. React Native has UIManager issues where the queue never empties

[mikehardy]

Not sure what step detox was on yet but it failed once on android with a device log of:

08-14 19:32:49.164 1906 4879 W ActivityManager: Invalid packageName: com.invertase.testing

As if the adb push / adb install failed? Not sure how often that happens, but those steps should be re-tryable - really everything should be retried via full emulator destruction up until the application logs in

Solve with https://github.com/wix/Detox/blob/master/docs/APIRef.Configuration.md#behavior-configuration reinstallapp: false in detox config and do a retry-able install first

[mikehardy]

New flaky E2E call:

firestore 'should clear any persisted data'

08-15 02:12:21.461  3937  8568 E unknown:ReactNative: Exception in native call
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: java.lang.IllegalStateException: FirebaseFirestore has already been started and its settings can no longer be changed. You can only call setFirestoreSettings() before calling any other methods on a FirebaseFirestore object.
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at com.google.firebase.firestore.FirebaseFirestore.setFirestoreSettings(com.google.firebase:firebase-firestore@@21.4.3:191)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at io.invertase.firebase.firestore.UniversalFirebaseFirestoreCommon.setFirestoreSettings(UniversalFirebaseFirestoreCommon.java:93)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at io.invertase.firebase.firestore.UniversalFirebaseFirestoreCommon.getFirestoreForApp(UniversalFirebaseFirestoreCommon.java:45)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at io.invertase.firebase.firestore.UniversalFirebaseFirestoreModule.clearPersistence(UniversalFirebaseFirestoreModule.java:83)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at io.invertase.firebase.firestore.ReactNativeFirebaseFirestoreModule.clearPersistence(ReactNativeFirebaseFirestoreModule.java:50)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at java.lang.reflect.Method.invoke(Native Method)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at com.facebook.react.bridge.JavaMethodWrapper.invoke(JavaMethodWrapper.java:372)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at com.facebook.react.bridge.JavaModuleWrapper.invoke(JavaModuleWrapper.java:151)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at com.facebook.react.bridge.queue.NativeRunnable.run(Native Method)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at android.os.Handler.handleCallback(Handler.java:873)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at android.os.Handler.dispatchMessage(Handler.java:99)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at com.facebook.react.bridge.queue.MessageQueueThreadHandler.dispatchMessage(MessageQueueThreadHandler.java:27)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at android.os.Looper.loop(Looper.java:193)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at com.facebook.react.bridge.queue.MessageQueueThreadImpl$4.run(MessageQueueThreadImpl.java:226)
08-15 02:12:21.461  3937  8568 E unknown:ReactNative: 	at java.lang.Thread.run(Thread.java:764)

got this one again 20200828

[mikehardy]

The brew applesimutils download step can fail (of course, it touches the network...) and needs a retry.

[mikehardy]

on android -

seen again 20210319
seen a lot 20210419 - this is becoming really troublesome

Thu, 04 Mar 2021 18:58:35 GMT   1) auth()
Thu, 04 Mar 2021 18:58:35 GMT        "before each" hook for "returns the same user with multiple subscribers #1815":
Thu, 04 Mar 2021 18:58:35 GMT      NativeFirebaseError: [auth/no-current-user] No user currently signed in.

[mikehardy]

  1) auth()
       onAuthStateChanged()
         calls callback with the current user and when auth state changes:
     expected 'spy' to be called twice but was called thrice

[mikehardy]

  1) auth()
       "before all" hook in "auth()":
     NativeFirebaseError: [auth/unknown] An internal error has occurred. [ unexpected end of stream on com.android.okhttp.Address@c2602640 ]

[mikehardy]

It appears Detox v18 #5734 has added a new wrinkle on iOS with a native crash in DetoxSync

The app has crashed, see the details below:

Signal 11 was raised
(
        0   Detox                               0x00000001104c5a15 +[NSThread(DetoxUtils) dtx_demangledCallStackSymbols] + 37
        1   Detox                               0x00000001104c8e10 __DTXHandleCrash + 464
        2   Detox                               0x00000001104c9555 __DTXHandleSignal + 59
        3   libsystem_platform.dylib            0x00007fff60335d7d _sigtramp + 29
        4   ???                                 0x00007f8fdf920050 0x0 + 140255907938384
        5   CoreFoundation                      0x0000000114170833 -[__NSDictionaryM dealloc] + 128
        6   libobjc.A.dylib                     0x0000000113f3b604 objc_object::sidetable_release(bool, bool) + 174
        7   libobjc.A.dylib                     0x0000000113f378ad _object_remove_assocations + 562
        8   libobjc.A.dylib                     0x0000000113f34c02 objc_destructInstance + 84
        9   libobjc.A.dylib                     0x0000000113f3ad88 -[NSObject dealloc] + 21
        10  libobjc.A.dylib                     0x0000000113f3b604 objc_object::sidetable_release(bool, bool) + 174
        11  CFNetwork                           0x0000000111038610 _CFNetworkHTTPConnectionCacheSetLimit + 162696
        12  CFNetwork                           0x0000000110e81183 CFNetwork + 24963
        13  DetoxSync                           0x0000000146cf0220 ____detox_sync_dispatch_wrapper_block_invoke + 23
        14  libdispatch.dylib                   0x00000001148c370d _dispatch_call_block_and_release + 12
        15  libdispatch.dylib                   0x00000001148c48df _dispatch_client_callout + 8
        16  libdispatch.dylib                   0x00000001148cae15 _dispatch_lane_serial_drain + 715
        17  libdispatch.dylib                   0x00000001148cb9c3 _dispatch_lane_invoke + 455
        18  libdispatch.dylib                   0x00000001148d5f81 _dispatch_workloop_worker_thread + 772
        19  libsystem_pthread.dylib             0x00007fff6034045d _pthread_wqthread + 314
        20  libsystem_pthread.dylib             0x00007fff6033f42f start_wqthread + 15
)

[github-actions]

Hello 👋, to help manage issues we automatically close stale issues.

This issue has been automatically marked as stale because it has not had activity for quite some time.Has this issue been fixed, or does it still require attention?

This issue will be closed in 15 days if no further activity occurs.

Thank you for your contributions.

[mikehardy]

Hasn't been flaky for a while!