Detox fails unless Android emulator is running prior to test execution - Exceeded timeout of 300000ms while handling jest-circus "setup" event

[BrentWardOx]

What happened?

I followed the Detox setup guide for a simple React Native test app I am working on. I managed to setup an Android Emulator which is compatible with Detox, and my first test works great!

The only issue is that my Android Emulator must be running before I execute my detox test command. Otherwise, running detox test launches my Android Emulator and hangs. The error message is Exceeded timeout of 300000ms while handling jest-circus "setup" event.

What was the expected behaviour?

Running detox test for an Emulator config should:

• - attach to the corresponding Emulator if it is already running and execute tests
• - launch the corresponding Emulator if it is not running and execute tests

Was it tested on latest Detox?

• I have tested this issue on the latest Detox release and it still reproduces.

Did your test throw out a timeout?

• I have followed the instructions under Identifying which synchronization mechanism causes us to wait too much.

Help us reproduce this issue!

• Generate a new React Native app
• Follow the Detox Setup Guide for creating a first test on an Android Emulator with Jest, creating an Android Emulator config
• Execute detox test for the Android Emulator config while the Emulator is not active
• The Emulator should launch, but the app will not launch on the Emulator and execution should hang and eventually timeout

In what environment did this happen?

Detox version: 19.6.5
React Native version: 0.66.3
Node version: 16.13.0
Device model: Pixel 5 (emulator)
Android version: API 32
Test-runner: jest-circus

Detox logs

Detox logs

11:46:19.836 detox[33000] DEBUG: [ALLOCATE_DEVICE] Trying to allocate a device based on "Pixel_5_API_32"
11:46:19.837 detox[33000] DEBUG: [EXEC_CMD, #2] "/Users/myUserName/Library/Android/sdk/platform-tools/adb"  devices
11:46:20.030 detox[33000] DEBUG: [EXEC_SUCCESS, #2] List of devices attached


11:46:20.031 detox[33000] DEBUG: [ALLOCATE_DEVICE] Settled on emulator-13184
11:46:20.032 detox[33000] DEBUG: [SPAWN_CMD] /Users/myUserName/Library/Android/sdk/emulator/emulator -verbose -no-audio -no-boot-anim -port 13184 @Pixel_5_API_32
11:46:39.690 detox[33000] ERROR: Exceeded timeout of 20000ms while handling jest-circus "setup" event
11:46:39.725 detox[33000] ERROR: Aborted detox.init() execution, and now running detox.cleanup()

HINT: Most likely, your test runner is tearing down the suite due to the timeout error

Device logs

Device logs

{"name":"detox","hostname":"myHostName.local","pid":33133,"__filename":"EmulatorAllocationHelper.js","level":20,"event":"ALLOCATE_DEVICE","msg":"Settled on emulator-15278","time":"2022-04-21T16:49:07.933Z","v":0}
{"name":"detox","hostname":"myHostName.local","pid":33133,"__filename":"launchEmulatorProcess.js","fn":"boot","level":20,"event":"SPAWN_CMD","msg":"/Users/myUserName/Library/Android/sdk/emulator/emulator -verbose -no-audio -no-boot-anim -port 15278 @Pixel_5_API_32","time":"2022-04-21T16:49:07.934Z","v":0}
{"name":"detox","hostname":"myHostName.local","pid":33133,"level":50,"msg":"Exceeded timeout of 20000ms while handling jest-circus \"setup\" event","time":"2022-04-21T16:49:27.775Z","v":0}
{"name":"detox","hostname":"myHostName.local","pid":33133,"__filename":"ArtifactsManager.js","level":10,"event":"ARTIFACTS_LIFECYCLE","fn":"onBeforeCleanup","msg":"artifactsManager.onBeforeCleanup()","time":"2022-04-21T16:49:27.803Z","v":0}

More data, please!

I've tried creating a number of Android Emulators on various devices and APIs to see if the issue persists. This issue persists on Android Emulators for each of the Pixel models (1-5) and for the last several APIs (28-32).

Also, adding --retries 2 to the end of my test script does not alleviate the issue. The process still times out on the first attempt and hangs without retrying.

[d4vidi]

@BrentWardOx happy to hear that setting up the first test went well!
Please try running the emulator launching line manually and report back the results:

 /Users/myUserName/Library/Android/sdk/emulator/emulator -verbose -no-audio -no-boot-anim -port 13184 @Pixel_5_API_32

Also, let's switch this to stack overflow.

[github-actions]

We use the issue tracker exclusively for bug reports and feature requests. This issue appears to be a general usage or support question. Instead, please ask a question on Stack Overflow with the detox tag.

Feel free to post your Stack Overflow question here for more visibility! We'll take a look at it.

Refer to this discussion for more information about our policy on issues.

[BrentWardOx]

@d4vidi Thanks for the suggestion! I ran the emulator command you posted, and it launched my emulator just fine. Given that, I managed to solve the issue with a simple workaround:

emulator @Pixel_5_API_32 & until adb shell true; do sleep .5; done && detox test

^ This command launches the emulator in a background process, starts a while loop which waits for the emulator to be available, and then executes my tests once the emulator is ready. Works like a charm.

I followed the Detox Setup Guide carefully and ran into this issue with a pretty basic React Native app, so there's a high chance someone else will run into this same issue. Posting the solution here. Hope this helps!

[d4vidi]

That's okay but not ideal. Technically, it's better to have Detox manage your test device allocation.

[dklymenk]

I am running into a similar issue, but only with a release build.

In debug mode detox starts the emulator, pushes the build and all the tests pass.
In release mode detox starts the emulator, but doesn't install the detox build and the test subsequently times out.

Similarly to the OP, if I start the emulator in advance, the app installs and the tests pass.

[dklymenk]

Seems to be the case only with google play images. The issue is not reproducible with AOSP (API 28 for example).

@BrentWardOx you don't mention the flavor of the android image in your issue, so in case you used googled images, give AOSP images a try.

[BrentWardOx]

@dklymenk Thanks for the suggestion! I was using an emulator with Android API 32 (Google APIs), which was a Google Play Image.

However, I created a new emulator using Android 12.0 (Android Open Source Project, API 31) and my results are the same: running detox test opens my new AOSP Emulator yet fails to run the tests.

My results are also the same whether I build for debug or release. The only method I've found which works, thus far, is pre-launching the emulator and waiting for it to become available.

[dklymenk]

@BrentWardOx Give API28 a try. I found that anything above that has that issue.

[BrentWardOx]

@dklymenk Yep, that works! Switching my Emulator to target AOSP API 28 works great. Thank you!

[d4vidi]

Yep. AOSP images FTW.

[ghitier]

I'm encountering this issue as well using AOSP API 32. The detox init ends up timing out in the same scenario (app not launched prior to detox test), therefore the tests fail.

Despite the workaround of launching the emulator manually, I'd suggest this to be re-evaluated as a potential bug (instead of a question), and the issue re-opened.

[MoOx]

There is definitely a bug here. Should not be closed.
It's very difficult and confusing to find the reason of this problem on CI for example...

[gabrieldonadel]

Yeah, I've been experiencing this very same error for months now,

detox[3089] ERROR: Exceeded timeout of 300000ms while handling jest-circus "setup" event
detox[3089] ERROR: Aborted detox.init() execution, and now running detox.cleanup()

At least once a day I get this error on CI running tests on a release build on iOS with two workers.

I'm using detox 19.7.1

[rikur]

Getting the same error. API 32 and Detox 19.7.1

[jd20]

Same here, AOSP API 32 and Detox 19.8.4.

[jd20]

I can confirm that downgrading to an AOSP API 28 emulator works as a temporary fix.

[sangliquidus]

Got the error with API 32 and Detox 19.7.1. Anything update?

[d4vidi]

Noted. Let's follow-up on the discussion in the dedicated issue (#3560).

[zacharyweidenbach]

@d4vidi
Another data point, for your consideration, this is also happening for my team with iOS running in github actions, presenting in the same was as @gabrieldonadel described here:
#3342 (comment)

It seems to be correlated with the --workers cli argument being greater than 1.

2022-08-26T17:40:33.1347720Z 17:40:32.722 detox[69887] TRACE: [WS_SEND] {"type":"isReady","params":{},"messageId":-1000}
2022-08-26T17:40:33.1450500Z 17:40:32.723 detox[69887] TRACE: [WSS_GET_FROM, #tester] {"type":"isReady","params":{},"messageId":-1000}
2022-08-26T17:40:33.1553010Z 17:40:32.723 detox[69887] TRACE: [WSS_SEND_TO, #app] {"type":"isReady","params":{},"messageId":-1000}
2022-08-26T17:40:33.1655690Z 17:40:32.864 detox[69885] ERROR: Exceeded timeout of 300000ms while handling jest-circus "setup" event
2022-08-26T17:40:33.6576540Z 17:40:33.654 detox[69886] TRACE: [EXEC_TRY_FAIL, #18] An error was encountered processing the command (domain=NSPOSIXErrorDomain, code=3):
2022-08-26T17:40:33.6698480Z Application termination failed.
2022-08-26T17:40:33.6805290Z FBSSystemService reported failure without an error, possibly because the app is not currently running.
2022-08-26T17:40:33.6938700Z 
2022-08-26T17:40:33.7040450Z 17:40:33.654 detox[69886] DEBUG: [EXEC_TRY, #18] Terminating com.OMITTED...
2022-08-26T17:40:34.1105360Z 17:40:34.101 detox[69885] TRACE: [ARTIFACTS_LIFECYCLE] artifactsManager.onBeforeCleanup()
2022-08-26T17:40:34.1118510Z 17:40:34.104 detox[69885] ERROR: Aborted detox.init() execution, and now running detox.cleanup()
2022-08-26T17:40:34.1220230Z 
2022-08-26T17:40:34.1321600Z HINT: Most likely, your test runner is tearing down the suite due to the timeout error
2022-08-26T17:40:34.1423250Z 17:40:34.120 detox[69885] DEBUG: [WSS_CLOSE] Detox server has been closed gracefully

The following test fails with TypeError: _detox.device.getPlatform is not a function
The getPlatform is invoked within the setup file linked via setupFilesAfterEnv in the Jest config. Detox is not properly instantiated in the global scope because of the timeout with the jest-circus "setup" event. This only seems to happen once over the lifetime of the test run, and detox is able to recover and start the next test. And frustratingly, it only seems to happen in CI. In our case, we are using the macos-12 container on github actions, in a very similar manner as react-native-firebase, although we use Jest instead of Mocha.

I would post this in the new thread you linked, but that seems dedicated to the issue as it pertains to Android API 32. This seems to be a concurrency issue/race condition that tends to only present in the very resource constrained environment like the virtual containers in CI.

EDIT:
I have tried using retries to alleviate the issue, but the --retries is just a passthrough to the Jest retry configuration. And from what I can see, a retry does not run through the environment setup step again, so what happens is a guaranteed failure no matter the amount of retries.

[d4vidi]

@zacharyweidenbach Please have a look at #3518

[zacharyweidenbach]

@d4vidi

That does not resemble the error that I'm seeing. I have also bumped detox to ^19.10.0, which should include Mike Hardy's fix, and the result is still the same. I actually suspect that in CI, Detox has trouble creating the other simulators and starting them. Jest-Circus cannot finish the setup because Detox can never complete the handshake with the new simulators in time. I'm going to try running xcrun simctl list before and after the first try at running the test suite. Then I will retry the test suite and see if, after already having the iPhone 13-Detox devices created, the suite succeeds. I will report my findings back here.

[zacharyweidenbach]

@d4vidi

Detox's ability to create a new simulator instance in CI before running the test seems to be the issue, or at least somewhere near the bullseye of the issue. The Exceeded timeout of 300000ms while handling jest-circus "setup" event only seems to occur on the very first tests that are attempted with multiple workers. I can achieve a successful test run of my full testing suite if I do the following:

1. Run a "dummy" test suite beforehand that consists of 3 tests that only launch the app before completing. I allocate 3 workers for these tests and configure CI to continue if there is an error on this step.

      - name: Initialize Detox Tests
        id: detox-init
        continue-on-error: true
        run: yarn detox test --use-custom-logger false --loglevel warn --configuration ios.sim.gha --cleanup --workers 3 --runner-config e2e/InitCITests/config.json --keepLockFile true

This step reliably fails for all three tests with the timeout error:

19:00:57.339 detox[23402] ERROR: Exceeded timeout of 300000ms while handling jest-circus "setup" event
19:01:01.021 detox[23404] ERROR: Exceeded timeout of 300000ms while handling jest-circus "setup" event
19:01:01.022 detox[23403] ERROR: Exceeded timeout of 300000ms while handling jest-circus "setup" event

2. Run xcrun simctl list to confirm that detox created the new devices:

    iPhone 13 (72E2C79D-E868-4657-A649-D975F0F11DDA) (Booted) 
    iPhone 13-Detox (DC469F9F-576A-408A-90F5-4E2160A0F425) (Booted) 
    iPhone 13-Detox (F74D9F6D-8288-45C8-B10E-70EA634F5B06) (Booted) 

3. Run the full suite of tests and they now all pass

If there is a more elegant way to preemptively create the simulator devices for the number of workers I intend to use (usually only 3 because github actions macOS containers run on boxes with only 3 cores), I can do that instead as a workaround until I find a better solution.

[d4vidi]

@zacharyweidenbach Thanks for sharing those insights! We seem to have drifted far away from the issue's topic. Could you please report your CI logs (running with --loglevel trace) in a dedicated issue? Thanks!

[zacharyweidenbach]

@d4vidi
Here is a link to the new issue per your request:
#3574

[jonathanmos]

@capezzbr @MoOx @rikur @gabrieldonadel @zacharyweidenbach @ghitier @jd20 @dklymenk @BrentWardOx @sangliquidus I've released a fix for this issue in the latest detox version (20.0.3) - please try it and see if this issue is now solved for you