Detox as a platform

[talkol]

There has been some discussion about third party tooling to be built on top of Detox. Core Detox features like device control, commands via websocket and native synchronization are useful for a large family of tools and isn't limited only for E2E tests.

Some examples for third-party tools

• Fructose is a tool by @rjanjua for functional testing of components on device
• @panarasi from Microsoft suggested creating an integration testing framework on top of Detox
• @Gongreg is thinking about snapshot testing with screenshots for components on device

Current Detox architecture and limitations

When using Detox, we have 2 different machines running side by side: the tester (node.js running mocha with your test scripts) and the testee (the app running on a device). Communication between the 2 machines is done via websocket.

JS code running on the tester (node.js) can have different behaviors across test scenarios. For example, if I have a mock server running on node.js, which is supposed to return A in one scenario and B in another scenario, I have no problem with configuring the server on the beginning of each scenario for the behavior I want.

JS code running on the testee (app) can't currently have different behaviors across scenarios. Since we compile and bundle the app once in the beginning of the entire suite, we can't change code between scenarios. If we want the app to behave differently in different scenarios, we can only control it differently with Detox and tap on different buttons. This is a limitation. It makes sense with E2E tests, but it doesn't make sense for the other tools people are thinking about.

Mechanism for dynamic JS evaluation on device

One way to overcome the above limitation is to introduce API to Detox that will allow changing the JS code running on the testee (app) dynamically during a test scenario. This API will basically allow the tester (node.js) to evaluate JS code on the testee (app).

Let's assume the API will look like device.evaluateJS('../device/file.js'). It will run on the tester (node.js) and take a local JS file found on the tester, read it into a string and send the string via the Detox websocket to the testee (app). On the app, the Detox native framework will take the JS string and evaluate it on the current React Native JS context.

What about require()? If this code needs to rely on other libraries currently loaded in the React Native JS context, like react-native itself, we're going to have problems requiring them due to name changes in the bundling process. We can overcome this difficulty by running react native bundler in the unbundle mode.

Third-party tools can rely on this mechanism to manipulate the behavior of the app running on device with Detox during runtime. For example, inject the various scenarios to run on device and get integration tests instead of E2E tests.

Communication between the app and outside the app

If we have code originating from the tester running on device, it should have a way to communicate back with the outside world - meaning with other code running on the tester (node.js). This communication can be done over websocket. We can use the Detox websocket or create a new dedicated one from the JS code we inject. The standard approach could be to have the injected code start a message loop and listen for incoming commands from the tester.

Feedback

Please write feedback about this feature in this thread. Do you have alternative ideas to turn Detox into a platform for third-party tools?

[DanielMSchmidt]

Hey, great that you opened this, was thinking about writing something similar, actually.

I would like to give one think to consideration. We could have a plugin API and provide the soon-to-be auto-generated bindings for EarlGrey as a separate package, so that you might extend detox for example with screenshots in your own plugin. That would allow us to keep the API surface small and concise while allowing others to build exactly the features they need.

[rjanjua]

JS code running on the testee (app) can't currently have different behaviors across scenarios. Since we compile and bundle the app once in the beginning of the entire suite, we can't change code between scenarios. If we want the app to behave differently in different scenarios, we can only control it differently with Detox and tap on different buttons. This is a limitation. It makes sense with E2E tests, but it doesn't make sense for the other tools people are thinking about.

In Fructose I've created a withComponent() method in which you declare your react-native component as you would in an app:

withComponent(<Text testID='fructose'>Fructose!</Text>, "description", () => {
  // tests go here
})

This method has 2 implementations, one for the 'tester', and the other in the app. When the tester encounters withComponent() it creates a unique id (in this case a json string of the component) from the component and fires off a load request to the app (using websockets).

With the app it is slightly different. Following a model similar to react native storybook, all the components are preloaded into the app (I am using react-native-storybook-loader for this). Any file ending in a matching a definable glob (e.g. components/**/*.fructose.js) gets loaded. When the app encounters withComponent() it stores the components to an object, creating a unique id as the key and the component as the value. When it receives the request to load a component, it loads up the component that matches the key.

We've used it for a bunch of components in different configurations and it's working really well so far.

Also the repo has now moved to https://github.com/newsuk/fructose/

[rjanjua]

Just a thought, would dependencies present an issue here?
Say you require a certain dependency to run your script, however unless you've already got that dependency in the app beforehand it won't be able to run the script.

[JAStanton]

Brain dump on communication between tests and app, with some possible solutions:

Right now I am running UI tests using Xamarin Ui Test, it's the successor to Calabash. They have a concept of a backdoor method, Backdoors are methods that can be invoked during a test run to perform some special action to configure or set up testing state on a device. In the test it looks like: app.Invoke("MyBackdoorMethod"); in practice I use it like so: app.Invoke("backdoor", { method: "fooBar", args: { baz: "bat" }});, and in react native I have:

import { NativeEventEmitter, NativeModules } from 'react-native';
const { BackdoorManager } = NativeModules;
const backdoorManagerEmitter = new NativeEventEmitter(BackdoorManager);
backdoorManagerEmitter.addListener('BackdoorEvent', this._onBackdoorEvent);

and my _onBackdoorEvent parses the response and executes whatever function, w/ whatever args I need to successfully run.

Where this falls down is that the communication is one way, I would love to be able to return values back from the app to my tests, I think it's possible, and it's going to critical I do so because right now my tests are all fire and forget so I don't even know when my backdoor finishes, which makes tests unpredictable.

All that said, I think Detox can do something similar. I could see all the kinks being ironed out and then executing something like:

  it('should have welcome screen', async () => {
    const user = await device.backdoor('loginWithUser', { user: 'joe', pass: '123' });
    await expect(element(by.id('dashboard-scene'))).toBeVisible();
    await element(by.text(`Welcome ${user.name}`));.toBeVisible();
  });

That would be a workable solution I believe, however my ideal would be to have the JS for my tests actually running on the device in the same env as my other JS, so I could do something like:

import { UserService } from '../services';

describe('Example', () => {
  it('should have welcome screen', async () => {
    const user = await UserService.LoginWithUser('joe', '123');
    await expect(element(by.id('dashboard-scene'))).toBeVisible();
    await element(by.text(`Welcome ${user.name}`));.toBeVisible();
  });
});

That way I have access to the same functionality my app does and I could have full and total control of environment.

[JAStanton]

I'm digging around learning a bit more how you guys work, it seems like you're already sending messages back and forth between either native or js via websockets. Do you think you could setup a JS listener to send any async message back and forth between client/server? The only thing preventing me from using this library is a way to arbitrarily pass messages back and forth so I can tell my app to do backdoor things, such as setting up the state of the app in preparation for a test.

[jeduden]

@talkol @LeoNatan would you accept a PR with the following extensions:

• expand the iOS websocket to allow multiple delegates (instead of just one)
• public method to register custom websocket delegates ?

[Salakar]

In need of this as well for the testing suite re-write of https://github.com/invertase/react-native-firebase that I'm working on - all the tests there need running inside the react native JS environment - mocking the vast amount of native modules required by it is just not feasible.

However, after some digging around in the internals I have managed to get communication via the app and outside the app going with minimal code and no modifications to detox code - it's not the greatest solution though... 🙈

---

Here's a rough code sample from Android (same applies to iOS with only small changes to the invoke target config)

Manually init detox

Need to do this to have access to the client property from detox internals.

const Detox = require('detox/src/Detox');
const config = require('../package.json').detox;

// ... 
const detox = new Detox(config);
await detox.init();

Create a custom Invocation Action

Detox internally calls native java/obj-c methods via it's Targeted Method Invocation actions, e.g. scrollToEdge action - as you can see in that example a specific class and it's method is named to be invoked with the specified args.

Custom action code

// custom action class
class Invoke {
  constructor(type, params = {}) {
    this.type = type;
    this.params = params;
  }

  // needs to be here for detox
  expectResponseOfType() {
  }

  // the only way to retrieve data back from an invocation currently
  // is if it's sent back as an error - non errors are hard coded
  // in detox to just return "(null)", see link:
  // https://github.com/wix/detox/blob/master/detox/android/detox/src/main/java/com/wix/detox/DetoxManager.java#L106
  async handle(response) {
    response.type = 'bar';
    response.params = JSON.parse(response.params.error);
  }
  // ^-- detox internally calls this after every response - we just modify the result not to
  // be an error by taking the error message and JSON parsing it.
}

const action = new Invoke({
  target: {
      type: "Class",
      value: "com.example.Foo"
  },
  method: "bar",
  args: [{
      type: "Integer",
      value: 123,
  }],
});

const result = detox.client.sendAction(action);

And the named Java module above:

package com.example;

import org.json.JSONObject;
import com.facebook.common.util.ExceptionWithNoStacktrace;

import java.util.Map;
import java.util.HashMap;

public class Foo {

    // just a test function to see it all working
    // has to throw an error with the JSON as the message as non error values
    // don't get sent back over the web socket - see comment on Invoke class above
    public static void bar(int intTest) throws ExceptionWithNoStacktrace {
        Map<String, Object> data = new HashMap<>();
        data.put("test", "foo");
        // pass the value back just to test
        data.put("id", intTest);
        JSONObject json = new JSONObject(data);
        throw new ExceptionWithNoStacktrace(json.toString());
    }
}

---

That was my work around anyway. As you can see it's probably not the best but it's a solution that seems to work ok for me nonetheless.

Based off that; as a PoC I created a way of reading the tests that are inside the RN environment and from them creating a stub test for each of them in the nodejs/mocha environment that internally just calls the RN test via a custom action.

Wish there was a better more official way of doing this.

Edit, see: #551 (comment) - have come up with a 'better' way.

---

My suggestions for initially approaching the topic of this issue would be:

• Open up the Invocation Manager to support custom actions properly - small change as far as I can tell - happy to PR if agreed
• Allowing native code and JS code to interface into the existing web socket connection
  • Perhaps Detox could let all ws events that it's unaware of to just fall through its code untouched - allowing custom code to pick these up.

[mujavidb]

@JAStanton Would it be possible to drop some of the logic in a gist?

[LeoNatan]

Closing old issues. This is out of scope for the project.