Running CTS on React Native

[wcandillon]

We have built a WebGPU binding for React Native based on Dawn and we also have a small test suite for it which we run against Chrome as well for reference.

We are now looking to run the CTS as well. I tried to bundle this module but been struggling in a few places.

So I wanted to try something simpler manually to get a sense on how things work.
I was able to load a TestGroup and it contains a few tests.
I couldn't find which APIs I need to use just to execute this single test. Could you give me some pointers?

I will still try to see if I can find a way to package CTS in a way that would fit React Native but if there is a minimum set of API I could use directly where I could build my own test loader, that would be great.

[greggman]

@kainino0x probably knows best but might be out for a few weeks.

I have zero experience with React Native.

In any case, there are multiple ways to run the tests. One way is via standalone.ts. Another is via cmdline.ts. You could look at both of those to see how to write your own runner.

As for what features, off the top of my head, most of the tests don't need more than JavaScript and WebGPU but there are tests, especially in the web_platform folder, that test interaction with WebGL, Canvas2D, ImageBitmap, ImageData, HTMLImageElement, HTMLVideoElement, HTMLCanvasElement, VideoFrame, OffscreenCanvas, requestAnimationFrame and I'm sure a few others

Also, maybe take a look at how dawn.node runs the tests

https://dawn.googlesource.com/dawn/+/refs/heads/main/src/dawn/node/README.md

You can see how it calls into the command line version here to get a list of tests

https://dawn.googlesource.com/dawn/+/refs/heads/main/tools/src/cmd/run-cts/common/cts.go

I didn't dig to see how it runs them.

IIRC dawn.node fails or skips tests under web_platform since I think none of those are implemented in node.

[kainino0x]

I think Gregg covered everything! I also do not know anything about React Native.

If React Native's runtime is Node or something similar, then looking at cmdline.ts is probably the best place to start. That has already been used to run WebGPU tests in both Dawn's Node bindings and Deno's bindings for WebGPU.

You should be able to run the whole suite; all tests (regardless of subdirectory) that have dependencies on other APIs, like DOM and browser APIs, should automatically skip themselves when those aren't available.

Would love to hear updates on your progress! Feel free to contribute changes to this repo if needed (but they might not be, since it already works in Node/Deno).

[wcandillon]

@kainino0x Thank you for your note. It will be quite a bit of work for us as we will need to do a custom bundle but I'm sure we can get there eventually. Lately any examples we throw at the module work immediately out of the box so we didn't feel this is priority yet but I'm sure this will come up eventually.

Here are demos from the WebGPU samples:

github3.mp4

And then we tried to run Three.js on top of it and it kind of worked out of the box:

threejs.mp4

But I definitely want to revisit this topic.

[kainino0x]

That's very nice, thanks for sharing!

[hmallen99]

Hey folks! I have been able to get the src/webgpu subset of tests running for React Native on a fork of the CTS: https://github.com/rebeckerspecialties/gpuweb-cts. I had to modify some functionality in the file loading logic to get it working.

The main challenge for getting the tests running with React Native is bundling the tests in the React Native app, as mentioned above. Since React Native does not currently allow dynamic imports, I went ahead and included all of the test files in the JavaScript bundle.

Metro (the JavaScript bundler for React Native) has an experimental require.context feature that allows us to import a bundled file given a file path at runtime, making it possible to support the expected behavior in DefaultTestFileLoader. From there, I was able to copy some of the setup from standalone.ts into a React Native test app and get the tests running on device. There are a few other changes I had to make to support .spec file imports in this PR: https://github.com/rebeckerspecialties/gpuweb-cts/pull/1/files.

In my current fork, I just overwrote the existing crawl.ts and file_loader.ts implementations, but it should be possible to create a standalone React Native test runner within the main cts repository that is compatible with the node.js and chromium test runners.

I would be happy to contribute that in the future, once I get the stress tests and unit tests running and clean up the import logic. This approach would mainly consist of creating a new class that implements TestFileLoader called ReactNativeTestFileLoader that would be used only in the React Native app. Then I would need to add some polyfills for fs and path to mimic file loading for the metadata and listing files.

Let me know if you would be interested in this approach! An alternative would be to implement a dev server that sends the test logic to a client device, but that would likely require more work.