Metro watcher doesn't trigger in time for files created during Babel transpilation

[tjzel]

I'm changing the way how the Babel plugin works in react-native-reanimated/react-native-worklets. These changes include creating a new file, that's immediately required in the newly transformed code.

To exemplify, let's say I have a file:

// file.js
function foo() {
  function bar(){};
  bar();
};

I'm transpiling it to:

// file.js
function foo() {
  const bar = require("react-native-worklets/generated/1111111.js").default;
  bar();
}

Creating respective file before manipulating the original AST:

// "react-native-worklets/generated/1111111.js"
export default function bar(){};

Such flow causes Metro to fail to resolve "react-native-worklets/generated/1111111.js" because the watcher triggers too late. From my debugging the flow is as follows:

1. file.js is picked up by Metro for Babel transpilation.
2. "react-native-worklets/generated/1111111.js" is created.
3. AST is manipulated by Worklets Babel plugin.
4. new AST is traversed by Babel, require("react-native-worklets/generated/1111111.js") is discovered.
5. Metro attempts to resolve this file using this method. It wasn't present in its filesystem snapshot, so it fails to resolve it.
6. Metro filesystem watcher kicks in here and adds an entry for "react-native-worklets/generated/1111111.js" using this method.

I understand this might be problematic to fix quickly since the issue spans towards the Watchman itself. I wonder if there's some elegant way to trigger the discovery event earlier so the resolution would succeed.

[robhogan]

These changes include creating a new file

We spoke offline, but just to add some context for this issue - creating files during transformation is not supported or recommended. Babel plugins must do no more than deterministically mutate an AST - they should never read or write from disk.

The limitation you're coming up against - that the new file event arrives too late - would always be a race condition, in any build system. Writing and then immediately reading a file from a different process is unreliable on an operating system level. You're going to hit EPERM errors trying that on Windows and could end up reading partial files.

Metro's watcher guarantees eventual consistency, but has inbuilt delays both for batching and to allow for files to be completely written before we try to read them. It's intrinsically asynchronous, whereas Metro resolution is synchronous (and even if it weren't, we wouldn't want the resolver to have to wait idle).

Virtual modules

I believe the feature that would support your use case would be virtual modules - the basic idea would be for your transform to inject a dependency into the AST that encodes the information you would've used to create the file - but instead of creating the file on disk, the resolver can create it in memory from the encoded information.

Metro doesn't support that yet either but that's definitely something we want to support soon - the implementation of require.context is along similar lines. It'd be great to look at your design end-to-end and come up with something we can build towards.

[tjzel]

@robhogan I was hacking a bit into the Metro bundling process and I figured out that using output metadata of Babel transpilation would be a convenient way of outputting virtual modules, I can put in there virtually anything Metro needs. It's very simple on my end, however trying to use it in Metro's complicated pipeline is a bit more difficult. Do you think the metadata approach is a reasonable one? metadata is already used for some functionalities https://github.com/facebook/react-native/blob/main/packages/react-native-babel-transformer/src/index.js#L227

[robhogan]

Could you maybe sketch something up that explains what you're doing and the architecture, big picture (RN side too - we have folks interested 🙂)?

It's tough to give any guidance when we're only seeing little pieces, but I'd love to see this built right.

[robhogan]

(Returning metadata from a pure transform is OK, but we don't ever use that to add nodes to the bundle and the resolver doesn't have access to it - so it might be a viable approach - it depends)

[tjzel]

Sure, let me outline the big picture here.

I'm trying to redefine the internals of how our multithreading JavaScript model, "worklets", work in "react-native-reanimated" ("react-native-worklets)". The key idea is to integrate closer with Metro to reduce the overall amount of hacks we do and increase performance.

Since JavaScript is not thread safe, for multi-threading we spawn additional JavaScript runtimes, executed in separate threads. The question is: how to get a function from runtime A to runtime B and execute it there? The function is bound to the runtime A and cannot be extracted in any way without external tools. This is achieved by Babel transformations.

Say we have the following file, which would be executed on runtime A:

function foo() {
  'worklet';
  return 1;
}
invokeOnRuntimeB(foo);

We transform this file to (simplified):

const foo = (() => {
  function foo_() {
    return 1;
  }
  foo_.code = "function foo() { return 1; }"
  return foo_;
})();
invokeOnRuntimeB(foo)

Thanks to this, we obtain the JavaScript code of a function, available in runtime. Then, when runtime A wants to invoke foo on runtime B, we evaluate code on runtime B.

This approach has many flaws - the key one here is that the user cannot use any 3rd party libraries like this, since there's no way of getting their code in runtime. To solve that problem, instead of injecting the functions code into runtime, we want to consume the bundle Metro is providing.

Say we got the bundle. The question here is, if a runtime A wants to invoke foo on runtime B, how do we get it from the bundle? We could add an export directive to foo and simply require it, but the file containing foo can have fatal side-effects. For instance it could instantiate whole React Native on another runtime, which is definitely something we don't want. Instead, we create a separate file for foo and require it in the original file.

function foo() {
  'worklet';
  return 1;
}
invokeOnRuntimeB(foo);

becomes

const foo = require("react-native-worklets/generated/111111.js").default;
invokeOnRuntimeB(foo);

and 111111.js is created:

// 111111.js
export default foo() {
  return 1;
}

Thanks to this, when we invoke foo on runtime B, we can simply require foo from 111111.js on runtime B and execute it, free of any side-effects - just like web workers. Moreover, if foo uses any 3rd party library:

import {bar} from '3rd-party-library';

function foo() {
  'worklet';
  bar();
}
invokeOnRuntimeB(foo);

We can preserve that import since we have the access to the whole bundle, including 3rd-party-library:

// 111111.js
import {bar} from '3rd-party-library';

export default foo() {
  bar();
}

The problem is that these newly created files won't be picked up by Metro instantly. This isn't fatal unless we want hot reloads and generally good DX. Therefore we need a way to somehow register the "virtual module" 111111.js during the Babel transpilation, so if require("react-native-worklets/generated/111111.js is encountered by Metro it would pull that file from its virtual module cache.

That's more or less the big picture. I tried not go into the implementation details, like closure capturing and the scheduling model, since they are agnostic to the bundling process.