Does not work in stackblitz

[daKmoR]

Using @parce/watcher in stackblitz result in the following error.

Error: not implemented
    at _0x3b9429 (https://node-igrvzm.w.staticblitz.com/blitz.d64d45524a435df425e7813603b52872ecfbe08e.js:15:7121)
    at process.dlopen (https://node-igrvzm.w.staticblitz.com/blitz.d64d45524a435df425e7813603b52872ecfbe08e.js:15:187719)
    at Object.Module._extensions..node (https://node-igrvzm.w.staticblitz.com/blitz.d64d45524a435df425e7813603b52872ecfbe08e.js:6:191058)
    at Module.load (https://node-igrvzm.w.staticblitz.com/blitz.d64d45524a435df425e7813603b52872ecfbe08e.js:6:188760)
    at Function.Module._load (https://node-igrvzm.w.staticblitz.com/blitz.d64d45524a435df425e7813603b52872ecfbe08e.js:6:186300)
    at Module.require (https://node-igrvzm.w.staticblitz.com/blitz.d64d45524a435df425e7813603b52872ecfbe08e.js:6:189078)
    at i (https://node-igrvzm.w.staticblitz.com/blitz.d64d45524a435df425e7813603b52872ecfbe08e.js:6:443011)
    at _0x4248b4 (https://node-igrvzm.w.staticblitz.com/blitz.d64d45524a435df425e7813603b52872ecfbe08e.js:15:95063)
    at load (/home/projects/node-igrvzm/node_modules/node-gyp-build/index.js:21:10)
    at Object.eval (/home/projects/node-igrvzm/node_modules/@parcel/watcher/index.js:1:117)

Reproduction

1. Open https://stackblitz.com/edit/node-igrvzm?file=index.js
2. Execute node server.js in the terminal

Expected

Watch works on stackblitz

[devongovett]

This is a native module with deep integration into operating systems. I'm not sure how it could work in a browser environment since there is not even a file system. What are you trying to do?

[daKmoR]

I have a static site generator that rebuilds and reloads on file change. For that I am using @parcel/watcher and it works amazingly 👍

I never expected it to work in stackblitz... but apparently stackblitz become really really good at simulating a file system and most packages seem to just work 😅

How I got the idea that it works is by looking at astros stackblitz examples. It's a static site generator that uses chokidar via vite. And it works in stackblitz.

So it seems there is a way to make it work... is that something @parcel/watch would like to support? 🤔

If so then I could dig a little deeper

[schickling]

I'm very interested in using @parcel/watch in some of my projects (which are currently using Chokidar). For some of those projects it's important to me that they also work on Stackblitz - for those I'll hold off switching over to @parcel/watch for now until support has landed.

[mischnic]

Can someone explain how deeply Stackblitz emulates the usual Node environment (and which parts)? If there are no primitives for watching file changes, it will have to use polling.

[EricSimons]

Hey all- Eric from StackBlitz here, happy to provide some context.

StackBlitz uses WebContainers (a WebAssembly-based operating system) to run Node.js inside your browser tab. This version of Node.js is pulled from Node.js core itself (w/ some minor tweaks) so it has broad support for nearly all Node.js APIs. Importantly, WebContainer includes a UNIX compliant filesystem, so all Node.js FS APIs you would expect (including synchronous operations, watching, etc) work exactly as they do on local.

The main limitation of this approach is that native addons must be available as Wasm binaries otherwise they will not work. This seems to be the issue happening here for the @parcel/watcher package. Shipping Wasm binaries alongside native binaries is the direction most packages in the ecosystem are heading towards, and is the impetus of folks like Next.js/etc pushing for the npm Package Distributions RFC. The ability to restrict native binary addons in Node.js is also important from a security PoV as progress is made towards complete sandboxing of Node.js code execution.

For filesystem operations in StackBlitz/WebContainer, you can use the normal fs.watch and fs.watchFile from Node.js' core fs module. This is what Chokidar uses under the hood when running on non-Darwin platforms. Perhaps @parcel/watcher could do something similar in this regard, where the Node.js fs events could be normalized, handed to your Wasm module, which then emits the appropriate results to listeners?

[mischnic]

Thanks @EricSimons!

So what would be needed is something like the existing BruteForceBackend (which already handles the "offline" watcher.getEventsSince even across program invocations) but additionally supporting live watching via fs.watch* as opposed to just throwing:

watcher/src/shared/BruteForceBackend.hh

Lines 12 to 18 in caf372c

	void subscribe(Watcher &watcher) override {
	throw "Brute force backend doesn't support subscriptions.";
	}
	void unsubscribe(Watcher &watcher) override {
	throw "Brute force backend doesn't support subscriptions.";
	}

And generally some boilerplate/a build step for an Emscripten interface as opposed to Napi will be needed.

[devongovett]

Does that mean that we would have to implement a completely different interface for this in JS? WASM doesn't have file system access afaict, so I'm not sure how this could be made to work. The whole point of this module is to avoid node's fs.watch, which has been quite problematic for us in the past. Maybe I'm missing something, but seems like if you want to use fs.watch you should just use that? What would a wrapper around it be providing?

[devongovett]

@EricSimons since you are here, I have questions. Basically, I am very confused about what a WebContainer is exactly. All of the marketing material kinda glosses over the details, and hand waves a bunch about how it works. The repo doesn't really give much more information. Is there a more technical overview available that describes the exact interface that is available and some of the implementation details?

• Is it shimming the Node.js API, or something lower level?
• For WASM, what POSIX APIs (if any) are available? What NAPI apis are available? libuv?? You describe a WebContainer as a "WebAssembly-based operating system", but what is the API?
• You mention that it's running Node.js "natively", but what does this mean exactly? Are you literally compiling Node.js to WASM (including V8??) and running it in the browser? Or is it actually just a shim of the API? Something else?
• Can you only run Node within a WebContainer, or do other processes work? What if we implemented a CLI entirely in Rust for example?
• For file watching specifically, what is the native API? There is no standard POSIX API for file watching, each OS implements their own (e.g. Darwin's FSEvents/kqueue, Linux inotify, etc.). So if a WebContainer provides such an API, what is it? Or is it only available to JS via a Node.js fs.watch shim?

Any details you can provide would be helpful. At the moment I really don't have any idea what a WebContainer actually is, other than some kind of magic that somehow runs Node or something like it in a browser. As we move more of Parcel away from JavaScript and more toward native Rust modules, having an understanding of this would be helpful.

[EricSimons]

Is there a more technical overview available that describes the exact interface that is available and some of the implementation details?

Beyond what's in the readme and the blog post not really, as it hasn't been an issue to date. I think the parcel filewatcher might be a unique case simply because it's purpose is a very specific syscall level integration, whereas other projects (like next.js/swc/etc) have implementations that are a bit higher level and/or syscall agnostic.

Is it shimming the Node.js API, or something lower level?

Lower level, largely the syscall layer where Node.js interacts with the host OS.

For WASM, what POSIX APIs (if any) are available? You describe a WebContainer as a "WebAssembly-based operating system", but what is the API?

The shell in WebContainer exposes a lot of the usual POSIX built-ins (ls, cd, cat, etc). Beyond that, the only interfaces to the Wasm OS today are exposed via the Node.js API surface. The reason for this is that we do not intend to expose a non-standardized WebAssembly system interface to end users, largely because WASI already exists. However, WASI isn't yet capable of providing all the syscalls we need to run Node.js wholesale on Wasm, but bridging this gap is one of our primary objectives in being a part of the Bytecode Alliance.

For Wasm applications that need syscall access, in the near future the solution would be leveraging Node.js' implementation of WASI. We haven't enabled access to this in WebContainer due to some upcoming changes I can't speak to yet. For the time being, you can instead leverage the Node.js API surface to perform syscalls. This is similar to how emscripten compiles down to the Node.js APIs for filesystem, networking, etc.

What NAPI apis are available?

None. WebContainer is an operating system designed to run inside a browser engine, and therefore is subject to the same limitations as any other browser context: if you want execute code from a language other than JS, it needs to be precompiled as a WASM.

There has been work from Node.js core on enabling Wasm support w/ NAPI but AFAIK has not yet come to fruition. Last I heard there was interest in WASI to help bridge the syscall gaps, but it needs to be sufficiently matured before it could be viable.

Can you only run Node within a WebContainer, or do other processes work? What if we implemented a CLI entirely in Rust for example?

Node.js is the only runtime we ship with WebContainer OOTB atm, but any language/runtime provided as a Wasm module can also run inside WebContainer today.

If you have an entirely Rust-based CLI that compiles to a Wasm module, the only thing you'd need to do is wrap it with some JS code that wires it up to the interfaces it requires (command line args, fs, network, etc).

For file watching specifically, what is the native API? There is no standard POSIX API for file watching, each OS implements their own (e.g. Darwin's FSEvents/kqueue, Linux inotify, etc.). So if a WebContainer provides such an API, what is it? Or is it only available to JS via a Node.js fs.watch shim?

Yes, only available to JS via Node.js' fs.watch/etc until WASI support is landed.

As we move more of Parcel away from JavaScript and more toward native Rust modules, having an understanding of this would be helpful.

Totally. I think the main thing is to include Wasm as a compile target alongside your existing native addons. This does require a little bit of extra work in loading those native bindings- here's how Next.js does this w/ SWC as an example.

Another quick way to test if this will work for existing (and future) sandboxed Node.js environments is by passing the --no-addons flag to any Node.js command. I think @padmaia was using this when developing the Next.js SWC Wasm and might have additional color.

[devongovett]

Sure, so for @parcel/watcher, I'm not really sure we even need a WASM module, we'd just implement it in JS. The C++ code in this module is pretty much providing low level OS bindings which don't exist in a WASM context, and we'd need to bridge them back to JS anyway. At that point, you might as well use fs.watch et al directly.

I suppose the benefit of including a JS wrapper here would be API compatibility, so you could use the module unmodified both natively in Node and in the browser. Perhaps there is a way to detect if native modules are supported, and if not, load the JS version?

[EricSimons]

Yeah I think that makes sense. It looks like Next.js does a try-catch to determine if the runtime can load the native addon, and if not then loads the Wasm version. I imagine you could use the same approach here. Alternatively there might be a more static way of analyzing this... cc @d3lm who might know more on this front.

[devongovett]

Yeah try/catch seems fine.

To be 100% transparent, I do not personally have time to work on this anytime soon. But, if someone wants to do it, please feel free to send a PR! Basically we'd expect a JS implementation of the same API exposed by the module currently, but written in JS using the Node fs APIs, as a fallback in case loading the native module fails.

[d3lm]

Hey ya all 👋

Maybe I can chime in on some of this stuff.

I think the solution that has been discussed here, a pure JS implementation that simply uses Node's fs module, sounds reasonable to me. Compiling to WASM and providing the syscalls via imports is just overhead which can be avoided. Also it means data constantly has to be copied from and to linear memory.

Yeah try/catch seems fine.

@devongovett Yep, you can use a try/catch or you leverage conditional exports. Loading a native addon will fail in WebContainer because they are disabled by default (via the new CLI option --no-addons). This also gets inherited to worker threads and child processes. If you try to load a native addon while it's disabled you'll get a descriptive error.

The other option would be to use conditional exports. In your package.json you could define your entry point to use a different one if native addons have been disabled. For example:

{
  "name": "@parcel/watcher",
  "exports": {
    ".": {
      "node-addons": "./index.js",
      "default": "./no-addons-main.js",
    }
  }
}

With this in place, it would use a different entry point (./no-addons-main.js) when native addons are disabled via --no-addons and otherwise load index.js, which can be the entry point that relies on native code.

In this particular case, it's recommended to treat the default condition as the progressive enhancement for environments where native addons may not be supported (e.g. the browser).

I would personally recommend the approach using conditional exports because it's good practice to define a clear interface that is exported as well as taking different environments into account.

[samdenty]

in #134 i've added support for a chokidar backend 🎊

[devongovett]

There is now an alpha version of a WASM version available in the @parcel/watcher-wasm package. Docs. I verified that all of the tests pass in the Stackblitz environment. I think the Stackblitz team should now be able to make this work as an automatic fallback for @parcel/watcher as they do for lightningcss and esbuild's wasm versions.

[pi0]

Thanks @devongovett you are a lifesaver! Just integrated parcel/watcher with unjs/listhen experimental watcher and this solved issue with stacklbitz (also my general concerns regarding bundling issues. FYI, we use this package for Nuxt and Nitro CLI which are zero dependency. Not yet the watcher).

One question: Other than being experimental, do you know any downsides of using wasm build in Node.js environments?

[devongovett]

I would not recommend using it except in unsupported environments (like browsers). It is significantly less efficient than the native watcher implementations that deeply integrate with the operating system. It must recursively traverse the directory hierarchy and watch each nested directory separately, whereas most of the native implementations use operating system APIs that avoid the need for this.

As an example of the difference, the Nuxt team tested their previous implementation using chokidar (which also uses node's watcher API under the hood) and the native Parcel watcher and saw startup time decrease from ~30s to 300ms nuxt/nuxt#20179 (comment). That's probably almost entirely due to not needing to walk the entire directory heirarchy, which can be avoided by using better OS APIs.

So unless you're running in a browser or on an unsupported operating system, I'd keep using @parcel/watcher. You could also implement a fallback where you use @parcel/watcher-wasm automatically when building for the browser and @parcel/watcher otherwise. I believe Stackblitz itself will start doing this automatically soon as well, so using @parcel/watcher there would begin working with no changes on your side.

[pi0]

Gotcha thanks for mentioning the recursive scanning difference. (maybe docs could include this note?)

For listhen, i will use try/catch fallback to unblock stackblitz and leverage full performances on local 👍🏼

As I mentioned, the main other concern is for the nuxt CLI (nuxi) standalone watcher (which is zero dependency and bundled with unbuild). It currently has fsevents as an optionalDependency. I will try to come up with some experiments on nuxi next gen how to leverage parcel watcher with native node module.

[devongovett]

This should now work in Stackblitz as of v2.3.0! https://stackblitz.com/edit/node-bpupxc?file=index.js