[question] Interest in a ChokidarWatcher?

[ccheever]

Hi-

I wrote a mostly complete ChokidarWatcher for sane.
I can see reasons why you wouldn't want this to be integrated into the main sane, (npm installing the native modules chokidar depends on can get hairy on certain platforms), but there does seem to be a bunch of interest in combining or having these projects work together, so I wanted to float the idea out there.

The code is here:
https://github.com/exponentjs/sane

The ChokidarWatcher isn't entirely complete; it can only handle one glob at a time. I can fix that but I don't need it for my own use, so I may not bother unless there is interest in the ChokidarWatcher in the main project.

I don't have a particular agenda or anything that I want this project to do -- just wanted to let you know that a mostly working chokidar backend is out there now in case that leads anywhere good.

[amasad]

This is cool. But, the native module is the issue. Can I know your use case?

[ccheever]

I'm working on a project to make the react native packager much easier to use for casual developers--starting with OS X but eventually Windows and *nix, and one big issue I've observed in user testing is that people end up with one of these problems:

(a) they don't correctly install watchman, or have an old version of watchman around that doesn't work properly (yields a cryptic error about relative roots)
(b) they want to npm install one small module (ex. moment) to try that out, and sane-sans-watchman can't handle it (maybe because the open file limits are so low on OS X? I'm not 100% sure why but could not figure out any easy fix for it)

So, I decided to make a chokidar backend. I plan on distributing the binary already compiled just for relatively recent Macs, rather than having people npm install stuff themselves.

And I wanted to use sane instead of just chokidar by itself because I want to make my fork of the react-native packager as minimally different from upstream as possible--or possibly even not a fork if that can be achieved--just because I don't want to spend my time maintaining that delta.

I played around with trying to run a second watchman daemon with a specific version that I would install for people, but I couldn't get 2 watchman daemons running on the same machine without having them occasionally interfere with each other (it may be possible but its certainly a bit tricky), and so the next best option seemed to be to make a ChokidarWatcher for sane.

I haven't thought this through completely, but maybe the chokidar watcher could be put in its own npm package, something like sane-chokidar-watcher, and if that was included as a peer dependency, and you specified an option like {chokidar: true}, it would only then pull in the ChokidarWatcher. That way I wouldn't have to fork sane, and sane could use Chokidar while still presenting the same old sane interface for people who wanted that (like I do for this react native packager stuff).

[stefanpenner]

@amasad it seems like we need to think about about making it possible for watchman to be installed by casual users. Although maybe funky, it could be installed (if not already present and running) in a postInstall hook. If it is already running, but an old version the tool that uses it could inform \w instructions on how to upgrade. etc.

cc @wez

[ccheever]

@stefanpenner @wez I would be interested in an easy-to-install version of watchman, but I think it might be hard to get right.

There are 3 cases this would have to cover my use case:
(1) the case where the user hasn't installed watchman of any sort yet.
(2) the case where the user has a relatively recent version of watchman installed that will work just fine
(3) the case where the user has an old version of watchman that won't work well with the react native packager.

Since I'm only targeting OS X to start, I think its probably reasonable to handle (1) in a relatively straightforward way. I can just make a build of watchman, maybe statically link some of the dependencies like libpcre in in case those aren't installed, and then include that in the .dmg I produce.

For (2), I think we could just lean on the already installed watchman and that would probably work just fine, esp. since the watchman docs indicate that the API will always be backwards compatible.

For (3), I think its really tricky. watchman might have been installed by brew, but if so, I don't want to perform brew upgrades on behalf of the user since that could be surprising or unwanted. And if watchman was installed through some other mechanism -- ex. if someone built it himself/herself -- then there may be no clean way to upgrade it.

To get around this, I played around with trying to get two watchman daemons running on my own laptop simultaneously -- one installed through brew and one I'd built myself using a different sockname, statedir, and conf file, and that was a newer version. But I couldn't get the daemons to not step on each other's toes. Ex. Running watchman version on one would cause a packager instance listening to the other to disconnect/reconnect. This may be solvable but I'm worried that getting it right will always be an uphill battle since it cuts against the grain of what watchman is trying to do -- be a singleton daemon that takes care of all your file watching needs on a machine and manages the limited resources around that in an efficient way.

This is how I ended up looking at chokidar. Even though its trickier in that it has to build this native module upon install (which from what I've read is optional, but without it, chokidar won't work well), I think if I just target one platform to start, it might be reasonable to distribute a binary.

---

@stefanpenner to address your specific suggestion about giving instructions to the user if they are running an old version of watchman, I think that's a very reasonable suggestion but I don't think that will work well for my use case. The biggest focus of what I'm developing is to be a piece of software that handles everything like getting watchman setup properly for you, and so I want to be able to run a specific version of watchman or chokidar or something else that I've tested a bunch with the other stuff I'm bundling so that I can minimize the chances that the user needs to adjust anything in their own configuration.

Anyway, thanks in advance for any guidance or other thoughts or ideas you have.

[stefanpenner]

For (3), I think its really tricky. watchman might have been installed by brew, but if so, I don't want to perform brew upgrades on behalf of the user since that could be surprising or unwanted. And if watchman was installed through some other mechanism -- ex. if someone built it himself/herself -- then there may be no clean way to upgrade it.

either:

1. warn with actionable upgrade steps, and fallback to the old one.
2. if within semver, upgrade the existing one and move on. The user wont know the difference.

[wez]

It sounds like part of the problem being discussed is that both chokidar and watchman need to be built with a compiler today.

If you're targeting folks that don't have a compilation environment, it sounds like chokidar won't help at all compared to sane's watchman-less mode of operation.

Regarding making watchman easier to install for casual users: the problem isn't delivering a binary to a user so much as making sure that that binary is up to date. I've been meaning to talk with the osquery team at FB about their build and deployment infrastructure; with Windows support for watchman coming up we'll have to start providing a channel for distributing binaries. Even with that channel, you'll still have the issue of stale or concurrently installed versions.

Taking a step back; if you are a react-native user you kind of have to have a working compilation environment. With that in mind, I think the real problem is that react-native and/or sane would benefit from having clearer error reporting and suggestions on next steps and we can just tell the user what they should do next to get their watchman installation working with homebrew.

[stefanpenner]

watchman need to be built with a compiler today.

@wez don't you ship binaries for popular OS configurations? If not, could one?

[wez]

The watchman project does not currently have its own binary distribution channel. Everyone builds from source. We're likely to change this soonish because that just won't work for people on Windows.

[stefanpenner]

Closing this in-favor of #54, allowing users to provide watcher plugins sounds like a great path forward.