combining callbacks per watched path

[noahgrant]

This is more of a question than anything: I'm using three browserify bundles for a fairly large project--2 separate bundles and 1 test bundle--and I end up watching almost 600 paths. Some of them are duplicate paths (and a couple are even triplicate), and I imagine each one has a callback to update its own bundle. I'm running into some issues that seem to either stem from fsevents or libuv, and I wonder if this might have something to do with it, and if you think it might be better to register multiple callbacks per path as opposed to multiple watchers of the same path, each with a single callback.

Also, I was wondering if you've experienced a realistic limit on how many files can be watched at once? I have updated my ulimit to be ~2500, so that shouldn't be a problem.

Thanks!

[zertosh]

I keep my ulimit maxfiles at 2048 and I haven't had any problems yet.

I recently added the ability to ignore paths (see #170). Although a watcher instance is created, chokidar is smart enough to not actually do any fs watching. I typically ignore the node_modules directory.

I've also been playing with the idea of using sane. It's a lot like chokidar but on linux/os x it can use watchman. react-native uses it, and so far it seems to really hold up.

[zertosh]

@amasad Can you comment on how watchman stacks up against fsevents?

[mattdesl]

Interesting idea. A small irk I have with fsevents is the node-gyp install time, since I often install watchify locally per-module (for demos/examples).

[noahgrant]

i'm definitely open to any alternative. i've used watchman briefly and it seemed fine, but i didn't use it with as many files, nor in the context of gulp. also i'd like to +1 the ability to ignore paths, it took the total number down in half--it used to be near 1200!!

[amasad]

First, watchman is more than just OS X, it works on linux and windows support is underway. Second -- and this is my experience with fsevents via chokidar -- watching a large root (in terms of number of files, not file size) takes a lot longer to start.

[noahgrant]

i'm just curious, i can't be using watchify that differently than a lot of other people, but i don't really see people reporting this segfaulting a whole lot. Even in the issues I linked to, it was only when someone has the watcher going and removes all of node_modules and then runs npm i. i see this about half the time when i try to run my default gulp task! it's like a cranky lawnmower. once it runs, it's great; but getting it running is a big pain. Here's some sample code (i have three such bundles that i run, as well as a separate watch task for things like css, svg, and js linting):

var browserify = require('browserify'),
    browserSync = require('browser-sync'),
    env = require('./env'),
    gulp = require('gulp'),
    gulpif = require('gulp-if'),
    gutil = require('gulp-util'),
    source = require('vinyl-source-stream'),
    uglify = require('gulp-uglify'),
    watchify = require('watchify');

module.exports = (function() {
  function errorHandler(err) {
    // handle error
  }

  gulp.task('mybundle', function() {
    var bundler = browserify({
          entries: [
            './path/to/entry.js'
          ],
          cache: {},
          debug: true,
          fullPaths: true,
          insertGlobals: true,
          noParse: [
            'backbone',
            'd3',
            'underscore'
          ],
          packageCache: {}
        }
      ),
      rebundle = function(ids, done) {
        if (ids) {
          console.log('File changed: ' + ids[0]);
        }

        return bundler.bundle()
          .on('error', errorHandler)
          .pipe(source('bundle-dest.js'))
          .pipe(buffer())
          .pipe(gulpif(env.prod(), uglify().on('error', gutil.log)))
          .pipe(gulp.dest('./build/js/'))
          .pipe(browserSync.reload({stream: true}));
      };

    if (env.dev()) {
      bundler = watchify(bundler, {
        ignoreWatch: [
          '**/node_modules/**'
        ]
      });
      bundler.on('update', rebundle);
    }

    return rebundle();
  });
})();

Is there anything here that seems out of the ordinary? (my reactify/6to5ify transforms are in package.json)? Why would this be more prone to segfaulting than other peoples' gulp browserify tasks?

[jmm]

@noahgrant This is a longshot, but you don't happen to also be running Atom, do you? That seems to have issues with consuming tons of watches.

[noahgrant]

@jmm nope :-/

[noahgrant]

@zertosh what about adding an option to force chokidar to useFsEvents = false?

[noahgrant]

nm i see that setting opts.poll sets the chokidar opts.usePolling to true, and so useFsEvents will be set to false.

[zertosh]

@noahgrant There's an undocumented - use at your own risk - way of setting your own options on chokidar.

// not guaranteed to work in the future
var w = watchify(/*...*/);
var watcher = w._watcher;
w._watcher = function(file, opts) {
  // change the opts or do whatever
  return watcher(file, opts);
};

[noahgrant]

Thanks @zertosh. I believe chokidar is reverting back to fs.watch just by setting the poll option on watchify. On a related note, I'm having an issue where just by setting poll, my other js watch tasks (like linting) will run for a file, but only once. It's as though watchify (or chokidar, I'm not sure which) isn't completing. I left an issue on the chokidar repo, would it be possible for you to affirm that this is not watchify related before closing this issue? Thanks!

[zertosh]

@noahgrant I looked at paulmillr/chokidar#282, and if I understand correctly, the problem was caused two chokidars running in different modes?

[noahgrant]

actually, no--when running in two different modes, everything works great. but when watchify is running in poll mode (-> chokidar is running in !useFsEvents mode), a bunch of my other watch tasks (like linting) stop running after the first save (it's as though watchify never completely finishes, preventing other watch tasks from being called, although watchify does run on successive saves despite the other tasks not running). this happens with gulp-watch (which uses chokidar) also in !useFsEvents mode for the linting task. Does that make sense?

[zertosh]

that's such a weird interaction. thank you for figuring that out.